I'm trying to run an application that reads an environment variable that contains a JSON with about 22k characters. The project setup tells me to use $(cat ./path/to/file) to correctly configure it, but as I'm using windows, this commands do not work.
I've tried copying the contents of the file to the variable using the GUI Environment Variable, but its input truncates the value to a certain limit which is not even on half of the file.
After this I tried setting the variable using the Powershell with the command:
$env:myvar = iex '$(type path/to/file)'
and then saving the result with:
[System.Environment]::SetEnvironmentVariable('MYVAR', $env:MYVAR, [System.EnvironmentVariableTarget]::Machine)
After these commands, Powershell is able to print the result correctly but CMD still prints only part of the value when I echo it.
This is very odd because the regedit shows the correct value as suggested here.
The application still can't process the value because it is not complete.
Is there any fix for this?
Note: This answer applies to Windows.
tl;dr
While you can store up to 32,766 characters in a single environment variable, the standard retrieval mechanisms in cmd.exe and PowerShell / .NET (as of v7.1 / 5.0) support only up to 4,095.
A workaround in PowerShell is possible, but ultimately it comes down to whether the target executable that is meant to read an environment-variable value supports reading values up to the technical maximum length.
The technical limit for the number of characters in a single environment variable is 32,766 (32KB = 32768, minus 2).
Starting with Windows Server 2008 / Windows Vista, there is no longer a limit on the overall size of the environment block - see the docs.
However, depending on how the environment-variable is retrieved, the limit may be lower:
Both cmd.exe and PowerShell, as of v7.1 / .NET 5.0, support retrieving at most 4,095 characters.
However, in PowerShell you can retrieve longer values, assuming the variable of interest is defined persistently in the registry, and assuming that you know whether it is defined at the machine or user level; e.g., for a MYVAR environment variable:
At the machine level:
Get-ItemPropertyValue 'registry::HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Environment' MYVAR
At the user level:
Get-ItemPropertyValue registry::HKEY_CURRENT_USER\Environment MYVAR
try the type command. It is the windows equivalent of the unix cat command. This means storing the json inside of a seperate file and using the command "type <path_to_file>".
Related
The variable in the Terraform file (infrastructure.tf) is declared like this:
variable "tags" {
type = map(string)
}
This is the PowerShell code that executes the terraform command-line program with the plan command:
$command = "plan"
$options = #(
"'--var=tags={a:\`"b\`"}'"
"--out=path/to/out.tfplan"
)
Write-Host terraform,$command,$options
& terraform $command $options
The Write-Host command's output is:
terraform plan '--var=tags={a:\"b\"}' --out=path/to/out.tfplan
If I copy paste that into an interactive PowerShell 7.2 (pwsh) session then it works. But the & terraform command fails with this error:
Error: Too many command line arguments
To specify a working directory for the plan, use the global -chdir flag.
I know the Terraform documentation for the plan command warns against using PowerShell to run the command, due to quoting issues (emphasis mine):
PowerShell on Windows cannot correctly pass literal quotes to external programs, so we do not recommend using Terraform with PowerShell when you are on Windows. Use Windows Command Prompt instead.
Unfortunately they don't specify if they mean Windows PowerShell 5.1 included in Windows (powershell.exe), or also PowerShell 7.2 running on Windows (pwsh). But with pwsh it is clearly possible to run the plan command in an interactive PowerShell session (I am using macOS, but some others use Windows) and pass literal quotes to an external program. It just seems that running the same command from a .ps1 file does not work.
Since our whole dev/ops tooling is PowerShell-based, I'd like to know: is this possible?
Because if it is not, then we will have to work around this limitation.
Edit:
Some things I've tried:
Use splatting for $options (e.g. #options)
Add the stop-parsing token (e.g. --%) as the first item in $options (that token is actually "only intended for use on Windows platforms")
Many variations of single/double quotes
Remove the \ (I am not actually sure why this seems required, the map literal syntax is either {"x"="y"} or {x:"y"}), but without the \ copy-pasting the printed command line in an interactive PowerShell also does not work.
tl;dr
Omit the embedded enclosing '...' around --var=..., because they will become a literal part of your argument.
The - unfortunate - need to manually \-escape the embedded " instances, even though PowerShell itself does not need it, is the result of a long-standing bug that was finally fixed in PowerShell (Core) 7.3.0; in 7.3.0 and up to at least 7.3.1, the fix is in effect by default, which breaks the solution below, and therefore requires $PSNativeCommandArgumentPassing = 'Legacy'; however, it looks like the fix will become opt-in in the future, i.e. the old, broken behavior (Legacy) will become the default again - see this answer.
Using Write-Host to inspect the arguments isn't a valid test, because, as a PowerShell command, it isn't subject to the same rules as an external program.
For ways to troubleshoot argument-passing to external programs, see the bottom section of this answer.
$command = "plan"
$options = #(
"--var=tags={a:\`"b\`"}" # NO embedded '...' quoting
"--out=path/to/out.tfplan"
)
# No point in using Write-Host
& { # Run in a child scope to localize the change to $PSNativeCommandArgumentPassing
# Note: Only needed if you're (also) running on PowerShell 7.3+
$PSNativeCommandArgumentPassing = 'Legacy'
& terraform $command $options
}
How to control the exact process command line on Windows / pass arguments with embedded double quotes properly on Unix:
Note: The solution above relies on PowerShell's old, broken behavior, and while it works in the case at hand, a fully robust and less conceptually confusing solution requires more explicit control over how the arguments are passed, as shown below.
A cross-edition, cross-version, cross-platform solution:
Assuming that terraform must see --var=tags={a:\"b\"} on its process command line on Windows, i.e. needs to see the argument as verbatim --var=tags={a:"b"} after parsing its command line, combine --%, the stop-parsing token, with splatting, which gives you full control over how the Windows process command line is built behind the scenes:
$command = "plan"
$options = #(
'--%'
'--var=tags={a:\"b\"}'
'--out=path/to/out.tfplan'
)
& { # Run in a child scope to localize the change to $PSNativeCommandArgumentPassing
# !! Required in v7.3.0 and up to at least v7.3.1, due to a BUG.
$PSNativeCommandArgumentPassing = 'Legacy'
& terraform $command #options
}
This creates the following process command line behind the scenes on Windows (using an example terraform path):
C:\path\to\terraform.exe plan --var=tags={a:\"b\"} --out=path/to/out.tfplan
Note:
In PowerShell (Core) 7.3.0 and at least up to 7.3.1, --% is broken by default, in that its proper functioning is mistakenly tied to value of the v7.3+ $PSNativeCommandArgumentPassing preference variable; thus, (temporarily) setting $PSNativeCommandArgumentPassing = 'Legacy' is required, as shown above - see GitHub issue #18664 for the bug report.
Even though --% is primarily intended for Windows, it works on Unix-like platforms too, as long as you use the Microsoft C/C++ command-line syntax rules to formulate the arguments; specifically, this means:
only use " characters for quoting (with syntactic function)
use \ only to escape " chars.
While you can use --% without splatting, doing so comes with severe limitations - see this answer.
A simpler, but Windows-only cross-edition, cross-version solution:
Calling via cmd /c also gives you control over how the command line is constructed:
$command = "plan"
$options = #(
'--var=tags={a:\"b\"}'
'--out=path/to/out.tfplan'
)
cmd /c "terraform $command $options"
Note: This is often more convenient than --%, but suboptimal, because:
The intermediary cmd.exe call creates extra overhead.
% characters may be interpreted by cmd.exe, and, in unquoted arguments, additional metacharacters such as & and ^ - preventing that requires extra effort.
A v7.3+ cross-platform solution:
Relying on PowerShell's corrected behavior in v7.3+ (no need for manual \-escaping anymore) requires setting $PSNativeCommandArgumentPassing to 'Standard'.
Note: If you target only Unix-like platforms, that isn't necessary.
$command = "plan"
$options = #(
'--var=tags={a:"b"}' # Note: NO \-escaping of " required anymore.
'--out=path/to/out.tfplan'
)
& { # Run in a child scope to localize the change to $PSNativeCommandArgumentPassing
# Necessary on Windows only.
$PSNativeCommandArgumentPassing = 'Standard'
& terraform $command $options
}
Note: On Windows, this creates a slightly different process command line than the solutions above; notably, --var=tags={a:\"b\"} is enclosed in "..." as a whole; however, well-behaved CLIs should parse this as verbatim --var=tags={a:"b"} too, whether enclosed in "..." or not.
C:\path\to\terraform.exe plan "--var=tags={a:\"b\"}" --out=path/to/out.tfplan
In our release pipeline, we have a console app that performs a function which generates an encryption key and outputs it to STDOUT. We need to be able to use this value in a variable during deployment (updating a configuration file with the results from the console app). We've tried using the Output Variables option in the command line task in Azure Devops but unfortunately we need it in a different format... and it just doesn't seem to work as expected.
E.g. Our cmd line tool outputs 908321093RANDOMLYGENERATEDKEY3422543 to STDOUT
The name in our config file for that key is something like Settings.Security.OurKey however the output variable in the command line task does not allow periods (.) and as such is set to SettingsSecurityOurKey... we've also tried SETTINGS_SECURITY_OURKEY, but the variable value is never set by the task.
Is it possible to somehow set the Azure Devops variable to the value of the output variable from the command line or a powershell script? Something like:
set $(Settings.Security.OurKey) = SettingsSecurityOurKey
Or is there a simpler method of achieving this? It seems like it shouldn't be that difficult..
This sounds like a Powershell issue rather than an issue with Azure DevOps.
# Variable name with special characters
$VariableName.That.Contains.Periods # This will NOT work.
${VariableName.That.Contains.Periods} # This will work.
Refer this for more information: https://blog.danskingdom.com/accessing-powershell-variables-with-periods-in-their-name/
If you want a PowerShell variable to contain the standard output from a command, just assign it:
$yourVariableName = your_command_that_writes_to_stdout
If the output is only one line, the PowerShell variable will contain a single string; otherwise it will contain an array.
This question already has answers here:
Displaying Unicode in Powershell
(7 answers)
Closed 2 years ago.
Hei all,
as a console/terminal enthusiast and database administrator (PostgreSQL) it is essential for me to work with the correct charcater encoding.
Therefore, I want my client console/terminal window always set to e.g. UTF-8.
Back with Windows' CMD.EXE this attempt was as easy as typing the command chcp 65001 to set the desired code page identifier.
Now, I am in the process of switching to PowerShell and setting the character encoding seems very odd, IMHO.
I've done some research on how to set the PowerShell session to UTF-8 and I figured out, that I need three steps/commmnds to accomplish that.
PS C:\> $OutputEncoding = [System.Text.Encoding]::UTF8
PS C:\> [Console]::OutputEncoding = [System.Text.Encoding]::UTF8
PS C:\> chcp 65001
Despite the fact that the first two commands are not intuitive and hard to remember...
Leaving out one of them leads to something not working out properly!
Also, setting just one of them seems to have no effect to the others.
So, I must set all three for working with the PostgreSQL's psql database client.
Otherwise I run into encoding issues while exporting/importing data.
Now my question is: "Why the heck? Isn't there an easier way to simply set the character encoding in PowerShell?"
Unfortunately, I did not find any plausible documentation myself about setting the character enconding!
Thanks in advance
/EDIT
The second comment by TheIncorrigible1 led me to the best answer fo far: Displaying Unicode in Powershell
- So one can set the whole PowerShell with two separated statements to the desired encoding (UTF-8).
PS C:\> $OutputEncoding = [System.Console]::OutputEncoding = [System.Console]::InputEncoding = [System.Text.Encoding]::UTF8
PS C:\> $PSDefaultParameterValues['*:Encoding'] = 'utf8'
Explanation:
$OutputEncoding sets the encoding for e.g. | (piping) and/or communication between programs and/or processes.
[System.Console]::OutputEncoding sets the encoding for STDOUT and the console/terminal output.
[System.Console]::InputEncoding sets the encoding for STDIN or keyboard input.
$PSDefaultParameterValues['*:Encoding'] sets the encoding for all cmdlets that support the -Encoding option like e.g. Out-File -Encoding.
You could use PowerShell profiles. PowerShell supports several profile files. Also, PowerShell host programs can support their own host-specific profiles.
For example, the PowerShell console supports the following basic profile files. The profiles are listed in precedence order. The first profile has the highest precedence.
THE PROFILE FILES
Description Path
All Users, All Hosts $PSHOME\Profile.ps1
All Users, Current Host $PSHOME\Microsoft.PowerShell_profile.ps1
Current User, All Hosts $Home\[My ]Documents\PowerShell\Profile.ps1
Current user, Current Host $Home\[My ]Documents\PowerShell\Microsoft.PowerShell_profile.ps1
The profile paths include the following variables:
The $PSHOME variable, which stores the installation directory for PowerShell
The $Home variable, which stores the current user's home directory
From the following article, if you wish to read more about this,
https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_profiles?view=powershell-7.1&viewFallbackFrom=powershell-7.
I have added the following command to the runonce registry to run a specific python script on start up with a long list of parameters as follows
HKLU..."fio-test"=C:\Python26\python.exe C:\Users\Administrator\Desktop\workspace\test_Windows_dc_policy_flush_on_restart\test_Windows_dc_policy_flush_on_restart.py -I /dev/fct4 -b E -v "C:\Users\Administrator\Desktop\workspace\test_Windows_dc_policy_flush_on_restart\test_actions.p"
However, the command will never run and the registry will not be removed for the script. IE. the runonce part of the windows boot manager will ignore the entry. I can find the entry and see it in the proper location and see other less complicated scripts run and be removed from the registry. My question is as follows.. What are the limitation of the run once that I am running into? am I running into a max command length issue? or, is it some type of unescaped character that causes the script to be ignored?
The Task Scheduler is a more flexible and powerful of launching these types of tasks. You should be able to use it to get around the command length limitation.
So, is there a limit to how deeply environment variables can be nested in Windows? I do a lot of development work and I'm trying to set up my development environment vars, and a lot of them nest off each other, ie.
GLEW=%THIRD_PARTY_ROOT%\GLEW
GLEW_1_5_5=%GLEW%\glew-1.5.5
GLEW_BIN_PATH=%GLEW_ROOT%\bin
GLEW_INCLUDE_PATH=%GLEW_ROOT%\include
GLEW_LIB_PATH=%GLEW_ROOT%\lib
GLEW_ROOT=%GLEW_1_5_5%
OSG=%THIRD_PARTY_ROOT%\OpenSceneGraph
OSG_2_8_3=%OSG%\OpenSceneGraph-2.8.3
OSG_BIN_PATH=%OSG_ROOT%\bin
OSG_INCLUDE_PATH=%OSG_ROOT%\include
OSG_LIB_PATH=%OSG_ROOT%\lib
OSG_ROOT=%OSG_2_8_3%
THIRD_PARTY_ROOT=C:\dev\third-party
But I was having a heck of a time getting them to actually expand properly. For a while when I looked at the output of set, I was just getting what looked like they were being expanded in order and so any ones that depended on foo_ROOT weren't being expanded properly. I tried enabling delayed expansion and that didn't help, but restarting seemed to... so maybe delayed expansion required a restart..
Either way, I have GLEW_BIN_PATH and OSG_BIN_PATH in my PATH variable and they refuse to expand. This is really frustrating because the dll's are residing there and of course I can get evvvverything else to expand... any thoughts?
Edit: I have them in the PATH variable as:
[everything else....];%GLEW_BIN_PATH%;%OSG_BIN_PATH%
So I'm not seeing an obvious cause to keep them from expanding out..
It looks like there is a lexicographic dependency on the variables definition.
Windows expands and populates the Enviroment Variables on lexicographic order (:-O)
You can only use on your variable, variables that are "lexicographically" lower than your variable.
Example:
Following Definition:
VAR_01=1
VAR_02=%VAR_01%-2
VAR_03=%VAR_02%-3
Produces
VAR_01 is 1
VAR_02 is 1-2
VAR_03 is 1-2-3
But
VAR_01=1
VAR_02=%VAR_03%-2
VAR_03=%VAR_01%-3
Produces
VAR_01 is 1
VAR_02 is -2
VAR_03 is 1-3
Due VAR_03 is not defined when VAR_02 is expanded.
Yeah, this is driving me crazy. Full repro by:
System Properties, Environment Varialbles, set up like so:
one = c:
two = %ONE%\two
three = %TWO%\three
Then click OK, and run cmd. Here's what I see:
C:\>set one
one=C:
C:\>set two
two=C:\two
C:\>set three
three=%TWO%\three
This link explains for Vista, but does not mention that it happens on Win7.
http://support.microsoft.com/kb/911089
...Jonas
I've had success with escaping the percent sign:
GLEW=%%THIRD_PARTY_ROOT%%\GLEW
THIRD_PARTY_ROOT=C:\dev\third-party
C:\>set GLEW
GLEW=C:\dev\third-party\GLEW
When viewing the variable from the Windows Environment Variable window, it will display as
GLEW | %THIRD_PARTY_ROOT%\GLEW
Note: The double percent signs will only work inside a script. If using on the command line, use the caret escape character (e.g. GLEW=^%THIRD_PARTY_ROOT^%\GLEW).
Have you saved all of the needed variables in the System Variables as well? Because in order to expand the values, the system will have to have a "permanent" memory of all the needed variables.
If you do all these together in a row on the command line, just saying X=something and Y=%X%;else, then when you set the path to PATH=%PATH%;%Y%, the shell expands the values of all the variables before it saves the new value of PATH; but as soon as you close the Command Prompt window, the system has forgotten %X% and %Y% entirely.
However, if you use the System Properties Control Panel to set the PATH to include the unexpanded variables, than all those variables are going to have to exist as system variables when you reboot, or they'll fail to expand.
To ensure that you are saving all the variables in the system so that they are still there after the reboot, either use the System Properties Control Panel or the SETX.EXE command. If you are going to use them in the system PATH (not just your user account's path), then you'll want to use SETX /M X=blah or the bottom part of the System Properties | Environment Variables tab, labeled "System variables".
I have been experiencing this in Windows 10, build 1903.
For me, the solution was to remove the PATH variable from 'user' (the upper half of the system properties dialog) and keep only the path in 'system'. Restart the cmd.exe shell or use refreshenv to reload the properties, and expansion should work again.
This looks like a bug in Windows for how it resolves user vs system properties, or possibly the processing order. For me it was not replacing values in the user PATH with values from the 'system' set of env vars. It might work better if all the variables are in user, but I've not tested this hypothesis.