This question already has answers here:
Displaying Unicode in Powershell
(7 answers)
Closed 2 years ago.
Hei all,
as a console/terminal enthusiast and database administrator (PostgreSQL) it is essential for me to work with the correct charcater encoding.
Therefore, I want my client console/terminal window always set to e.g. UTF-8.
Back with Windows' CMD.EXE this attempt was as easy as typing the command chcp 65001 to set the desired code page identifier.
Now, I am in the process of switching to PowerShell and setting the character encoding seems very odd, IMHO.
I've done some research on how to set the PowerShell session to UTF-8 and I figured out, that I need three steps/commmnds to accomplish that.
PS C:\> $OutputEncoding = [System.Text.Encoding]::UTF8
PS C:\> [Console]::OutputEncoding = [System.Text.Encoding]::UTF8
PS C:\> chcp 65001
Despite the fact that the first two commands are not intuitive and hard to remember...
Leaving out one of them leads to something not working out properly!
Also, setting just one of them seems to have no effect to the others.
So, I must set all three for working with the PostgreSQL's psql database client.
Otherwise I run into encoding issues while exporting/importing data.
Now my question is: "Why the heck? Isn't there an easier way to simply set the character encoding in PowerShell?"
Unfortunately, I did not find any plausible documentation myself about setting the character enconding!
Thanks in advance
/EDIT
The second comment by TheIncorrigible1 led me to the best answer fo far: Displaying Unicode in Powershell
- So one can set the whole PowerShell with two separated statements to the desired encoding (UTF-8).
PS C:\> $OutputEncoding = [System.Console]::OutputEncoding = [System.Console]::InputEncoding = [System.Text.Encoding]::UTF8
PS C:\> $PSDefaultParameterValues['*:Encoding'] = 'utf8'
Explanation:
$OutputEncoding sets the encoding for e.g. | (piping) and/or communication between programs and/or processes.
[System.Console]::OutputEncoding sets the encoding for STDOUT and the console/terminal output.
[System.Console]::InputEncoding sets the encoding for STDIN or keyboard input.
$PSDefaultParameterValues['*:Encoding'] sets the encoding for all cmdlets that support the -Encoding option like e.g. Out-File -Encoding.
You could use PowerShell profiles. PowerShell supports several profile files. Also, PowerShell host programs can support their own host-specific profiles.
For example, the PowerShell console supports the following basic profile files. The profiles are listed in precedence order. The first profile has the highest precedence.
THE PROFILE FILES
Description Path
All Users, All Hosts $PSHOME\Profile.ps1
All Users, Current Host $PSHOME\Microsoft.PowerShell_profile.ps1
Current User, All Hosts $Home\[My ]Documents\PowerShell\Profile.ps1
Current user, Current Host $Home\[My ]Documents\PowerShell\Microsoft.PowerShell_profile.ps1
The profile paths include the following variables:
The $PSHOME variable, which stores the installation directory for PowerShell
The $Home variable, which stores the current user's home directory
From the following article, if you wish to read more about this,
https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_profiles?view=powershell-7.1&viewFallbackFrom=powershell-7.
Related
TLDR: I want know how to detect from the output of a shell (e.g. zsh, bash) the location of the prompts (e.g. user#machine /etc % ).
Details
I have made a working shell frontend in the browser based on xtermjs. It is now equivalent feature-wise to e.g. the default macOS terminal application with zsh, bash and powershell. In a nutshell, it works by executing a shell process (e.g. zsh) as the child of a parent process that pipes the input/output from/to the browser via web sockets.
I want now to step up and implement a "collapse" functionality that hides the output of the selected commands in the history (like Visual Studio Code does now).
To this end, I need to detect the location of the prompts from the terminal output: the collapse function would then hide the characters between two consecutive prompts.
I know I can use the approaches below:
detect the prompt with a regular expression (I would need to parse the PS1 variable)
inject some special character sequence before and after the prompt (e.g. in variable PS1)
But both do not seem very robust, and may not work with some specific command interpreter. I could not find yet the location where this functionality is implemented in the source code of Visual Studio Code.
My question is: is there a robust way to achieve this functionality for at least zsh, bash and powershell (it is fine if it is specific to xterm.js)
Edit 1
This SO question is related: ANSI escape sequence for collapsing/folding text (maybe hierarchically)
It links to this interesting thread: https://github.com/PerBothner/DomTerm/issues/54
It appears that DomTerm uses escapes sequences at folding points (my solution 2).
Yet I don't see how to inject them into the terminal, besides hacking the PS1 env var.
Edit 2
While parsing iTerm's documentation I found out that it takes advantage of the hooks provided by the shell (e.g. for zsh) in order to print some special escape sequence at various locations, including before showing the prompt.
For example, in zsh, I can print string "🐮" before each prompt be executing precmd() { echo '🐮' }. Then when I execute e.g. ls I get
$> ls
[...]
🐮
$>
There is a more extensive explanation of the various available hooks for various shells here.
It looks like PowerShell uses a very different system though.
Just a pretty stand curl command call an S3 end point for download using all default values. On a mac, or on a PC using command line I get 103MBsec if cached on cdn and 80mbsec otherwise. Same command, same bucket, same object, using "curl.exe" and I get 1MBSec when call through powershell. I guess powershell does something different that make it's totally slow? I tried using newest curl binary but still the same.
I guess I am misunderstanding what powershell is doing when I use a curl command
curl.exe yourfileonS3 >> output.bin
To complement briantist's helpful answer:
In PowerShell, the redirection operators > and >> are in effect aliases of Out-File and Out-File -Append.
> and >> are therefore not mere byte-stream conduits, and, in fact, PowerShell as of v7.2 does not support sending raw byte output to a file.
Instead, PowerShell invariably decodes output from any external program as text ([string] instances), based on the character encoding reported by [Console]::OutputEncoding] and then, on saving to the target file with Out-File (possibly via > / >>), re-encodes these strings, using that cmdlet's default character encoding (unless overridden with -Encoding in an explicit Out-File call).
Not only does this not preserve the external program's raw byte
output, it adds significant overhead.
To get raw byte processing, call cmd.exe[1] and use its redirection operators:
cmd /c 'curl.exe yourfileonS3 >> output.bin'
See this answer for more information.
[1] On Unix-like platforms, use sh -c 'curl yourfileonS3 >> output.bin'
See mklement0's answer for full context on this (I recommend accepting that one!), and the important point that handling of byte streams in redirection is problematic and error prone in PowerShell and should be avoided.
So I looked into this and I believe the reason is that >> (file redirection) is the slow part.
I originally suspected you might be calling curl (which is aliased to Invoke-WebRequest in Windows PowerShell), but I was able to reproduce the speed difference between curl.exe directly in PowerShell vs cmd.exe, and measure it, this way:
# call curl.exe and do redirection in PowerShell
Measure-Command -Expression { curl.exe https://uploader.codecov.io/v0.1.0_6943/linux/codecov >> delme.bin }
del delme.bin
# call cmd.exe and do redirection there
Measure-Command -Expression { & cmd.exe /c 'curl.exe https://uploader.codecov.io/v0.1.0_6943/linux/codecov >> delme.bin' }
del delme.bin
This was enough to show a stark difference.
I also confirmed that this problem is a little bit worse in Windows PowerShell as opposed to later cross-platform versions (pwsh.exe). In Windows, with version 7.1.0, the same commands above still show a large difference.
I'm trying to run an application that reads an environment variable that contains a JSON with about 22k characters. The project setup tells me to use $(cat ./path/to/file) to correctly configure it, but as I'm using windows, this commands do not work.
I've tried copying the contents of the file to the variable using the GUI Environment Variable, but its input truncates the value to a certain limit which is not even on half of the file.
After this I tried setting the variable using the Powershell with the command:
$env:myvar = iex '$(type path/to/file)'
and then saving the result with:
[System.Environment]::SetEnvironmentVariable('MYVAR', $env:MYVAR, [System.EnvironmentVariableTarget]::Machine)
After these commands, Powershell is able to print the result correctly but CMD still prints only part of the value when I echo it.
This is very odd because the regedit shows the correct value as suggested here.
The application still can't process the value because it is not complete.
Is there any fix for this?
Note: This answer applies to Windows.
tl;dr
While you can store up to 32,766 characters in a single environment variable, the standard retrieval mechanisms in cmd.exe and PowerShell / .NET (as of v7.1 / 5.0) support only up to 4,095.
A workaround in PowerShell is possible, but ultimately it comes down to whether the target executable that is meant to read an environment-variable value supports reading values up to the technical maximum length.
The technical limit for the number of characters in a single environment variable is 32,766 (32KB = 32768, minus 2).
Starting with Windows Server 2008 / Windows Vista, there is no longer a limit on the overall size of the environment block - see the docs.
However, depending on how the environment-variable is retrieved, the limit may be lower:
Both cmd.exe and PowerShell, as of v7.1 / .NET 5.0, support retrieving at most 4,095 characters.
However, in PowerShell you can retrieve longer values, assuming the variable of interest is defined persistently in the registry, and assuming that you know whether it is defined at the machine or user level; e.g., for a MYVAR environment variable:
At the machine level:
Get-ItemPropertyValue 'registry::HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Environment' MYVAR
At the user level:
Get-ItemPropertyValue registry::HKEY_CURRENT_USER\Environment MYVAR
try the type command. It is the windows equivalent of the unix cat command. This means storing the json inside of a seperate file and using the command "type <path_to_file>".
I am having trouble getting vimdiff to work on a Windows 10 machine. I am running vim from Powershell. Powershell is also declared in $myvimrc as my shell of choice:
set shell=C:\WINDOWS\system32\WindowsPowershell\v1.0\powershell.exe
The documents I am attempting to compare are not saved as files. I open two vertical splits, enter text into each, and run :windo diffthis. The output is E97: Cannot create diffs.
This article says I may need to download and use a different diff.exe than the one installed with gvim. I have downloaded the recommended "GnuWin32 diff" package and added the install directory to my Windows Path ($env:path). Continuing to follow these directions, I've commented out the default diffexpr declaration, but still get E97.
I have also tried calling my own function to no avail. In this attempt, I've made sure to escape backslashes, and have also copied the downloaded diff.exe to a directory I am confident I have full permissions to. To help with troubleshooting, I've temporarily saved the two files I wish to compare, and specified their full paths explicitly rather than using vim's v:fname_in and v:fname_new (and v:fname_out).
set diffexpr=TestDiff()
function TestDiff()
silent execute "!& " . "C:\\diff.exe" . " -a --binary " "C:\\a.edi" . " " . "C:\\b.edi" . " > " . "C:\\tmp.txt"
endfunction
I've researched this error by running :h E97, which returns the following information:
Vim will do a test if the diff output looks alright. If it doesn't,
you will get an error message. Possible causes:
The "diff" program cannot be executed.
The "diff" program doesn't produce normal "ed" style diffs (see above).
The 'shell' and associated options are not set correctly. Try if filtering works with a command like ":!sort".
You are using 'diffexpr' and it doesn't work. If it's not clear what the problem is set the 'verbose' option to one or more to see
more messages.
The self-installing Vim for MS-Windows includes a diff program. If
you don't have it you might want to download a diff.exe. For example
from http://gnuwin32.sourceforge.net/packages/diffutils.htm.
I believe that I pass the first two of these requirements, as tmp.txt is generated and follows the example "ed" style diff that is provided in the help file. I have not set other shell-related parameters (shelltype, shellpipe, etc) in $myvimrc, but executing other commands with :! complete without issue.
There is no additional information in :messages, only the E97 error.
Edit:
If I remove set shell=C:\WINDOWS\system32\WindowsPowershell\v1.0\powershell.exe from $myvimrc, defaulting the shell back to cmd.exe, diffthis works as expected. It seems that others have also had this problem when using Powershell.
Below are captures of the command windows that pop up when running diffthis with both shells. The commands used are nearly identical.
I had speculated that the forward slashes apparent in the Powershell version of this attempt were problematic, but I am able to run this command exactly (with dummy files in place of the .tmp files) and it outputs what seems to be an adequate file for use with diff. I've also tried adding a substitute of forward slashes to back slashes to no avail.
The problem boils down to how vim/Powershell are A) encapsulating the command in quotes and B) handling white space in path names.
I am able to bypass this problem with the following changes to $myvimrc:
set shell=powershell
set shellcmdflag=-c
set shellquote="
set shellxquote=
Also a change in the default MyDiff() function within the if that sets the path to diff in the cmd variable:
" below is the default
" let cmd = substitute($VIMRUNTIME, ' ', '" ', '') . '\diff"'
let cmd = "C:/diff"
This approach is dependent upon copying the diff.exe that ships with vim to a directory that doesn't contain spaces in the path for simplicity (C:\diff.exe).
The resulting command that is executed is:
powershell -c C:/diff -a --binary C:/<redacted>/a.tmp C:/<redacted>/b.tmp > C:/<redacted>/c.tmp
Running Node.js from PowerShell on Windows assumes that stdout must be a UTF-16 text stream. This generates surprising results for tools which send binary Buffer objects to stdout. Is there a way to work around this?
Command-line example:
node -e "process.stdout.write(new Buffer('abc'))" >tmp.dat
Using PowerShell, the resulting file, tmp.dat is now 12 bytes long rather than only 3 bytes long.
The exact same example launched from CMD.EXE produces the expected 3-byte file.
P.S. I originally thought this was a node.js question, but its now clear it's a PowerShell issue.
The best answer for this seems to be to spawn cmd.exe and do native redirection with it, rather than depend on PowerShell's redirection model. It is a little odd as it requires putting the redirection in quotes, as follows:
cmd /c node -e "process.stdout.write(new Buffer('abc'))" '>tmp.dat'
It really seems too bad that PowerShell doesn't recognize that since "node" is a native command any redirection should be handled using the native redirection mechanisms.