I want to make the font as bold for the path that I'm printing using Write-Host. I'm flexible to using other methods like echo or something else.
I've tried other methods like Write-Debug, etc, also checked the module WindowsConsoleFonts.
But none of them supports font properties like making them bold or italic while printing them.
$pathString = "[" + (Get-Location) + "]"
Write-Host $pathString -ForegroundColor Cyan
I'm using PowerShell 5.1 which doesn't support MarkDown rendering, else I would have done it using Markdown.
You can achieve bold text via VT (Virtual Terminal) escape sequences.
However, regular Windows console windows (conhost.exe) do not support italics, and neither does their upcoming successor, Windows Terminal (at least as of this writing).[1]
In recent versions of Windows 10, support for VT sequences is enabled by default in both Windows PowerShell and PowerShell Core.
However, Write-Host has no support for them, so you must embed the escape sequences directly into the strings you pass to Write-Host (or strings you send to the success output stream, if it goes to the console):
Note:
I'm omitting Write-Host from the examples below, because it isn't strictly necessary, but colored text generally should indeed be written to the display (host), not to the success output stream.
While it is better to consistently use VT sequences for all formatting needs - including colors - it is possible to combine them with Write-Host -ForegroundColor /-BackgroundColor`.
PowerShell Core:
PowerShell Core supports embedding escape sequences directly in "..." (double-quoted strings), via the `e escape sequence, which expands to a literal ESC character, which is the character that initiates a VT escape sequence.
"To `e[1mboldly`e[m go ..., in `e[36mcyan`e[m."
Windows PowerShell:
There's no support for `e; the easiest solution is to use \e as placeholders and use -replace to substitute actual ESC characters (Unicode code point 0x1b) for them:
"To \e[1mboldly\e[m go ..., in \e[36mcyan\e[m." -replace '\\e', [char] 0x1b
[1] From PowerShell Core, you can run the following test command to see if the word italics prints in italics: "`e[3mitalics`e[m after"
Note that italics in the terminal are supported on macOS and at least in some Linux distros; e.g., Ubuntu 18.04
Related
In powershell(v2) condition,I run the command,it goes gibberish.
echo "すみません" | ruby -e "puts gets"
?????
My powershell is V2.How should I get the correct output?
As implied by the answer to your related question, you need to set $OutputEncoding / [Console]::OutputEncoding first to match the character encoding that ruby expects / outputs; e.g., for UTF-8:
# Make PowerShell both send and receive data as UTF-8 when talking to
# external (native) programs.
# Note:
# * In *PowerShell (Core) 7+*, $OutputEncoding *defaults* to UTF-8.
# * You may want to save and restore the original settings.
$OutputEncoding = [Console]::OutputEncoding = [Text.UTF8Encoding]::new()
echo "すみません" | ruby -e "puts gets"
As for what you tried:
With Windows PowerShell using ASCII(!) encoding by default when sending data via the pipeline to external programs (see below), characters such as す can NOT be represented, and get "lossily" translated to literal ? characters.
Since none of the characters in your input string have an ASCII representation, the (ASCII representation of) verbatim string ?????
was sent to ruby.
Background information:
In PowerShell it is the $OutputEncoding preference variable that determines the encoding used for sending data TO external programs, and in Windows PowerShell it defaults to ASCII(!), i.e. only characters in the 7-bit ASCII subrange of Unicode can be represented.
In PowerShell (Core) 7+, $OutputEncoding now more sensibly defaults to (BOM-less) UTF-8.
Unfortunately, with respect to receiving data from external programs, where [Console]::OutputEncoding matters for correct interpretation of the output, even PowerShell (Core) 7+ still uses the legacy OEM code page by default as of v7.3.1, which means that UTF-8 output from external programs is by default misinterpreted - see GitHub issue #7233 for a discussion.
In PowerShell (5.1):
Calling an external command (in this case nssm.exe get logstash-service Application) the output is displayed in PowerShell as I would have expected (ASCII-string "M:\logstash-7.1.1\bin\logstash.bat"):
PS C:\> M:\nssm-2.24\win64\nssm.exe get logstash-service Application
M:\logstash-7.1.1\bin\logstash.bat
But the following command (which pipes the output into Out-Default) results in:
PS C:\> M:\nssm-2.24\win64\nssm.exe get logstash-service Application | Out-Default
M : \ l o g s t a s h - 7 . 1 . 1 \ b i n \ l o g s t a s h . b a t
(Please note all that "whitespace" separating all characters of the resulting output string)
Also the following attempt to capture the output (as an ASCII string) into variable $outCmd results in :
PS C:\> $outCmd = M:\nssm-2.24\win64\nssm.exe get logstash-service Application
PS C:\> $outCmd
M : \ l o g s t a s h - 7 . 1 . 1 \ b i n \ l o g s t a s h . b a t
PS C:\>
Again, please note the separating whitespace between the characters.
Why is there a difference in the output between the first and the latter 2 commands?
Where are the "spaces" (or other kinds of whitespace chars) coming from in the output of the latter 2 commands?
What exactly needs to be done in order to capture the output of that external command as ASCII string "M:\logstash-7.1.1\bin\logstash.bat" (i.e. without the strange spaces in between)?
If the issue is related to ENCODING, please specify what exactly needs to be done/changed.
Yes, the problem is one of character encoding, and the problem often only surfaces when an external program's output is either captured in a variable, sent through the pipeline, or redirected to a file.
Only in these cases does PowerShell get involved and decodes the output into .NET strings before any further processing.
This decoding happens based on the encoding stored in the [Console]::OutputEncoding property, so for programs that do not themselves respect this encoding for their output you'll have to set this property to match the actual character encoding used.
Your symptom implies that nssm.exe outputs UTF-16LE-encoded ("Unicode") strings[1], so to capture them properly you'll have to do something like the following:
$orig = [Console]::OutputEncoding
[Console]::OutputEncoding = [System.Text.Encoding]::Unicode
# Store the output lines from nssm.exe in an array of strings.
$output = M:\nssm-2.24\win64\nssm.exe get logstash-service Application
[Console]::OutputEncoding = $orig
The underlying problem is that external programs are expected to use the current console's output code page for their output encoding, which defaults to the system's active legacy OEM code page, as reflected in [Console]::OutputEncoding (and reported by chcp), but some do not, in an attempt to:
either: overcome the limitations of the legacy, single-byte OEM encodings in order to provide full Unicode support (as is the case here, although it is more common to do that with UTF-8 encoding, as the Node.js CLI, node.exe does, for instance)
or: use the more widely used active ANSI legacy code page instead (as python does by default).
See this answer for additional information, which also links to two helper functions:
Invoke-WithEncoding, which wraps capturing output from an external program with a given encoding (see example below), and Debug-NativeInOutput, for diagnosing what encoding a given external program uses.
With function Invoke-WithEncoding from the linked answer defined, you could then call:
$output = Invoke-WithEncoding -Encoding Unicode {
M:\nssm-2.24\win64\nssm.exe get logstash-service Application
}
[1] The apparent spaces in the output are actually NUL characters (code point 0x0) that stem from the 0x0 high bytes of 8-bit-range UTF-16LE code units, which includes all ASCII characters and most of Windows-1252): Because PowerShell, based on the single-byte OEM code page stored in [Console]::Encoding (e.g., 437 on US-English systems), interprets each byte as a whole character, the 0x0 bytes of 2-byte (16-bit) Unicode code units (in the 8-bit range) are mistakenly retained as NUL characters, and in the console these characters present like spaces.
Note: This self-answered question describes a problem that is specific to using Eclipse Mosquitto on Windows, where it affects both Windows PowerShell and the cross-platform PowerShell (Core) edition, however.
I use something like the following mosquitto_pub command to publish a message:
mosquitto_pub -h test.mosquitto.org -t tofol/test -m '{ \"label\": \"eé\" }'
Note: The extra \-escaping of the " characters, still required as of Powershell 7.1, shouldn't be necessary, but that is a separate problem - see this answer.
Receiving that message via mosquitto_sub unexpectedly mangles the non-ASCII character é and prints Θ instead:
PS> $msg = mosquitto_sub -h test.mosquitto.org -t tofol/test; $msg
{ "label": "eΘ" } # !! Note the 'Θ' instead of 'é'
Why does this happen?
How do I fix the problem?
Problem:
While the mosquitto_sub man page makes no mention of character encoding as of this writing, it seems that on Windows mosquitto_sub exhibits nonstandard behavior in that it uses the system's active ANSI code page to encode its string output rather than the OEM code page that console applications are expected to use.[1]
There also appears to be no option that would allow you to specify what encoding to use.
PowerShell decodes output from external applications into .NET strings, based on the encoding stored in [Console]::OutputEncoding, which defaults to the OEM code page. Therefore, when it sees the ANSI byte representation of character é, 0xe9, in the output, it interprets it as the OEM representation, where it represents character Θ (the assumption is that the active ANSI code page is Windows-1252, and the active OEM code page IBM437, as is the case in US-English systems, for instance).
You can verify this as follows:
# 0xe9 is "é" in the (Windows-1252) ANSI code page, and coincides with *Unicode* code point
# U+00E9; in the (IBM437) OEM code page, 0xe9 represents "Θ".
PS> $oemEnc = [System.Text.Encoding]::GetEncoding([int] (Get-ItemPropertyValue HKLM:\SYSTEM\CurrentControlSet\Control\Nls\CodePage OEMCP));
$oemEnc.GetString([byte[]] 0xe9)
Θ # Greek capital letter theta
Note that the decoding to .NET strings (System.String) that invariably happens means that the characters are stored as UTF-16 code units in memory, essentially as [uint16] values underlying the System.Char instances that make up a .NET string. Such a code unit encodes a Unicode character either in full, or - for characters outside the so-called BMP (Basic Multilingual Plane) - half of a Unicode character, as part of a so-called surrogate pair.
In the case at hand this means that the Θ character is stored as a different code point, namely a Unicode code point: Θ (Greek capital letter theta, U+0398).
Solution:
Note: A simple way to solve the problem is to activate system-wide support for UTF-8 (available in Windows 10), which sets both the ANSI and the OEM code page to 65001, i.e. UTF-8. However, this feature is (a) still in beta as of this writing and (b) has far-reaching consequences - see this answer for details.
However, it amounts to the most fundamental solution, as it also makes cross-platform Mosquitto use work properly (on Unix-like platforms, Mosquitto uses UTF-8).
PowerShell must be instructed what character encoding to use in this case, which can be done as follows:
PS> $msg = & {
# Save the original console output encoding...
$prevEnc = [Console]::OutputEncoding
# ... and (temporarily) set it to the active ANSI code page.
# Note: In *Windows PowerShell* - only - [System.TextEncoding]::Default work as the RHS too.
[Console]::OutputEncoding = [System.Text.Encoding]::GetEncoding([int] (Get-ItemPropertyValue HKLM:\SYSTEM\CurrentControlSet\Control\Nls\CodePage ACP))
# Now PowerShell will decode mosquitto_sub's output correctly.
mosquitto_sub -h test.mosquitto.org -t tofol/test
# Restore the original encoding.
[Console]::OutputEncoding = $prevEnc
}; $msg
{ "label": "eé" } # OK
Note: The Get-ItemPropertyValue cmdlet requires PowerShell version 5 or higher; in earlier version, either use [Console]::OutputEncoding = [System.TextEncoding]::Default or, if the code must also run in PowerShell (Core), [Console]::OutputEncoding = [System.Text.Encoding]::GetEncoding([int] (Get-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Control\Nls\CodePage ACP).ACP)
Helper function Invoke-WithEncoding can encapsulate this process for you. You can install it directly from a Gist as follows (I can assure you that doing so is safe, but you should always check):
# Download and define advanced function Invoke-WithEncoding in the current session.
irm https://gist.github.com/mklement0/ef57aea441ea8bd43387a7d7edfc6c19/raw/Invoke-WithEncoding.ps1 | iex
The workaround then simplifies to:
PS> Invoke-WithEncoding -Encoding Ansi { mosquitto_sub -h test.mosquitto.org -t tofol/test }
{ "label": "eé" } # OK
A similar function focused on diagnostic output is Debug-NativeInOutput, discussed in this answer.
As an aside:
While PowerShell isn't the problem here, it too can exhibit problematic character-encoding behavior.
GitHub issue #7233 proposes making PowerShell (Core) windows default to UTF-8 to minimize encoding problems with most modern command-line programs (it wouldn't help with mosquitto_sub, however), and this comment fleshes out the proposal.
[1] Note that Python too exhibits this nonstandard behavior, but it offers UTF-8 encoding as an opt-in, either by setting environment variable PYTHONUTF8 to 1, or via the v3.7+ CLI option -X utf8 (must be specified case-exactly!).
I have a source csv file which is quite big and in order to be able to work more efficiently with it I decided to split it into smaller file chunks. In order to do that, I execute the following script:
Get-Content C:\Users\me\Desktop\savedDataframe.csv -ReadCount 250000 | %{$i++; $_ | Out-File C:\Users\me\Desktop\Processed\splitfile_$i.csv}
As you can see, these are csv files which contain alphanumeric data. So, I have an issue with strings similar to this one:
Hämeenkatu 33
In the target file it looks like this:
Hämeenkatu 33
I've tried to determine the encoding of the source file and it is UTF-8 (as described here). I am really wondering why it gets so messed up in the target. I've also tried the following to explicitly tell that I want the encoding to be UTF8 but without success:
Get-Content C:\Users\me\Desktop\savedDataframe.csv -ReadCount 250000 | %{$i++; $_ | Out-File -Encoding "UTF8" C:\Users\me\Desktop\Processed\splitfile_$i.csv}
I am using a Windows machine running Windows 10.
Does the input file have a bom? Try get-content -encoding utf8. Out-file defaults to utf16le or what windows and powershell call "unicode".
Get-Content -encoding utf8 C:\Users\me\Desktop\savedDataframe.csv -ReadCount 250000 |
%{$i++; $_ |
Out-File -encoding utf8 C:\Users\me\Desktop\Processed\splitfile_$i.csv}
The output file will have a bom unless you use powershell 6 or 7.
js2010's answer provides an effective solution; let me complement it with background information (a summary of the case at hand is at the bottom):
Fundamentally, PowerShell never preserves the character encoding of a [text] input file on output:
On reading, file content is decoded into .NET strings (which are internally UTF-16 code units):
Files with a BOM for the following encodings are always correctly recognized (identifiers recognized by the -Encoding parameter of PowerShell's cmdlets in parentheses):
UTF-8 (UTF8) - info
UTF-16LE (Unicode) / UTF-16BE (BigEndianUnicode) - info
UTF-32LE (UTF32) / UTF-32BE (BigEndianUTF32) - info
Note the absence of UTF-7, which, however, is rarely used as an encoding in practice.
Without a BOM, a default encoding is assumed:
PowerShell [Core] v6+ commendably assumes UTF-8.
The legacy Windows PowerShell (PowerShell up to v5.1) assumes ANSI encoding, i.e the code page determined by the legacy system locale; e.g., Windows-1252 on US-English systems.
The -Encoding parameter of file-reading cmdlets allows you to specify the source encoding explicitly, but note that the presence of a (supported) BOM overrides this - see below for what encodings are supported.
On writing, .NET strings are encoded based on a default encoding, unless an encoding is explicitly specified with -Encoding (the .NET strings created on reading carry no information about the encoding of the original input file, so it cannot be preserved):
PowerShell [Core] v6+ commendably uses BOM-less UTF-8.
The legacy Windows PowerShell (PowerShell up to v5.1) regrettably uses various default encodings, depending on the specific cmdlet / operator used.
Notably, Set-Content defaults to ANSI (as for reading), and Out-File / > defaults to UTF-16LE.
See this answer for the full picture.
As noted in js2010's answer, using -Encoding UTF8 in Windows PowerShell invariably creates files with a BOM, which can be problematic for files read by tools on Unix-like platforms / tools with a Unix heritage, which are often not equipped to deal with such a BOM.
See the answers to this question for how to create BOM-less UTF-8 files in Windows PowerShell.
As with reading, the -Encoding parameter of file-writing cmdlets allows you to specify the output encoding explicitly:
Note that in PowerShell [Core] v6+, in addition to its defaulting to BOM-less UTF-8, -Encoding UTF8 too refers to the BOM-less variant (unlike in Windows PowerShell), and there you must use -Encoding UTF8BOM in order to create a file with BOM.
Curiously, as of PowerShell [Core] v7.0, there is no -Encoding value for the system's active ANSI code page, i.e. for Windows PowerShell's default (in Windows PowerShell, -Encoding Default explicitly request ANSI encoding, but in PowerShell [Core] this refers to BOM-less UTF-8). This problematic omission is discussed in this GitHub issue. By contrast, targeting the active OEM code page with -Encoding OEM still works.
In order to create UTF-32BE files, Windows PowerShell requires identifier BigEndianUtf32; due to a bug in PowerShell [Core] as of v7.0, this identifier isn't supported, but you can use UTF-32BE instead.
Windows PowerShell is limited to those encodings listed in the Microsoft.PowerShell.Commands.FileSystemCmdletProviderEncoding enumeration, but PowerShell [Core] allows you to pass any of the supported .NET encodings to the -Encoding parameter, either by code-page number (e.g., 1252) or by encoding name (e.g., windows-1252); [Text.Encoding]::GetEncodings().CodePage and [Text.Encoding]::GetEncodings().Name enumerate them in principle, but note that due to lack of .NET Core API support as of v7.0 this enumeration lists only a small subset of the actually supported encodings; running these commands in Windows PowerShell will show them all.
You can create UTF-7 files (UTF7), but they won't have a BOM; even input files that do have one aren't automatically recognized on reading, so specifying -Encoding UTF7 is always necessary for reading UTF-7 files.
In short:
In PowerShell, you have to know an input file's encoding in order to match that encoding on writing, and specify that encoding explicitly via the -Encoding parameter (if it differs from the default).
Get-Content (without -Encoding) provides no information as to what encoding it detected via a BOM or which one it assumed in the absence of a BOM.
If needed, you can perform your own analysis of the opening bytes of a text file to look for a BOM, but note that in the absence of one you'll have to rely on heuristics to infer the encoding - that is, you can make a reasonable guess, but you cannot be certain.
Also note that PowerShell, as of v7, fundamentally lacks support for passing raw byte streams through the pipeline - see this answer.
Your particular case:
Your problem was that your input file was UTF-8-encoded, but didn't have a BOM (which is actually preferable for the widest compatibility).
Since you're using Windows PowerShell, which misinterprets such files as ANSI-encoded, you need to tell it to read the file as UTF-8 with -Encoding Utf8.
As stated, on writing -Encoding Utf8 inevitably creates a file with BOM in Windows PowerShell; if that is a concern, use the .NET framework directly to produce a BOM-less files, as shown in the answers to this question.
Note that you would have had no problem with your original command in PowerShell [Core] v6+ - it defaults to BOM-less UTF-8 both on reading and writing, across all cmdlets.
This sensible, standardized default alone is a good reason for considering the move to PowerShell v7.0, which aims to be a superior replacement for the legacy Windows PowerShell.
I'm putting together a script and need to take a file's content as input for setting a variable. I'm using Out-File to produce a text file:
$string | Out-File -FilePath C:\Full\Path\To\file.txt -NoNewLine
Then I am using that file to set a variable in batch:
set /P variablename=<C:\Full\Path\To\file.txt
The content of that file is a unique id string that looks practically like this:
1i32l54bl5b2hlthtl098
When I echo this variable, I get this:
echo %variablename%
■1
When I have tried a different string in the input file, I see that what is being echoed is the ■ character and then the first character in the string. So, if my string was "apfvuu244ty0vh" then it would echo "■a" instead.
Why isn't the variable being set to the content of the file? I'm using the method from this stackoverflow post where the chosen answer says to use this syntax with the set command. Am I doing something wrong? Is there perhaps a problem with using a full path as input to a set variable?
tl;dr:
Use Out-File -Encoding oem to produce files that cmd.exe reads correctly.
This effectively limits you to the 256 characters available in the legacy "ANSI" / OEM code pages, except NUL (0x0). See bottom section if you need full Unicode support.
In Windows PowerShell (but not PowerShell Core), Out-File and its effective alias > default to UTF-16LE character encoding, where most characters are represented as 2-byte sequences; for characters in the ASCII range, the 2nd byte of each sequence is NUL (0x0); additionally, such files start with a BOM that indicates the type of encoding.
By contrast, cmd.exe expects input to use the legacy single-byte OEM encoding (note that starting cmd.exe with /U only controls the encoding of its output).
When cmd.exe (unbeknownst to it) encounters UTF-16LE input:
It interprets the bytes individually as characters (even though characters in UTF-16LE are composed of 2 bytes (typically), or, in rare cases, of 4 (a pair of 2-byte sequences)).
It interprets the 2 bytes that make up the BOM (0xff, 0xfe) as part of the string. With OEM code page 437 (US-English) in effect, 0xff renders like a space, whereas 0xfe renders as ■.
Reading stops once the first NUL (0x0 byte) is encountered, which happens with the 1st character from the ASCII range, which in your sample string is 1.
Therefore, string 1i32l54bl5b2hlthtl098 encoded as UTF-16LE is read as ■1, as you state.
If you need full Unicode support, use UTF-8 encoding:
Use Out-File -Encoding utf8 in PowerShell.
Before reading the file in cmd.exe (in a batch file), run chcp 65001 in order to switch to the UTF-8 code page.
Caveats:
Not all Unicode chars. may render correctly, depending on the font used in the console window.
Legacy applications may malfunction with code page 65001 in effect, especially on older Windows versions.
A possible strategy to avoid problems is to temporarily switch to code page 65001, as needed, and then switch back.
Note that the above only covers communication via files, and only in one direction (PowerShell -> cmd.exe).
To also control the character encoding used for the standard streams (stdin, stdout, stderr), both when sending strings to cmd.exe / external programs and when interpreting strings received from them, see this answer of mine.