Powershell and German umlauts - windows

I have written a script for my MySQL database with the Powershell ISE, which creates a backup for me with the help of the MySQL dump tool.
When I run the script in the PowerShell ISE, everything works. If I execute the same script now in the normal PowerShell, he does not show me the German umlauts correctly.
Here is my script:
# delete everything older than 30 days
foreach ($ordner in (ls D:\Backup -Depth 0))
{
if ($ordner.LastWriteTime.Date -lt (Get-Date).AddDays(-30).Date)
{
rm -Recurse ("D:\Backup\"+ $ordner.Name)
}
}
mkdir ("D:\Backup\" + (Get-Date -Format "yyyy_MM_dd") + "\Datenbank")
C:\xampp\mysql\bin\mysqldump.exe -uroot --default-character-set=latin1 --opt MY_DATABASE > ("D:\Backup\"+(Get-Date -Format "yyyy_MM_dd") + "\Datenbank\backup.sql")
However, I need the normal PowerShell for the automated execution and the script.
How can I fix the problem that the German umlauts are displayed in the normal PowerShell correctly?

The Windows PowerShell ISE differs from regular (conhost.exe) PowerShell console windows in that it interprets the output from external programs (such as mysqldump.exe) as encoded based on the active ANSI code page (e.g., Windows-1252 on US-English systems) - which is what --default-character-set=latin1 in your command requests.
Note: The PowerShell ISE is no longer actively developed and there are reasons not to use it (bottom section), notably not being able to run PowerShell [Core] 6+. The actively developed editor that offers the best PowerShell development experience, across platforms, is Visual Studio Code, combined with its PowerShell extension.
By contrast, regular PowerShell console windows default to the active OEM code page (e.g., 437 on US-English systems).
It is the encoding reported by [console]::OutputEncoding that determines how PowerShell interprets the output from external programs (though for mere display output that may not matter).
Therefore, you have two options:
Adjust the --default-character-set option to match the code page reported by [console]::OutputEncoding - assuming MYSQL supports it (this documentation suggests that the US-English OEM code page 437 is not supported, for instance).
Adjust [console]::OutputEncoding to (temporarily) match the specified --default-character-set option:
[console]::OutputEncoding = [System.Text.Encoding]::GetEncoding(1252)
In general, it is (Get-Culture).TextInfo.ANSICodePage / (Get-Culture).TextInfo.OEMCodePage that reports the a given system's active ANSI / OEM code page number.

Related

Windows 10 PowerShell cmdlet's doesn't allow UTF-8 strings as parameters [duplicate]

$logstring = Invoke-Command -ComputerName $filesServer -ScriptBlock {
param(
$logstring,
$grp
)
$Klassenbuchordner = "KB " + $grp.Gruppe
$Gruppenordner = $grp.Gruppe
$share = $grp.Gruppe
$path = "D:\Gruppen\$Gruppenordner"
if ((Test-Path D:\Dozenten\01_Klassenbücher\$Klassenbuchordner) -eq $true)
{$logstring += "Verzeichnis für Klassenbücher existiert bereits"}
else {
mkdir D:\Dozenten\01_Klassenbücher\$Klassenbuchordner
$logstring += "Klassenbuchordner wurde erstellt!"
}} -ArgumentList $logstring, $grp
My goal is to test the existence of a directory and create it on demand.
The problem is that the path contains German letters (umlauts), which aren't seen correctly by the target server.
For instance, the server receives path "D:\Dozent\01_Klassenbücher" instead of the expected "D:\Dozent\01_Klassenbücher".
How can I force proper UTF-8 encoding?
Note: Remoting and use of Invoke-Command are incidental to your problem.
Since the problem occurs with a string literal in your source code (...\01_Klassenbücher\...), the likeliest explanation is that your script file is misinterpreted by PowerShell.
In Windows PowerShell, if your script file is de facto UTF-8-encoded but lacks a BOM, the PowerShell engine will misinterpret any non-ASCII-range characters (such as ü) in the script.[1]
Therefore: Re-save your script as UTF-8 with BOM.
Note:
A UTF-8 BOM is no longer strictly necessary in the install-on-demand, cross-platform PowerShell (Core) 7+ edition (which consistently defaults to (BOM-less) UTF-8), but continues to be required if you want your scripts to work in both PowerShell editions.
Why you should save your scripts as UTF-8 with BOM:
Visual Studio Code and other modern editors create UTF-8 files without BOM by default, which is what causes the problem in Windows PowerShell.
By contrast, the PowerShell ISE creates "ANSI"-encoded[1] files, which Windows PowerShell - but not PowerShell Core - reads correctly.
You can only get away with "ANSI"-encoded files:
if your scripts will never be run in PowerShell Core - where all future development effort will go.
if your scripts will never run on a machine where a different "ANSI" code page is in effect.
if your script doesn't contain characters - e.g., emoji - that cannot be represented with your "ANSI" code page.
Given these limitations, it's safest - and future-proof - to always create PowerShell scripts as UTF-8 with BOM.
(Alternatively, you can use UTF-16 (which is always saved with a BOM), but that bloats the file size if you're primarily using ASCII/"ANSI"-range characters, which is likely in PS scripts).
How to make Visual Studio Code create UTF-8 files with-BOM for PowerShell scripts by default:
Note: The following is still required as of v1.11.0 of the PowerShell extension for VSCode, but not that there's a suggestion to make the extension default PowerShell files to UTF-8 with BOM on GitHub.
Add the following to your settings.json file (from the command palette (Ctrl+Shift+P, type settings and select Preferences: Open Settings (JSON)):
"[powershell]": {
"files.encoding": "utf8bom"
}
Note that the setting is intentionally scoped to PowerShell files only, because you wouldn't want all files to default to UTF-8 with BOM, given that many utilities on Unix platforms neither expect nor know how to handle such a BOM.
[1] In the absence of a BOM, Windows PowerShell defaults to the encoding of the system's current "ANSI" code page, as determined by the legacy system locale; e.g., in Western European cultures, Windows-1252.

German Umlauts in Powershell called by vbs

i do have a ps1 file, which create a Link
create-link.ps1
$path = $env:HOMESHARE + "\My Projects\"
If(!(test-path $path))
{
New-Item -ItemType Directory -Force -Path $path
}
$WshShell = New-Object -comObject WScript.Shell
$Shortcut = $WshShell.CreateShortcut($env:HOMESHARE + "\My Projects\" + "linkname.lnk")
$Shortcut.TargetPath = "\\path\for\link"
$Shortcut.Description = "äöüß"
$Shortcut.IconLocation = $env:SYSTEMROOT + "\\system32\\shell32.dll,3"
$Shortcut.Save()
I also do have a vbs file which calls the ps1
create-link.vbs
command = "powershell.exe Get-Content ""C:\path\to\file\create-link.ps1"" | PowerShell.exe -noprofile"
set shell = CreateObject("WScript.Shell")
shell.Run command,0
Both files are saved with utf-8 encoding.
This construction was necessary, because the ps1 needed to run completly headless without any noticable things for the user. Calling a ps1 through a vbs solved this problem, if there is a better way i would be happy if you let me know.
If i am calling the powershell script directly or with "powershell.exe Get-Content ""C:\path\to\file\create-link.ps1"" | PowerShell.exe -noprofile" (by using cmd) everything works fine.
However, if i call the vbs to do the work it works in general, but the german umlauts from 'Description' are just questions marks, so somehow the encoding got scrambled. Is there any way to fix this?
tl;dr:
Save your *.ps1 file as UTF-8 with BOM.
Simplify your command by using the PowerShell CLI's -File parameter:
command = "powershell.exe -NoProfile -File ""C:\path\to\file\create-link.ps1"""
See also: GitHub issue #3028, which requests the ability to launch PowerShell itself completely hidden - obviating the need for an aux. VBScript script - which a future version may support (but it won't be back-ported to Windows PowerShell).
If you're using Windows PowerShell (versions up to v5.1), you must save your *.ps1 files as UTF-8 with a BOM in order for them to be interpreted correctly with respect to characters outside the ASCII (7-bit) range, such as äöüß.
This is no longer necessary in PowerShell [Core] v6+, which consistently defaults to UTF-8, but if your scripts need to run in both editions, you should always use UTF-8 with BOM.
If a given *.ps1 doesn't have a BOM, Windows PowerShell interprets each byte that is part of an UTF-8 encoding sequence (all non-ASCII characters are encoded as 2-4 bytes) individually as a character, based on the system's active ANSI code page (a single-byte encoding such as Windows-1252).
On a US-English system, where the active ANSI code page is Windows-1252, the above sample string therefore surfaces as garbage string äöüß
Note that question marks, or, more accurately, instances of � (REPLACEMENT CHARACTER, U+FFFD), would only surface in the reverse scenario: when ANSI-encoded text is misinterpreted as UTF-8.
As an aside, re your approach of providing the source code to the PowerShell CLI via the pipeline (stdin):
Since your script apparently runs hidden, it won't make a difference in your case, but note that this technique exhibits pseudo-interactive mode and also doesn't support passing arguments to the script being provided via stdin - see GitHub issue #3223

Convert localized path to the English

if user installs windows in his native language(not english) he can access files using localized paths ie. insted C:\Users\UserName\Desktop he can access Desktop using C:\LocalizedUsersName\LocalizedDesktopName Is there any Powershell command which can convert/translate local version of the Path to the English version?
There is nothing specifically for a path, yet, basically, along the same lines as you'd do this for other strings you are working with. See the discussions, examples, and answers below.
International Settings Cmdlets in Windows PowerShell
Using the Microsoft Translator API from PowerShell
Localization and PowerShell
plattsoft_PSUICultureExample
Forcing PowerShell errors output in English on localized systems
You can change the pipeline thread's CurrrentUICulture like so:
[Threading.Thread]::CurrentThread.CurrentUICulture = 'fr-FR'; Get-Help
Get-Process
I'm on an English system but before I executed the line above, I
updated help like so: Update-Help -UICulture fr-FR
$ENUS="[Threading.Thread]::CurrentThread.CurrentUICulture = 'en-US'"
then invoke-expression $ENUS; Get-Help Get-Process "invoke-expression
$ENUS;" is little shorter if you need to run many commands

How to make win32 console recognize ANSI/VT100 escape sequences?

I'm building a lightweight version of the ncurses library. So far, it works pretty well with VT100-compatible terminals, but win32 console fails to recognise the \033 code as the beginning of an escape sequence:
# include <stdio.h>
# include "term.h"
int main(void) {
puts(BOLD COLOR(FG, RED) "Bold text" NOT_BOLD " is cool!" CLEAR);
return 0;
}
What needs to be done on the C code level, in order that the ANSI.SYS driver is loaded and the ANSI/VT100 escape sequences recognized?
[UPDATE] For latest Windows 10 please read useful contribution by #brainslugs83, just below in the comments to this answer.
While for versions before Windows 10 Anniversary Update:
ANSI.SYS has a restriction that it can run only in the context of the MS-DOS sub-system under Windows 95-Vista.
Microsoft KB101875 explains how to enable ANSI.SYS in a command window, but it does not apply to Windows NT. According to the article: we all love colors, modern versions of Windows do not have this nice ANSI support.
Instead, Microsoft created a lot of functions, but this is far from your need to operate ANSI/VT100 escape sequence.
For a more detailed explanation, see the Wikipedia article:
ANSI.SYS also works in NT-derived systems for 16-bit legacy programs executing under the NTVDM.
The Win32 console does not natively support ANSI escape sequences at all. Software such as Ansicon can however act as a wrapper around the standard Win32 console and add support for ANSI escape sequences.
So I think ANSICON by Jason Hood is your solution. It is written in C, supports 32-bit and 64-bit versions of Windows, and the source is available.
Also I found some other similar question or post which ultimately have been answered to use ANSICON:
How to load ANSI escape codes or get coloured file listing in WinXP cmd shell?
how to use ansi.sys in windows 7
How can I get cmd.exe to display ANSI color escape sequences?
ansi color in windows shells
enable ansi colors in windows command prompt
Starting from Windows 10 TH2 (v1511), conhost.exe and cmd.exe support ANSI and VT100 Escape Sequences out of the box (although they have to be enabled).
See my answer over at superuser for more details.
Base on #BrainSlugs83 you can activate on the current Windows 10 version via register, with this command line:
REG ADD HKCU\CONSOLE /f /v VirtualTerminalLevel /t REG_DWORD /d 1
For Python 2.7 the following script works for me fine with Windows 10 (v1607)
import os
print '\033[35m'+'color-test'+'\033[39m'+" test end"
os.system('') #enable VT100 Escape Sequence for WINDOWS 10 Ver. 1607
print '\033[35m'+'color-test'+'\033[39m'+" test end"
Result should be:
[35mcolor-test[39m test end
color-test test end
Starting from Windows 10, you can use ENABLE_VIRTUAL_TERMINAL_PROCESSING to enable ANSI escape sequences:
https://msdn.microsoft.com/en-us/library/windows/desktop/mt638032(v=vs.85).aspx
If ANSICON is not acceptable since it requires you to install something on the system, a more lightweight solution that parses and translates the ANSI codes into the relevant Win32 API console functions such as SetConsoleTextAttribute.
https://github.com/mattn/ansicolor-w32.c
For coloring the cmd you need Windows.h and use SetConsoleTextAttribute() more details can be found in http://msdn.microsoft.com/en-us/library/windows/desktop/ms686047%28v=vs.85%29.aspx
In lastest win10, it can be done by SetConsoleMode(originMode | ENABLE_VIRTUAL_TERMINAL_PROCESSING). See https://learn.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences#example
Maybe ANSICON can help u
Just download and extract files, depending on your windows os: 32bit or 64bit
Install it with: ansicon -i
I personally like clink. It not only processes ANSI codes, it also adds many other features so Windows Console behaves like bash (history, reverse history search, keyboard shortcuts, etc.):
The same line editing as Bash (from GNU's Readline library).
History persistence between sessions.
Context sensitive completion;
Executables (and aliases).
Directory commands.
Environment variables
Thirdparty tools; Git, Mercurial, SVN, Go, and P4.
New keyboard shortcuts;
Paste from clipboard (Ctrl-V).
Incremental history search (Ctrl-R/Ctrl-S).
Powerful completion (TAB).
Undo (Ctrl-Z).
Automatic "cd .." (Ctrl-PgUp).
Environment variable expansion (Ctrl-Alt-E).
(press Alt-H for many more...)
Scriptable completion with Lua.
Coloured and scriptable prompt.
Auto-answering of the "Terminate batch job?" prompt.
Ansi.sys (in the system32 folder) is an "MSDOS driver" provided as part of Windows XP, 2000, and earlier versions of NT. In 2000 and XP, it is located in the system32 folder (I don't remember the structure of earlier versions of NT). Programs that run in the DOS subsystem and use standard output can use ANSI.SYS just as they could running over MSDOS.
To load ansi.sys, you must use the device= or devicehigh= command in config, just as you would in MSDOS. On Windows NT 5 (2K & XP), each copy of the DOS subsystem can be given a separate config file in the pif/shortcut (use the "advanced" button), and there is a default file called CONFIG.NT (also in the system32 folder), which is used if the pif/shortcut does not specify a special config file.
When ansi.sys is loaded correctly, mem /d will report that it is loaded. On earlier versions of NT, you can and must load a proper DOS environment to load ansi.sys, and ansi art will work at the prompt. On Win 2K and XP, loading ansi.sys will have no effect on your "CMD prompt" because CMD is not a DOS program: it is a 32 bit Windows console program. For some reason that I do not understand, on WinXP, even if you load a fixed copy of command.com using "command.com /p", the command prompt will not be ansi enabled: perhaps when you do it that way it only emulates loading command.com?
In any case, when you use an actual DOS version of command.com, ansi is enabled after being loaded: you can demonstrate it's use with a bit of ansi art like this:
command /c type ansiart.ans
(here is an example: http://artscene.textfiles.com/ansi/artwork/beastie.ans)
CONFIG.NT (in the system32 folder) contains an example of the syntax for loading device drivers. You will need to be an Administrator to edit that default file, or you can make a copy of it.
On Win 2K and XP, the default "shortcut" for MSDOS is a .PIF file, not a .LNK file. If you create a .lnk file to CMD, you won't be able to set special config and autoexec files, it will use the default CONFIG.NT. If you want to use a special config file for just one DOS application, you can make a copy of the "MSDOS shortcut", or you can make a copy of "_default.pif", found in your Windows folder.
I found this tool to be working for my end.
Microsoft Color Tool from GitHub
Unzip the compressed file then open CMD with Administration permission.
Go to the folder where you unzip the file in CMD.
Then execute this command "colortool -b scheme-name"
The scheme-name needs to be replaced with any of these options below:
campbell.ini
campbell-legacy.ini
cmd-legacy.ini
deuternopia.itermcolors
OneHalfDark.itermcolors
OneHalfLight.itermcolors
solarized_dark.itermcolors
solarized_light.itermcolors
In my case, the command would be like this "colortool -b solarized_dark.itermcolors"
Click right on the console window and select Properties.
You don't need to change any value just click "OK" to save the setting. (You will notice that your font already contains colors).
Console Property
Then restart your cmd or powerShell.
The ANSI color should be enabled and working with the color scheme you chose before.
Had the same issue. I installed ConEmu and that one solved my problem.
Somehow in Windows you just need to call any shell command first, rather call the system function. Just in start of your main method put system("");, and don't forget to include stdlib.h.
I noticed this when I looked at some of my old programs that also used ANSI codes to understand why they work, but my new code is not

How to use cmd type pipe (/piping) in PowerShell?

In cmd (and bash), pipe "|" pushes output to another command in the original format of the first command's output (as string).
In PowerShell, everything that comes out the pipe is an object (even a string is a string object).
Because of that, some commands fail when run in a PowerShell command window as opposed to a Windows command window.
Example:
dir c:\windows | gzip > test.gz
When this command is run in the Windows command prompt window it works properly - directory listing of C:\windows gets compressed into test.gz file.
The same command in PowerShell fails, because PowerShell does not use cmd-style pipe and replaces it with PowerShell pipe (working with array of file system items).
Q. How do you disable the default piping behavior in PowerShell to make traditional Windows commands work identically in PowerShell?
I tried using the escape character "`" before the pipe "`|", but it didn't work. I also tried invoke-expression -command "command with | here", but it also failed.
if you want to send strings down the pipeline you can use the cmdlet "out-string"
For Example:
get-process | out-string
If you are specifically looking for a PowerShell way to zip up files, check out the PowerShell Community Extensions. there are a bunch of cmdlets to zip and unzip all kinds of files.
http://pscx.codeplex.com
If you can pipe the output of (CMD) dir into gzip, then gzip apparently knows how to parse dir output. The (string) output from the PowerShell dir command (aka Get-ChildItem) doesn't look the same, so gzip likely would not be able to parse it. But, I'd also guess that gzip would be happy to take a list of paths, so this would probably work:
dir c:\windows | select -ExpandProperty FullName | gzip > test.gz
No warrantees express or implied.
If you really need to use the old school DOS pipe system in PowerShell, it can be done by running a command in a separate, temporary DOS session:
& cmd /c "dir c:\windows | gzip > test.gz"
The /c switch tells cmd to run the command then exit. Of course, this only works if all the commands are old school DOS - you can't mix-n-match them with PowerShell commands.
While there are PowerShell alternatives to the example given in the question, there are lots of DOS programs that use the old pipe system and will not work in PowerShell. svnadmin load is one that I've the pleasure of having to deal with.
You can't. PowerShell was designed to pass objects down a pipeline, not text. There isn't a backwards-compatability mode to DOS.

Resources