Just a pretty stand curl command call an S3 end point for download using all default values. On a mac, or on a PC using command line I get 103MBsec if cached on cdn and 80mbsec otherwise. Same command, same bucket, same object, using "curl.exe" and I get 1MBSec when call through powershell. I guess powershell does something different that make it's totally slow? I tried using newest curl binary but still the same.
I guess I am misunderstanding what powershell is doing when I use a curl command
curl.exe yourfileonS3 >> output.bin
To complement briantist's helpful answer:
In PowerShell, the redirection operators > and >> are in effect aliases of Out-File and Out-File -Append.
> and >> are therefore not mere byte-stream conduits, and, in fact, PowerShell as of v7.2 does not support sending raw byte output to a file.
Instead, PowerShell invariably decodes output from any external program as text ([string] instances), based on the character encoding reported by [Console]::OutputEncoding] and then, on saving to the target file with Out-File (possibly via > / >>), re-encodes these strings, using that cmdlet's default character encoding (unless overridden with -Encoding in an explicit Out-File call).
Not only does this not preserve the external program's raw byte
output, it adds significant overhead.
To get raw byte processing, call cmd.exe[1] and use its redirection operators:
cmd /c 'curl.exe yourfileonS3 >> output.bin'
See this answer for more information.
[1] On Unix-like platforms, use sh -c 'curl yourfileonS3 >> output.bin'
See mklement0's answer for full context on this (I recommend accepting that one!), and the important point that handling of byte streams in redirection is problematic and error prone in PowerShell and should be avoided.
So I looked into this and I believe the reason is that >> (file redirection) is the slow part.
I originally suspected you might be calling curl (which is aliased to Invoke-WebRequest in Windows PowerShell), but I was able to reproduce the speed difference between curl.exe directly in PowerShell vs cmd.exe, and measure it, this way:
# call curl.exe and do redirection in PowerShell
Measure-Command -Expression { curl.exe https://uploader.codecov.io/v0.1.0_6943/linux/codecov >> delme.bin }
del delme.bin
# call cmd.exe and do redirection there
Measure-Command -Expression { & cmd.exe /c 'curl.exe https://uploader.codecov.io/v0.1.0_6943/linux/codecov >> delme.bin' }
del delme.bin
This was enough to show a stark difference.
I also confirmed that this problem is a little bit worse in Windows PowerShell as opposed to later cross-platform versions (pwsh.exe). In Windows, with version 7.1.0, the same commands above still show a large difference.
Related
I have a program that I work with, that has an onboard lua compiler to allow for custom written actions.
Since the tool itself is very limited, especially if it goes for complex reactions over networks, I want to use Powershell over lua.
Methods like os.execute() or io.popen() use the standard command line from windows and not Powershell.
Is there a way to use Powershell with lua?
I tried to write a command line script with the Powershell editor and run this script with os.execute, but it opens it as a textfile, it would be better to write the commands directly in lua but if there is no other way, executing a Powershell script directly would also be fine. (In Windows itself you can execute the script with right mouse "click/Execute with Powershell")
-- You can generate PowerShell script at run-time
local script = [[
Write-Host "Hello, World!"
]]
-- Now create powershell process and feed your script to its stdin
local pipe = io.popen("powershell -command -", "w")
pipe:write(script)
pipe:close()
Your description of the problem makes it sound like you're using a command such as os.execute("powershellscript.ps1"), and that call invokes cmd.exe with your string as the proposed command line. Normally, Windows will open a .PS1 file for editing; this was a deliberate decision for safety. Instead, try altering the os.execute() command to explicitly call PS: os.execute("powershell.exe -file powershellscript.ps1"). If you need to pass parameters to your script, enclose them in {}. See https://msdn.microsoft.com/en-us/powershell/scripting/core-powershell/console/powershell.exe-command-line-help for more info on invoking PowerShell from the command line.
Running Node.js from PowerShell on Windows assumes that stdout must be a UTF-16 text stream. This generates surprising results for tools which send binary Buffer objects to stdout. Is there a way to work around this?
Command-line example:
node -e "process.stdout.write(new Buffer('abc'))" >tmp.dat
Using PowerShell, the resulting file, tmp.dat is now 12 bytes long rather than only 3 bytes long.
The exact same example launched from CMD.EXE produces the expected 3-byte file.
P.S. I originally thought this was a node.js question, but its now clear it's a PowerShell issue.
The best answer for this seems to be to spawn cmd.exe and do native redirection with it, rather than depend on PowerShell's redirection model. It is a little odd as it requires putting the redirection in quotes, as follows:
cmd /c node -e "process.stdout.write(new Buffer('abc'))" '>tmp.dat'
It really seems too bad that PowerShell doesn't recognize that since "node" is a native command any redirection should be handled using the native redirection mechanisms.
If you open up a command prompt and type this:
echo foobar > nul
it will print nothing, since nul swallows all of its input. But if you run the command with PowerShell:
powershell "echo foobar" > nul
it will output foobar to the console. Why is this, and what can I do to fix it?
edit: Here is the output of $PSVersionTable. It looks like I'm using PowerShell v5.0.
Note: I'm assuming you're invoking your command from cmd.exe, not from within PowerShell, which is consistent with the symptoms I'm seeing.
Methinks you've stumbled upon a bug in PS (PowerShell) v5 (not present in v3; comments on the question suggest it's also not in v4), though I don't fully understand why PS is to blame, because I'd expect cmd.exe to handle the redirection.
I may be missing something, however, so do let me know.
PowerShell should send its so-called success stream - things output by default, including with echo, which is an alias of Write-Output - to the outside world's stdout.
In older PS versions >NUL does effectively suppresses PowerShell's output.
Curiously, the bug in v5 only affects NUL, whereas redirecting to an actual file works.
As for workarounds:
If your code is v2-compatible, try this:
powershell -version 2 "echo foobar" > NUL
Otherwise, redirect to an actual file and delete that file afterward:
powershell "echo foobar" > "%TEMP%\NUL-bug-workaround" & del "%TEMP%\NUL-bug-workaround"
I have a PowerShell script that produces a text file. At the end, I would like to copy this file to a Linux server.
From CMD.EXE, I can use PSCP (from Putty), it works and copies the file.
But from PowerShell, either interactively or from a PowerShell batch, PSCP has no visible effect: no error messages and the file is not copied.
Even if I run simply .\PSCP.EXE without arguments, on the CMD command line it displays the options, but from PowerShell it does nothing.
Can PSCP be used from inside PowerShell?
Executing a program from within PowerShell should work identically to CMD, but depending upon how that program produces its output (does it write to STDOUT, STDERR, other?) that may behave differently.
I've been using Rebex's components for FTPS & SFTP within .NET apps & PowerShell scripts; the SFTP package includes an SCP class. Yes, it costs money, but depending upon your usage it may be worthwhile.
Just attempted to automate PSCP from PowerShell. Remember to use pscp's -batch parameter so that, should you do something like enter the wrong password, you won't get asked for input.
$Cmd = "pscp -l username -pw password -batch c:\folder\file.txt server:/home/user1"
Invoke-Expression "& $( $Cmd )"
Otherwise your script will just grind to a halt.
Yes - most any executable can be called from PowerShell. There isn't anything peculiar about pscp.exe in this regard. You may need to preface it with the call operator - the ampersand - &:
PS C:\>& "C:\Program Files (x86)\Putty\pscp.exe" -V
pscp: Release 0.62
The above is direct output from my PowerShell prompt. The call operator is particularly helpful if the path to your executable contains spaces - the call operator is used to tell PowerShell to treat what would be considered a string as something it should try to execute instead.
Please include the full command your are trying to execute as it will help in providing a better answer. You may have a problem with your PATH variable or something else weird if you don't get any output.
If using pscsp from inside a script, e.g. perl
no ampersand
quote like this "my password"
e.g.
"C:\Program Files\Putty\pscp.exe" -C -p -pw "password" /local_dir/file_to_copy user#hostname:/remote_directory
in perl (beware that \ is an escape char in a "string" )
$cmd = q("C:\Program Files\Putty\pscp.exe" -C -p -pw "password" /local_dir/file_to_copy user#hostname:/remote_directory);
system($cmd);
Just tests a very simple command like:
while true; do bash -c "echo hello"; done
You will find how much slow the bash in Cygwin is. Does anybody know why?
It's a fresh install of cygwin 1.7 on win7.
thanks to Jared's testing idea, I modified the command to this(adds bash -c):
time for i in {1..10} ; do bash -c "echo Hello" ; done
Hello
...
real 0m7.711s //it's the problem
user 0m0.091s
sys 0m0.476s
How about excluding Cygwin paths from your antivirus software ?
Check your path. Referring a non-existant path or a very slow network share can cause the symptoms you are describing.
Below are 3 possible causes which I encountered. Of course, other problems not mentioned can also cause slow shell commands in Cygwin.
If you have the option "Automatically detect settings" in "LAN Settings", Windows will use the WPAD protocol to discover the local HTTP proxy. First it will send a DHCP "Inform" request with option 252, then it will try a DNS lookup on "wpad". These 2 operations can take a few seconds to time-out.
If the shell accesses some paths like /cygdrive/... , a NetBIOS name query will be executed, which can also take some time to time out.
If the shell accesses some paths like //mypath/... , a NetBIOS name query will be executed, which can also take some time to time out.
Solutions :
Disable "Automatically detect settings" in "LAN Settings" in the Windows "Internet Options" control panel.
Add the following entry in %SystemRoot%\system32\drivers\etc\hosts :
127.0.0.1 localhost cygdrive wpad
Make sure to avoid double slashes at the beginning of all paths.
Finally, I found the source - a service named "QQPCMgr RTP Service" running on my office computer, it's the real time protection service of "QQ PC Manager".
By disabling it, the time of the script in the question falls back to:
real 0m0.943s
user 0m0.105s
sys 0m0.231s
I have told the developers of QQPCMgr about this, hope they will find the reason.
This still much slower than Linux, but gets the same "real time" of other cygwin computers.
Thank you all!
This is how this problem manifested for me. For the first say 20 commands entered at the cygwin prompt it is fast, then it abruptly becomes painfully slow.
I checked that every item in my path was a valid directory.
I tried an authentication fix suggested elsewhere -
https://superuser.com/questions/877051/cygwin-from-windows-very-slow-laggy/1247517 .
It didn't work for me.
I replaced my exisitng virus scanner (System Center Endpoint Protection) with Sophos. That fixed the problem for me. Hope this helps.
I voted up on James McLeod, because starting a bash process takes some time, but it doesn't mean it will run commands slower than in UNIX.
Invoking bash -c from within a bash script is near to senseless, and Makefiles can call a lot of bash subprocesses, unless you append ; \ at the end of the commands.
For example, if a Makefile has the following:
echo Hello World
echo Good Bye
It will call two bash processes. To make it faster and call just one bash process:
echo Hello World; \
echo Good Bye
Debian has adopted dash instead of bash as the main shell, because starting many init scripts using bash will make the system take much longer to boot (each script call its own bash process).
Let's put some numbers to it. I ran the following:
time for i in {1..1000} ; do echo "Hello" ; done
The result I get from a standard Cygwin bash window is:
...
Hello
Hello
real 0m0.584s
user 0m0.031s
sys 0m0.000s
And from a xterm bash window on the same system I get:
...
Hello
Hello
real 0m0.037s
user 0m0.016s
sys 0m0.000s
I think this pretty much answers the question for you. The problem is you're going through a "Windows" "cmd" like window, which is inherently slow. Cygwin itself isn't the problem, it's the display trying to keep up that is slowing things down (for this test).
I was able to fix this problem by uninstalling bash-completion and switching from Kaspersky to Windows Security Essentials. (I diagnosed the Kaspersky interference using Process Explorer while running the "echo Hello" benchmark.) Brought my benchmark time from 7 seconds to 0.2.
On Comodo Internet Security Premium 10, i added "C:\Users\\Documents\MobaXterm\slash\bin" to the list of trusted files. (Settings -> File rating -> File list). Local terminal is fast again now.
For Windows Defender:
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows Defender]
"DisableAntiSpyware"=dword:00000001
[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows Defender\Real-Time Protection]
"DisableBehaviorMonitoring"=dword:00000001
"DisableOnAccessProtection"=dword:00000001
"DisableScanOnRealtimeEnable"=dword:00000001
or less dramatically:
powershell -Command Add-MpPreference -ExclusionPath "C:\opt\cygwin64"
powershell -Command Add-MpPreference -ExclusionPath "C:\home"
powershell -Command Add-MpPreference -ExclusionProcess "git.exe"
powershell -Command Add-MpPreference -ExclusionExtension ".c"
powershell -Command Add-MpPreference -ExclusionExtension ".cpp"
powershell -Command Add-MpPreference -ExclusionExtension ".cxx"
For Sophos:
REG ADD "HKLM\SYSTEM\CurrentControlSet\Services\Sophos Endpoint Defense\Scanning\Config" /v "OnAccessExcludeFilePaths" /t REG_MULTI_SZ /d "C:\Program Files (x86)\Sophos\Sophos Anti-Virus\\0C:\home\\0D:\Video\\" /f
REG ADD "HKLM\SYSTEM\CurrentControlSet\Services\Sophos Endpoint Defense\Scanning\Config" /v "OnAccessExcludeProcessPaths" /t REG_MULTI_SZ /d "C:\opt\cygwin64\\" /f
Invoking bash from a shell script can't be fast. Is it faster to just use
while true; echo hello; done
?