Running Node.js from PowerShell on Windows assumes that stdout must be a UTF-16 text stream. This generates surprising results for tools which send binary Buffer objects to stdout. Is there a way to work around this?
Command-line example:
node -e "process.stdout.write(new Buffer('abc'))" >tmp.dat
Using PowerShell, the resulting file, tmp.dat is now 12 bytes long rather than only 3 bytes long.
The exact same example launched from CMD.EXE produces the expected 3-byte file.
P.S. I originally thought this was a node.js question, but its now clear it's a PowerShell issue.
The best answer for this seems to be to spawn cmd.exe and do native redirection with it, rather than depend on PowerShell's redirection model. It is a little odd as it requires putting the redirection in quotes, as follows:
cmd /c node -e "process.stdout.write(new Buffer('abc'))" '>tmp.dat'
It really seems too bad that PowerShell doesn't recognize that since "node" is a native command any redirection should be handled using the native redirection mechanisms.
Related
Just a pretty stand curl command call an S3 end point for download using all default values. On a mac, or on a PC using command line I get 103MBsec if cached on cdn and 80mbsec otherwise. Same command, same bucket, same object, using "curl.exe" and I get 1MBSec when call through powershell. I guess powershell does something different that make it's totally slow? I tried using newest curl binary but still the same.
I guess I am misunderstanding what powershell is doing when I use a curl command
curl.exe yourfileonS3 >> output.bin
To complement briantist's helpful answer:
In PowerShell, the redirection operators > and >> are in effect aliases of Out-File and Out-File -Append.
> and >> are therefore not mere byte-stream conduits, and, in fact, PowerShell as of v7.2 does not support sending raw byte output to a file.
Instead, PowerShell invariably decodes output from any external program as text ([string] instances), based on the character encoding reported by [Console]::OutputEncoding] and then, on saving to the target file with Out-File (possibly via > / >>), re-encodes these strings, using that cmdlet's default character encoding (unless overridden with -Encoding in an explicit Out-File call).
Not only does this not preserve the external program's raw byte
output, it adds significant overhead.
To get raw byte processing, call cmd.exe[1] and use its redirection operators:
cmd /c 'curl.exe yourfileonS3 >> output.bin'
See this answer for more information.
[1] On Unix-like platforms, use sh -c 'curl yourfileonS3 >> output.bin'
See mklement0's answer for full context on this (I recommend accepting that one!), and the important point that handling of byte streams in redirection is problematic and error prone in PowerShell and should be avoided.
So I looked into this and I believe the reason is that >> (file redirection) is the slow part.
I originally suspected you might be calling curl (which is aliased to Invoke-WebRequest in Windows PowerShell), but I was able to reproduce the speed difference between curl.exe directly in PowerShell vs cmd.exe, and measure it, this way:
# call curl.exe and do redirection in PowerShell
Measure-Command -Expression { curl.exe https://uploader.codecov.io/v0.1.0_6943/linux/codecov >> delme.bin }
del delme.bin
# call cmd.exe and do redirection there
Measure-Command -Expression { & cmd.exe /c 'curl.exe https://uploader.codecov.io/v0.1.0_6943/linux/codecov >> delme.bin' }
del delme.bin
This was enough to show a stark difference.
I also confirmed that this problem is a little bit worse in Windows PowerShell as opposed to later cross-platform versions (pwsh.exe). In Windows, with version 7.1.0, the same commands above still show a large difference.
The windows API provides GetCommandLine() which returns the cooked command line used to start a process.
For instance, if a console application is started with the command:
MyProg.exe > OutputHere
The command line seen by MyProg.exe will not include the portion
> OutputHere
I'd like to somehow get the command line exactly as it was. Is this possible ? and if yes, how ?
Suggestions in C and/or plain (no objects) Delphi greatly appreciated. Creative solutions welcome (hopefully, not requiring ring 0 code.)
Thank you very much for your help.
NOTE: I can tell if the input/output, etc has been redirected but, that is not what I'm looking for. I need the original/uncooked command line.
The redirection or piping of stdin, stdout and stderr is handled the command interpreter, typically cmd.exe. The interpreter parses the command and creates the necessary files and pipes, and then creates the one or more processes needed to implement your command.
The processes that are created have no knowledge of the original command, they only get that part of the command that is not related to piping and redirection.
So what you are trying to do is not possible, at least within your process. The only thing that knows the original command is the command interpreter.
Whether or not you can retrieve the full command line including the pipe commands depends on whether your start the program in a command window or for example using the "Run" command from the Start menu. If you use the "Run" command from the Start menu GetCommandLine actually retrieves the full command line including the redirection commands, but redirection does not work as it seems to be a feature of CMD.EXE.
As others have pointed out, what are you trying to achieve here / why do you need to capture the redirection commands?
I want to send commands in the ADB shell itself as if i had done the following in cmd.
>adb shell
shell#:/ <command>
I am using python 3.4 on a windows 7 OS 64bit machine. I can send one-line shell commands simply using subprocess.getoutput such as:
subprocess.getoutput ('adb pull /storage/sdcard0/file.txt')
as long as the adb commands themselves are recognized by ADB specifically, such as pull and push, however there are other commands such as grep that need to be run IN the shell, like above, since they are not recognized by adb. for example, the following line will not work:
subprocess.getoutput ('adb shell ls -l | grep ...')
To enter the commands in the shell I thought I needed some kind of expect library as that is what 'everyone' suggests, however pexpect, wexpect, and winexpect all failed to work. they were written for python 2 and after being ported to python 3 and my going through the .py files by hand, even those tweaked for windows, nothing was working - each of them for different reasons.
how can i send the input i want to the adb shell directly?
If none of the already recommended shortcuts work for you you can still go the 'regular' way using 'subprocess.Popen' for entering commands in the adb shell with Popen:
cmd1 = 'adb shell'
cmd2 = 'ls -l | grep ...'
p = subprocess.Popen(cmd1.split(), stdin=PIPE)
time.sleep(1)
p.stdin.write(cmd2.encode('utf-8'))
p.stdin.write('\n'.encode('utf-8'))
p.stdin.flush()
time.sleep(3)
p.kill()
Some things to remember:
even though you import subprocess you still need to invoke subprocess.Popen
sending cmd1 as a string or as items in a list should work too but '.split()' does the trick and is easier on the eyes
since you only specidfied you want to enter input to the shell you only need stdin=PIPE. stdout would only be necessary if you wanted to receive output from the shell
time.sleep(1) isn't really necessary, however since many complained about input issues being faster or slower in python 2 vs 3 consider maybe using it. 'they' might have been using versions of 'expect' that need the shell's reply first. this code also worked when i tested it with simply swapping out and in the process with time.sleep(0)
stdin.write will return an error if the input is not encoded properly. python's default is unicode. entering by binary did not work for me in my tests like this "b\ls ..." but .encode() worked. dont forget the endline!
if you use .encode() there is a worry that the line might not get sent properly, so to be sure it might be good to include a flush().
time.sleep(3) is completely uneccesary, but if your command takes a long time to execute (eg a regressive search through the entire device piped out to a txt file on the memory card) maybe give it some extra time before killing anyhting.
remember to kill. if you didnt kill it, the pipe may remain open, and even after exiting the test app on the console the next commend still went to the shell even though the prompt appearsed to be my regular cmd prompt.
Amichai, I have to start with pointing out that your own "solution" is pretty awful. And your explanation makes it even worse. Doing all those unnecessary things just because you do not understand how shell (here I mean your PC's OS shell, not adb) command parsing works.
When all you needed was just this one command:
subprocess.check_output(['adb', 'shell', 'ls /storage/sdcard0 | grep ...']).decode('utf-8')
If I create a process from a cmd prompt using the start command (opening a new cmd) is it possible to redirect the stdout and stderr from that process back to the calling cmd?
If you want the output of the STARTed process to appear in the parent command console, then simply use the START /B option.
If you want to process the output of your command, then you should use FOR /F ... in ('someCommand') DO ... instead.
OK. I have yet to find a straightforward answer to this question. I didn't want to bog down my question with what I thought unnecessary detail but seeing as I'm being criticized for the lack of this I'll expand a bit here.
I want to automate the updating of FWs on our production line, so I've a python app that gets the FWs from ftp and then uses the processors flash tool via python subprocess command to upload it to the board's flash. OK for all but one of the tools.
The one tool seems to have a problem when it's not running in its own terminal, so I provide a start to the subprocess command string which allows it to run OK in its own terminal. However, I need the output from this other terminal for logging reasons.
A possible solution was to log stdout and stderr to file using >> or wintee and then poll this file via a thread in my original app (perhaps a rather convoluted solution to the question). However the tool also has a separate problem where it doesn't like any std redirection, so this doesn't work for me.
I am trying to implement a terminal emulator in Java. It is supposed to be able to host both cmd.exe on Windows and bash on Unix-like systems (I would like to support at least Linux and Mac OS X). The problem I have is that both cmd.exe and bash repeat on their standard output whatever I send to their standard input.
For example, in bash, I type "ls", hit enter, at which point the terminal emulator sends the input line to bash's stdin and flushes the stream. The process then outputs the input line again "ls\n" and then the output of the ls command.
This is a problem, because other programs apart from bash and cmd.exe don't do that. If I run, inside either bash, or cmd.exe, the command "python -i", the python interactive shell does not repeat the input in the way bash and cmd.exe does. This means a workaround would have to know what process the actual output came from. I doubt that's what actual terminal emulators do.
Running "bash -i" doesn't change this behaviour. As far as I know, cmd.exe doesn't have distinct "interactive" and "noninteractive" modes.
EDIT: I am creating the host process using the ProcessBuilder class. I am reading the stdout and stderr and writing to the stdin of the process using a technique similar to the stream gobbler. I don't set any environment variables before I start the host process. The exact commands I use to start the processes are bash -i for bash and cmd for cmd.exe. I'll try to post minimal code example as soon as I manage to create one.
On Unix, run stty -echo to disable "local echo" (i.e. the shell repeating everything that you type). This is usually enabled so a user can edit what she types.
In your case, BASH must somehow allocate a pseudo TTY; otherwise, it would not echo every command. set +x would have a similar effect but then, you'd see + ls instead of ls in the output.
With cmd.exe the command #ECHO OFF should achieve the same effect.
Just execute those after the process has been created and it should work.