Hi I am at a loss on how to run multiple commands using popen,
I am trying to automate a series of steps that are normally run on the Windows command line. The basic steps are usually run from the Windows cmd line are
Run a windows command script (.cmd) file to setup environment variables i.e C:\Program Files (x86)\appsettings\setupvariables.cmd
type in the command to connect to the database
type in the command to get data from the database
Stop connection to the database
All these commands must run in the same command line window one after another, not separate processes or separate command line windows. Instead of opening a cmd window and typing in the command I want to use python's subprocess.popen command
So far I have:
args=[]
args.append(r'C:\Program Files (x86)\appsettings\setmyvars.cmd')
args.append(r'start db on db_path="my_url"')
args.append(r'get_data_from_db>c:\temp\output.txt')
args.append(r'stop db on db_path="my_url"')
p=Popen(args,stdout=PIPE,sterr=PIPE,shell=True)
stdout,stderr=p.communicate()
if stderr:
print "you have an error", stderr
else:
print "well done you have data", stdout
This isn't quite working I can see that the first line is run i.e the setmyvars.cmd is executed, but nothing else, none of the other arguments get called, if they did I would see the results in the ouput.txt file.
How do I run a series of commands one after the other using popen. Why is it only the first command seems to be executed and none of the others
I am using python2.7 on Windows
Regards.
You have a couple of issues going on. You still have to tell popen() which program to run. Just using shell=True does not obviate the need to provide cmd.exe as the program to run. If you really want to run all of these commands with one invocation of cmd.exe, then you will need to string them together with &&.
from subprocess import *
args=[]
args.append(r'C:\Windows\System32\cmd.exe')
args.append(r'/C')
args.append(r'(echo 1 && echo 2 && echo 4)')
p = Popen(args,stdout=PIPE,stderr=PIPE,shell=True)
stdout,stderr=p.communicate()
if stderr:
print "you have an error", stderr
else:
print "well done you have data", stdout
It would probably be better to use the %ComSpec% environment variable than it would be to hardcode the location of cmd.exe. The path you have is -usually- correct. :-)
Related
On Windows 10 I have a special developer command prompt (DevEnv). Every time another program (Matlab) creates a file output, I have to run a command in this command prompt to start another process:
myDevEnv.bat (sets environment variables, starts server instances, ...)
Matlab script creates myFile.foo
In DevEnv: doSomethingWithFiles.bat myFile.foo
I would like Matlab to call the third command from within the DevEnv. So my idea was to write the commands to a file, that is permanently read and executed in the DevEnv. Is this possible in a Windows command prompt?
This question already has answers here:
WinSCP script not executing in batch file
(2 answers)
Closed last year.
I am trying to run multiple commands in one line. The codes are below.
WinSCP.com
open user:pw#address
lcd C:\Users\xx\Desktop
get *.xlsx
exit
If I run them one by one, it will work. However, if I want to run them all together using one line, it will fail, no matter I am using ; or &.
For example, if I run WinSCP.com & open user:pw#address, only the first part will be executed.
WinSCP.com; open user:pw#address doesn't work either.
How to execute them using one line?
Thanks
This is happening, because both command line and winscp.com are programs. When you're entering commands one by one you start winscp.com and next entered commands are going directly to the WinSCP. When you're trying to execute all commands at once it is starting WinSCP and waiting for it to finish and then executing next commands.
You definitely can create a text file with your commands and save it as script.txt
open user:pw#address
lcd C:\Users\xx\Desktop
get *.xlsx
exit
And then pass it to the winscp.com application like this:
winscp.com /script=script.txt
It may be possible to send all the commands at once by using /command cmd1 cmd2 ... where cmd1,2 are your commands, but it might be tricky.
References
Command-line Options :: WinSCP
Parameters
You can put the sequence of commands in a script file used by WinSCP and call it using the command sequence:
WinSCP.com /script="C:\Users\{name}\Documents\Scripts\{SomeFileName}.scp"
I have found it easier to keep track of what you are doing and keeps the command sequence easier to read.
These are all the possible ways of running multiple commands on a single line :
When you use command1 & command2 on the command prompt, it
executes the first command, and then the second command.
command1 && command2 it runs the first command, and then runs the second command only if the first command completed successfully.
command1 || command2 it runs the first command, and then runs the second command only if the first command did not complete successfully (receives an error code greater than zero).
(command1 & command2) used to group or nest multiple commands.
Since in your case, you want to run the first command and then do the other operation with-in it; it isn't possible to use any of the above.
You have to write a script which you may call in the same line using a command.
You may do something like this : (WinSCP.com /script=<script_path>) && exit
There are many more options in WinSCP.com command.
I would like to communicate with a (remote) non-interactive shell via its stdin/stdout to run multiple commands and read the outputs. The problem is that if I stuff multiple commands on shell stdin, I am not able to detect the boundaries between outputs of individual commands.
In Python-like pseudo-code:
sh = Popen(['ssh', 'user#remote', '/bin/bash'], stdin=PIPE, stdout=PIPE)
sh.stdin.write('ls /\n')
sh.stdin.write('ls /usr\n')
sh.stdin.close()
out = sh.stdout.read()
But obviously out contains the outputs of both commands concatenated, and I have no way of reliably splitting them.
So far my best idea is to insert \0 bytes between the outputs:
sh.stdin.write('ls /; echo -ne "\0"\n')
sh.stdin.write('ls /usr; echo -ne "\0"\n')
Then I can split out on zero characters.
Other approaches that don't work for me:
I don't want to run a separate ssh session per command, as the handshake is too heavyweight.
I'd prefer not to force ControlMaster options on the created shells to respect end-user's ssh_config.
I'd prefer to not need require users to install specific server programs.
Is there a better way of running several commands in one session and getting individual outputs? Is there a widely-deployed shell with some sort of binary output mode?
PS. There is a duplicate question, but it doesn't have a satisfactory answer:
Run multiple commands in a single ssh session using popen and save the output in separate files
For SSH I used paramiko and its invoke_shell method to create a programmatically-manageable shell instance.
The following is not a complete answer, it's still hacky, but I feel it's a step in the right direction.
I required the same read/write shell instance functionality in Windows but have had no luck, so I extended your approach a little (thank you for the idea by the way).
I verify each command executes successfully based on its exit code by placing a conditional exit between each command, then I use the text of said conditional check (a known string) as the delimiter to define each command's response.
A crude example:
from subprocess import Popen, PIPE
sh = Popen('cmd', stdin=PIPE, stdout=PIPE)
sh.stdin.write(b'F:\r\n')
sh.stdin.write(b"if not %errorlevel% == 0 exit\r\n")
sh.stdin.write(b'cd F:\\NewFolder\r\n')
sh.stdin.write(b"if not %errorlevel% == 0 exit\r\n")
sh.stdin.write('...some useful command with the current directory confirmed as set to F:\NewFolder...')
sh.stdin.close()
out = sh.stdout.read()
sh.stdout.close()
# Split 'out' by each line that ends with 'if not %errorlevel% == 0 exit' and do what you require with the responses
I have the following cmd script which calls a cygwin bash script:
C:\cygwin\bin\bash -l /D/Temp/testScript/cygScript.sh
echo back in cmd.
exit
The bash script is simple:
#!/bin/bash
echo Hello World!
read
The calling part works nicely - the bash shell logs in, echos as expected, reads as expected and control passes back to the cmd as expected.
But the cmd will not exit. This is fine if I run it from a command window, but I will be calling this by double clicking on the cmd file or launching it from RUN etc.
Output I see:
D:\Temp\testScript>C:\cygwin\bin\bash -l /D/Temp/testScript/shellScript.sh
Hello World!
D:\Temp\testScript>echo back in cmd.
back in cmd.
D:\Temp\testScript>exit
How do I get the cmd to exit?
Found the problem - too many bashes
I think I found the issue - probably something in my .bashrc and other files I load during cygwin login. If I change the main cmd line to remove the login flag), it works as expected - everything closes.
C:\cygwin\bin\bash /D/Temp/testScript/cygScript.sh
But then I put the flag back:
C:\cygwin\bin\bash -l /D/Temp/testScript/cygScript.sh
and run again. I see the output Hello World! which shows me that control is with bash, and I check Task Manager. Four instances of bash.exe are created. Then I press ENTER and see the output back in cmd. showing me that control is back with cmd. Now Task Manager shows me that three bash.exe instances remain.
So, something in my login scripts are creating extra bash shells. So it's not you, it's me.
Avoid creating persistent subshells in login scripts by using cygstart instead of cmd.
Previously I wrote that the problem was too many bashes. My .bash_profile was doing something to create bash sub-shells (that were persistent because the jobs they launched kept going in the background). This meant that when my cmd created a bash (via login) and the first bash exited, the sub-shells didn't exit.
I found that part of my login scripts involved launching some Autohotkey scripts like this:
cmd /c "$thePath" &
The fix was so easy... just use cygstart:
cygstart "$thePath"
I want to send commands in the ADB shell itself as if i had done the following in cmd.
>adb shell
shell#:/ <command>
I am using python 3.4 on a windows 7 OS 64bit machine. I can send one-line shell commands simply using subprocess.getoutput such as:
subprocess.getoutput ('adb pull /storage/sdcard0/file.txt')
as long as the adb commands themselves are recognized by ADB specifically, such as pull and push, however there are other commands such as grep that need to be run IN the shell, like above, since they are not recognized by adb. for example, the following line will not work:
subprocess.getoutput ('adb shell ls -l | grep ...')
To enter the commands in the shell I thought I needed some kind of expect library as that is what 'everyone' suggests, however pexpect, wexpect, and winexpect all failed to work. they were written for python 2 and after being ported to python 3 and my going through the .py files by hand, even those tweaked for windows, nothing was working - each of them for different reasons.
how can i send the input i want to the adb shell directly?
If none of the already recommended shortcuts work for you you can still go the 'regular' way using 'subprocess.Popen' for entering commands in the adb shell with Popen:
cmd1 = 'adb shell'
cmd2 = 'ls -l | grep ...'
p = subprocess.Popen(cmd1.split(), stdin=PIPE)
time.sleep(1)
p.stdin.write(cmd2.encode('utf-8'))
p.stdin.write('\n'.encode('utf-8'))
p.stdin.flush()
time.sleep(3)
p.kill()
Some things to remember:
even though you import subprocess you still need to invoke subprocess.Popen
sending cmd1 as a string or as items in a list should work too but '.split()' does the trick and is easier on the eyes
since you only specidfied you want to enter input to the shell you only need stdin=PIPE. stdout would only be necessary if you wanted to receive output from the shell
time.sleep(1) isn't really necessary, however since many complained about input issues being faster or slower in python 2 vs 3 consider maybe using it. 'they' might have been using versions of 'expect' that need the shell's reply first. this code also worked when i tested it with simply swapping out and in the process with time.sleep(0)
stdin.write will return an error if the input is not encoded properly. python's default is unicode. entering by binary did not work for me in my tests like this "b\ls ..." but .encode() worked. dont forget the endline!
if you use .encode() there is a worry that the line might not get sent properly, so to be sure it might be good to include a flush().
time.sleep(3) is completely uneccesary, but if your command takes a long time to execute (eg a regressive search through the entire device piped out to a txt file on the memory card) maybe give it some extra time before killing anyhting.
remember to kill. if you didnt kill it, the pipe may remain open, and even after exiting the test app on the console the next commend still went to the shell even though the prompt appearsed to be my regular cmd prompt.
Amichai, I have to start with pointing out that your own "solution" is pretty awful. And your explanation makes it even worse. Doing all those unnecessary things just because you do not understand how shell (here I mean your PC's OS shell, not adb) command parsing works.
When all you needed was just this one command:
subprocess.check_output(['adb', 'shell', 'ls /storage/sdcard0 | grep ...']).decode('utf-8')