Python issue running subprocess.Popen() from a .py in console - bash

I am running a program which uses subprocess.Popen() in order to do some tasks (I cannot use os.system() for them). The program, which correctly runs within an IDE (I will explain later) stops when I run it from the console, although I can resume it by writing fg in the console.
The code is the following (this is a piece of code,the full code implements similar tasks to which the same problem occurs)...
import subprocess
p,o = subprocess.Popen(['/bin/bash', '-c', '-i', 'which python3.5'], stdout=subprocess.PIPE).communicate()
p = p.decode('ascii')
print(p)
print('Installing pysamstats...')
subprocess.Popen(['/bin/bash', '-c', '-i', 'conda install -y -c bioconda pysamstats']).communicate()
print('OK')
If the code is run from an IDE (I use PyCharm with anaconda as interpreter) the output occurs and all the script runs smoothly, with the exception that the following message appears each time Popen is called...
bash: cannot set terminal process group (3556): Inappropriate ioctl for device
bash: no job control in this shell
With the exception of the "error", as I have said, the code is correctly executed.
However, if I run the .py file from the console...
python3 '/DIRTOFILE/ZZZ.py'
The following message appears...
/home/labcombio1/anaconda3/bin/python3.5
Installing pysamstats...
[1]+ Stopped python3 '/media/labcombio1/ZZZ.py'
That is, the first command is executed, and the output is correctly printed, whereas the second command is stopped. If I resume the command with fg, it works fine. The same happens if the first Popen command is not run, that is, only with the second one.
I have tried shell=True, removing .communicate(), adding stdout=subprocess.PIPE, and none of these things solves the "Stopped" line at the console. Other commands yield the same results.
Finally, I have tried running the following commands:
subprocess.Popen('conda install -y -c bioconda pysamstats', shell=True).communicate()
subprocess.Popen(['/bin/bash', '-c', 'conda install -y -c bioconda pysamstats']).communicate()
And although they both work correctly in Pycharm (they don't even prompt bash: cannot set terminal process group (3556): Inappropriate ioctl for device), the terminal still fails, with the following message:
/bin/sh: 1: conda: not found
I know little about subprocess module, and despite reading other questions and some pages taking about it, I have been unable to solve that problem.
Thank you in advance

First problem:
bash: cannot set terminal process group (3556): Inappropriate ioctl for device
bash: no job control in this shell
Remove -i flag from bashinterpreter.
-i If the -i option is present, the shell is interactive.
Example:
p, o = subprocess.Popen(['/bin/bash', '-c', 'which python3.5'], stdout=subprocess.PIPE).communicate()
Second issue:
/bin/sh: 1: conda: not found
The condapath is not exported in your environment.
Possible Solutions
1) Load it in your IDE.
2) Use the full path to the executable.
subprocess.Popen(['/path/to/conda/conda', 'install', '-y', '-c', 'bioconda', 'pysamstats']).communicate()
3) Pass an env to Popen.
subprocess.Popen(['conda', 'install', '-y', '-c', 'bioconda', 'pysamstats'], env={"PATH": "/path/to/conda"}).communicate()

Related

Mainscript executes subscripts in a new terminal window

"Debian 9 64x - LXDE"
I try to create an install script in bash. Lets assume i want to install samba. I call from the mainscript the install script for samba \folder\samba.sh. The script samba.sh should get executed in a new terminal window, so i can watch for install errors.
The script should work like follow description:
The script /mainscript.sh provides only user information, interaction, and executes multiple subscripts (/folder/subscripts.sh).
The script /mainscript.sh needs to create a new terminal window, passes the path, and the name of subscript.sh and executes them in the new terminal window.
The script /mainscript.sh must only execute one subscript (/folder/subscript.sh) at the time! If a subscript.sh is running then the mainscript must wait until the new terminal window gets closed.
The subscript.sh executes some code with root privileges.
Questions:
How can I create a new terminal window, pass the subscript, and execute it in the new terminal window?
How can I make sure that the script (mainscript.sh) only runs one subscript (subscript.sh) at the time?
Example:
mainscript.sh
#!/bin/sh
# This is the content of the mainscript.sh
# subscript1 and subscript2 must not be executed at the same time!
# the mainscript needs to wait when a subscript gets executed!
echo "hello, this script runs in terminal window (((A)))"
xterm /opt/subscript1.sh
echo "samba - Installed"
xterm /opt/subscript2.sh
echo "samba - removed"
subscript1.sh
#!bin/sh
# This is the content of the subscript1
echo "This script runs in a new terminal window (((B)))"
apt-get install samba
# instructions done .... close the terminal window (((B))) now
subscript2.sh
#!bin/sh
# This is the content of the subscript2
echo "This script runs in a new terminal window (((C)))"
apt-get remove samba
# instructions done .... close the terminal window (((C))) now
After clarification that you actually want a new terminal window to appear in LXDE here is a possible solution.
Debian LXDE is likely to have xterm or lxterminal installed. The example below is using lxterminal. For xterm use "xterm -e command"
Start by executing manscript.sh in its own window:
$ lxterminal --command=/mainscript.sh
#!/usr/bin/sh
<section that provides user information>
# Call subscripts that will run in sequence
lxterminal --command=/folder/subscripts.sh
When subscripts.sh finishes, the new terminal window will close and return control to mainscript.sh
You get only one subscript to run at a time by calling these in sequence.

Intermediate bash output in Jupyter notebook

When I run this in terminal:
cd 1st_flask_app_1/
python3 app.py
I get the output:
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
When I try to run the same command in a Jupyter notebook with the cell magic %%bash, I get no printed output, but the web app still starts up and I can visit it. If I then stop the cell I get the output:
Process is interrupted.
So it looks like the Jupyter notebook with %%bash cell magic is printing final command outputs, but not intermediate outputs. Is there any way to also print the intermediate outputs?
For some reason it looks like your stdout/stderr are not properly attached to your bash...
Could you try to redirect your command stdout and stderr to a log file in append mode?
python3 app.py >> application.log 2>&1
or you could also pipe it with tee
python3 app.py 2>&1 | tee -a application.log
I hope this helps you!

Jenkins Pipeline cannot execute SH command file in a Windows slave

I'm executing this code:
node('my_windows_slave') {
sh 'ls'
}
In my Windows slave I can properly execute sh command:
But the pipeline script can't run the .sh file:
[Pipeline] sh
[D:\workspace\sandbox_pipeline] Running shell script
sh: D:\workspace\sandbox_pipeline#tmp\durable-2d7dd2f8\script.sh: command not found
What I could notice is that this .sh file is not even created, once I tried with bat and worked fined.
Any clue what could be the problem?
[UPDATE]
Jenkins somehow can't create the SH temporary file. Already checked the log, permissions, everything that came to my mind.
I will leave my workaround as an answer for while before approve it once I'm still not 100% sure about the root cause and might someone else show up with a elegant solution...
def shell(command) {
return bat(returnStdout: true, script: "sh -x -c \"${command}\"").trim()
}
Attention
You still executing SH commands in a CMD, it means some %d for example can break your SH command.
Use the bat step instead of sh.
From Jenkins docs:
Windows-based systems should use the bat step for executing batch commands.

How to execute bunch of commands in one pipe using python?

I have issue about executing commands in python.
Problem is:
In our company we have bought commercial software that can be used either GUI or Command line interface. I have been assigned a task that automize it as possible as. First I thought about using CLI instead of GUI. But then i have encountered a problem about executing multiple commands.
Now, I want to execute CLI version of that soft with arguments and continue executing commands in its menu(I dont mean execute script with args again.I want , once initial commands executed , it will open menu and i want to execute soft's commands inside Soft's menu at background). Then redirect output to variable.
I know, I must use subprocess with PIPE , but I didn't manage it.
import subprocess
proc=subprocess.Popen('./Goldbackup -s -I -U', shell=True, stdout=subprocess.PIPE)
output=proc.communicate()[0]
proc_2 = subprocess.Popen('yes\r\n/dir/blabla/\r\nyes', shell=True, stdout=subprocess.PIPE)
# This one i want to execute inside first subprocess
Set stdin=PIPE if you want to pass commands to a subprocess via its stdin:
#!/usr/bin/env python
from subprocess import Popen, PIPE
proc = Popen('./Goldbackup -s -I -U'.split(), stdin=PIPE, stdout=PIPE,
universal_newlines=True)
output = proc.communicate('yes\n/dir/blabla/\nyes')[0]
See Python - How do I pass a string into subprocess.Popen (using the stdin argument)?

IRB analogue of python -i

I want to use IRB to run a script then give me an interactive prompt. I do this with python -i xy.py in Python, however irb xy.rb exits after execution.
> python --help
-i When a script is passed as first argument or the -c option is
used, enter interactive mode after executing the script or the
command
irb -r xy.rb
It simply requires the file mentioned before giving you a normal IRB prompt.

Resources