python 2.6 subprocess enter psql password C shell - python-2.6

I am not able to install any additional modules on an AIX server I am scripting on. It has Python 2.6.2 installed and basically I am trying to run the below command
psql -c "select * from cms_agprm;" -o u350932.txt -d tctmsv80 -U tctmsv80
Now I have done some goggling and found a few different recommendation on how to do this. None with much success. I'm also a little paranoid because I am dealing with processes and want to make sure that i get things right in this script otherwise things could go pear shaped very quickly.
I have tried setting the enviroment variable so i am not prompted for a password when making a psql call.
p_env = subprocess.Popen(["setenv", "PGPASSWORD", "password"], shell=True)
print_env = subprocess.Popen(["env"], shell=True)
print 'sending psql command'
p = subprocess.Popen(["psql", "-c", "select * from cms_agprm;", "-o", "/apps/ctm/80/devsv/temp/old_jobs/u350932.txt", "-d", "tctmsv80", "-U", "tctmsv80"], shell=True, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
this seems to be the most elegant way of doing this as I do not have to deal with input and output of this process command (password prompts etc). However the code seems to run without error but i am not left with an output file of that query. The environment variable also appears to be set. I have no idea what is going on, any help on what I might be doing wrong here would be greatly appreciated.
UPDATE:
I realised that I need to pass all the environmental variables when using env=.... This will be more of a challenge as I have to set quite a few. It would be easier just to enter the password. I have tried the below code with no luck.
print 'sending psql command'
p = subprocess.Popen(["psql", "-c", "'select * from cms_agprm;'", "-d", "tctmsv80", "-U", "tctmsv80"], shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p.stdout.read()
p.stdin.write('password\n')
p.stdin.flush()
p.kill()
hanging output:
ctmtest1-tctmsv80 [8] python clean_jobs_main.py
script to clean up old jobs
sending psql command
Password:
after I press enter:
Traceback (most recent call last):
File "clean_jobs_main.py", line 21, in <module>
sys.exit(main())
File "clean_jobs_main.py", line 18, in main
old_job_list = get_old_jobs()
File "clean_jobs_main.py", line 12, in get_old_jobs
p.stdin.write('password\n')
IOError: [Errno 32] Broken pipe
UPDATE 2:
I went back to the drawing board and feel I have gotten closer.
command_line = "psql -c 'select * from cms_agprm;' -o u350932.txt -d tctmsv80 -U tctmsv80"
print(command_line)
p_cmd = shlex.split(command_line)
print p_cmd
print 'sending psql command'
p = subprocess.Popen(p_cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
r = p.communicate('password\n')
print r
output: it hangs until i press enter
ctmtest1-tctmsv80 [6] python clean_jobs_main.py
script to clean up old jobs
psql -c 'select * from cms_agprm;' -o u350932.txt -d tctmsv80 -U tctmsv80
['psql', '-c', 'select * from cms_agprm;', '-o', 'u350932.txt', '-d', 'tctmsv80', '-U', 'tctmsv80']
sending psql command
Password for user tctmsv80:
when I press enter:
('', 'psql: fe_sendauth: no password supplied\n')
So it doesn't look like my communicate method is working but im not sure why?
Cheers

Related

Subprocess with ruby

I try to find the good way to call subprocess with ruby.I want to launch a new second terminal with a commande inside and then execute a new command after the last one on the first terminal.I want to do the same in ruby like this python short code.
import os
import subprocess
htop = "x-terminal-emulator -e 'bash -c \"htop; exec bash\"'"
ls = "ls"
p = subprocess.Popen(htop, shell=True, stderr=subprocess.PIPE)
p1 = subprocess.Popen(ls, shell=True, stderr=subprocess.PIPE)
here htop is started on second terminal and then ls on the first, with ruby im stuck at htop and after ls is not started.Here is my ruby code
stdout, stdeerr, status = Open3.capture3("x-terminal-emulator -e bash -c htop")
sleep 2
system "iwconfig wlan0"
Solved by using fork
fork { exec "x-terminal-emulator -e bash -c htop" }
sleep 2
system "ls"
I will come back soon im learning ruby.

Python issue running subprocess.Popen() from a .py in console

I am running a program which uses subprocess.Popen() in order to do some tasks (I cannot use os.system() for them). The program, which correctly runs within an IDE (I will explain later) stops when I run it from the console, although I can resume it by writing fg in the console.
The code is the following (this is a piece of code,the full code implements similar tasks to which the same problem occurs)...
import subprocess
p,o = subprocess.Popen(['/bin/bash', '-c', '-i', 'which python3.5'], stdout=subprocess.PIPE).communicate()
p = p.decode('ascii')
print(p)
print('Installing pysamstats...')
subprocess.Popen(['/bin/bash', '-c', '-i', 'conda install -y -c bioconda pysamstats']).communicate()
print('OK')
If the code is run from an IDE (I use PyCharm with anaconda as interpreter) the output occurs and all the script runs smoothly, with the exception that the following message appears each time Popen is called...
bash: cannot set terminal process group (3556): Inappropriate ioctl for device
bash: no job control in this shell
With the exception of the "error", as I have said, the code is correctly executed.
However, if I run the .py file from the console...
python3 '/DIRTOFILE/ZZZ.py'
The following message appears...
/home/labcombio1/anaconda3/bin/python3.5
Installing pysamstats...
[1]+ Stopped python3 '/media/labcombio1/ZZZ.py'
That is, the first command is executed, and the output is correctly printed, whereas the second command is stopped. If I resume the command with fg, it works fine. The same happens if the first Popen command is not run, that is, only with the second one.
I have tried shell=True, removing .communicate(), adding stdout=subprocess.PIPE, and none of these things solves the "Stopped" line at the console. Other commands yield the same results.
Finally, I have tried running the following commands:
subprocess.Popen('conda install -y -c bioconda pysamstats', shell=True).communicate()
subprocess.Popen(['/bin/bash', '-c', 'conda install -y -c bioconda pysamstats']).communicate()
And although they both work correctly in Pycharm (they don't even prompt bash: cannot set terminal process group (3556): Inappropriate ioctl for device), the terminal still fails, with the following message:
/bin/sh: 1: conda: not found
I know little about subprocess module, and despite reading other questions and some pages taking about it, I have been unable to solve that problem.
Thank you in advance
First problem:
bash: cannot set terminal process group (3556): Inappropriate ioctl for device
bash: no job control in this shell
Remove -i flag from bashinterpreter.
-i If the -i option is present, the shell is interactive.
Example:
p, o = subprocess.Popen(['/bin/bash', '-c', 'which python3.5'], stdout=subprocess.PIPE).communicate()
Second issue:
/bin/sh: 1: conda: not found
The condapath is not exported in your environment.
Possible Solutions
1) Load it in your IDE.
2) Use the full path to the executable.
subprocess.Popen(['/path/to/conda/conda', 'install', '-y', '-c', 'bioconda', 'pysamstats']).communicate()
3) Pass an env to Popen.
subprocess.Popen(['conda', 'install', '-y', '-c', 'bioconda', 'pysamstats'], env={"PATH": "/path/to/conda"}).communicate()

Nest execute commands in bash

I am trying to associate a hotkey with opening vim with recent history browsing, thus I have wrote the following line
gnome-terminal -x "vim -c ':browse old'"
However this gives
Error: Failed to execute child process "vim -c ':browse old'" (No such file or directory)
What am I doing wrong?
Good news! The -x option of gnome-terminal makes it very easy to start a new terminal and run a new program in it. Just do:
gnome-terminal -x vim -c ':browse old'
The meaning of -x is that all subsequent arguments are passed to the program that you run, so no quoting is needed.

How to execute bunch of commands in one pipe using python?

I have issue about executing commands in python.
Problem is:
In our company we have bought commercial software that can be used either GUI or Command line interface. I have been assigned a task that automize it as possible as. First I thought about using CLI instead of GUI. But then i have encountered a problem about executing multiple commands.
Now, I want to execute CLI version of that soft with arguments and continue executing commands in its menu(I dont mean execute script with args again.I want , once initial commands executed , it will open menu and i want to execute soft's commands inside Soft's menu at background). Then redirect output to variable.
I know, I must use subprocess with PIPE , but I didn't manage it.
import subprocess
proc=subprocess.Popen('./Goldbackup -s -I -U', shell=True, stdout=subprocess.PIPE)
output=proc.communicate()[0]
proc_2 = subprocess.Popen('yes\r\n/dir/blabla/\r\nyes', shell=True, stdout=subprocess.PIPE)
# This one i want to execute inside first subprocess
Set stdin=PIPE if you want to pass commands to a subprocess via its stdin:
#!/usr/bin/env python
from subprocess import Popen, PIPE
proc = Popen('./Goldbackup -s -I -U'.split(), stdin=PIPE, stdout=PIPE,
universal_newlines=True)
output = proc.communicate('yes\n/dir/blabla/\nyes')[0]
See Python - How do I pass a string into subprocess.Popen (using the stdin argument)?

Problems in sending input to detached 'screen' via readreg and paste

I am trying to send input to a interactive command running via screen. Here is my initial command
screen -L -c ./customrc -S psql -d -m /opt/PostgreSQL/9.0/bin/psql
The above command will run interactive psql in screen detach mode. The customrc is used to define a log file for the output (which I will read from another process by polling)
I am using following two commands to send input to psql running in screen
screen -S psql -X readreg p psqlcommands.sql
screen -S psql -X paste p
The problem is that the above commands do not work unless I reattach screen at least once. Once I have attached screen and detached, the above commands work as expected. I have to launch these commands via background java process hence the interactive shell (bash) is not available. My goal is to run psql in interactive mode and pass input to it and capture its output via a log file.
So far I have tried to run screen via xterm (or konsole or gnome-terminal) in attach mode, use readreg/paste and then detach, but I realise that xterm will not be available in my production environment. I've also tried sending output to /proc//fd/0 but I am unable to emulate 'ENTER' from keyboard (I have to attach and press in order for the output to be accepted by psql). I think pipes and fifo may help but I am unable to figure out how to proceed with them using screen and psql.
I appreciate any further hints or workarounds.
Thank you,
Usman.
Well, you can use
screen -S psql -p 0 -X stuff $'\n'
or better (works for me)
screen -S mname -p 0 -X stuff `echo -ne '\015'`
-p 0 is needed to select the window.
Have you tried this to "press enter" after your readreg and paste?
screen -S psql -X stuff $'\n'
FINAL ANSWER: It is a bug/feature in 'GNU screen' that it needs a DISPLAY atleast once for 'paste' command to work. Following are possible workarounds this problem:
Finally figured out how to utilise psql with pipes and screen. Here is the solution:
mkfifo psql.pipe
screen -L -c ./customrc -S psql -d -m bash -i -c "while (true); do cat psql.pipe; done | /opt/PostgreSQL/9.0/bin/psql -a"
After that, I can cat my commands to the pipe:
cat ./mycommands.sql > psql.pipe
To quit from screen and terminating psql, I used
screen -S psql -X quit
EDIT: (finally) figured out the solution for my problem without using screen command. Meet 'empty' utility.
empty -f -i psql.in -o psql.o -p psql.pid <psqlpath>
This allows psql to run in full interactive mode as opposed to the original solution that I used (in which psql does not run in interactive mode).
Thanks.
Usman
I had this same problem. My workaround was to launch screen attached but pass it a screenrc file where the last command is "detach"
So this is my screenrc
#change the hardstatus settings to give an window list at the bottom of the
#Set this first otherwise messes with bash profile
hardstatus alwayslastline
#screen, with the time and date and with the current window highlighted
#hardstatus string '%{= kG}%-Lw%{= kW}%50> %n%f* %t%{= kG}%+Lw%< %{= kG}%-=%c:%s%{-}'
hardstatus string '%{= kG}[ %{G}%H %{g}][%= %{= kw}%?%-Lw%?%{r}(%{W}%n*%f%t%?(%u)%?%{r})%{w}%?%+Lw%?%?%= %{g}][%{B} %m-%d %{W}%c %{g}]'
#set scrollback
defscrollback 4096
#detach
detach
Hope this helps
P

Resources