I'm learning Python while converting some bash scripts to Python shell scripts. One thing I don't yet understand is how to deal with the heredocs used in these scripts. Here are two examples of how the bash scripts use heredocs:
The most important thing I need to know how to do in Python is this first case where the heredoc is used to provide standard responses to commands so the command can run non-interactively:
sudo command << 'EOF'
prompt_response1
prompt_response2
EOF
Second, tee is used like to this to create a file for which sudo permissions are required:
sudo tee /etc/xdg/autostart/updateNotificationChecker.desktop > /dev/null << 'EOF'
[Desktop Entry]
Name=Update Notification
Exec=bash /usr/local/bin/updateNotification.sh
Terminal=false
Type=Application
NoDisplay=true
EOF
How would I do these things in Python?
Heredoc in Python
Use multiline string (triple-quote string ''' or """). See Strings from tutorial.
Run command
import subprocess
subprocess.Popen(['cat'], stdin=subprocess.PIPE).communicate('''
Hello multiline-string
simliar to heredoc.
''')
sh (previously pbs) is a full-fledged subprocess interface for Python that allows you to call any program as if it were a function:
from sh import ifconfig
print(ifconfig("wlan0"))
Full docs: http://amoffat.github.com/sh
Follow on Github: http://github.com/amoffat/sh
Example of how it can solve the first problem from this question:
from sh import ssh
import os, sys
# open stdout in unbuffered mode
sys.stdout = os.fdopen(sys.stdout.fileno(), "wb", 0)
aggregated = ""
def ssh_interact(char, stdin):
global aggregated
sys.stdout.write(char.encode())
aggregated += char
if aggregated.endswith("password: "):
stdin.put("correcthorsebatterystaple\n")
p = ssh("10.10.10.100", _out=ssh_interact, _out_bufsize=0, _tty_in=True)
p.wait()
It can handle sudo like this:
with sudo:
print(ls("/root"))
It has a neat feature called STDOUT/ERR callbacks:
sh can use callbacks to process output incrementally. This is done much like redirection: by passing an argument to either the _out or _err (or both) special keyword arguments, except this time, you pass a callable. This callable will be called for each line (or chunk) of data that your command outputs.
Finally, as part of the standard Python tools there is raw_input which writes to standard output and reads from standard input. That will also solve the second problem in this question.
Related
I have issue about executing commands in python.
Problem is:
In our company we have bought commercial software that can be used either GUI or Command line interface. I have been assigned a task that automize it as possible as. First I thought about using CLI instead of GUI. But then i have encountered a problem about executing multiple commands.
Now, I want to execute CLI version of that soft with arguments and continue executing commands in its menu(I dont mean execute script with args again.I want , once initial commands executed , it will open menu and i want to execute soft's commands inside Soft's menu at background). Then redirect output to variable.
I know, I must use subprocess with PIPE , but I didn't manage it.
import subprocess
proc=subprocess.Popen('./Goldbackup -s -I -U', shell=True, stdout=subprocess.PIPE)
output=proc.communicate()[0]
proc_2 = subprocess.Popen('yes\r\n/dir/blabla/\r\nyes', shell=True, stdout=subprocess.PIPE)
# This one i want to execute inside first subprocess
Set stdin=PIPE if you want to pass commands to a subprocess via its stdin:
#!/usr/bin/env python
from subprocess import Popen, PIPE
proc = Popen('./Goldbackup -s -I -U'.split(), stdin=PIPE, stdout=PIPE,
universal_newlines=True)
output = proc.communicate('yes\n/dir/blabla/\nyes')[0]
See Python - How do I pass a string into subprocess.Popen (using the stdin argument)?
I am writing a Python script that will help me submit homework assignments. What I want to do is pass the login credentials I have stored in a file to a submission script, after which I will hand over to the user for further input. I have come up with the shell command, cat login_credentials.txt /dev/stdin | python3 submit_hw0.py. I am almost finished, I just don't know how to execute the shell command cat login_credentials.txt /dev/stdin | python3 submit_hw0.py, where submit_hw0.py is replaced by an arbitrary Python file I have stored in a variable named submission_script. Also, I have tried to say:
import subprocess
subprocess.call("cat login_credentials.txt /d")
However, python3 just closes with no output.
Any help would be appreciated,
Kind regards, Kabelo Moiloa.
Take a look on the Popen class of subprocess module (link). You can pass stdin and stdout parameters to it. That should solve your problems.
I've been wanting to run some ruby scripts on remote computers (in a bash shell)
I could create a sequence of bash commands of ruby -e "<command>", but some of these scripts are over 100 lines.
ruby -e with a HEREDOC or %{} & eval() doesn't work well with the mixture of single and double quotes.
Is there a better way to attempt this?
Edit:
The protocol being used is Apple Remote Desktop, which executes these commands in the scope of the remote shell.
If I understand you correctly, you want to run local ruby script on remote machine via SSH or similar protocol. If the script is non-interactive (i.e. doesn't require any user input), you could create it locally and deliver through stdin.
In other words, first write the script and save it locally as, say, foo.rb. Then:
ssh remotehost ruby < foo.rb
That with start the SSH session and execute the remote ruby interpreter. With no arguments, the ruby interpreter executes commands from standard input, and thus we feed SSH with the program on stdin.
As I also want to run ruby scripts via ARD (which I don't think can embed a ctrl-D), I first thought you could combine joraff's solution (to his own problem) with Kelvin's:
cat << ENDOFSCRIPT | ruby
#Here be code ...
ENDOFSCRIPT
Which saves creating/deleting a file.
But there's an even better way:
It turns out (duh) that ARD embeds an EOF or just otherwise terminates what it sends in such a way that you can simply do:
ruby
#Paste whole script here
Works at least in ARD 3.6.1. Win!
This worked:
cat << 'EOF' > /tmp/myscript.rb
# ruby script contents go here. any syntax is valid, except for your limit string (EOF)
EOF
ruby /tmp/myscript.rb;
rm /tmp/myscript.rb;
Since this isn't relying on how an interpreter binary handles stdin-style commands, it will work for most other languages as well (python, perl, php).
Why not send the script over first?
scp foo.rb remotehost:
ssh remotehost "ruby foo.rb"
You could even clean up the file after:
ssh remotehost "rm foo.rb"
I would like to execute a number of bash commands from a Rakefile.
I have tried the following in my Rakefile
task :hello do
%{echo "World!"}
end
but upon executing rake hello there is no output?
How do I execute bash commands from a Rakefile?
NOTE:This is not a duplicate as it's specifically asking how to execute bash commands from a Rakefile.
I think the way rake wants this to happen is with: http://rubydoc.info/gems/rake/FileUtils#sh-instance_method
Example:
task :test do
sh "ls"
end
The built-in rake function sh takes care of the return value of the command (the task fails if the command has a return value other than 0) and in addition it also outputs the commands output.
There are several ways to execute shell commands in ruby. A simple one (and probably the most common) is to use backticks:
task :hello do
`echo "World!"`
end
Backticks have a nice effect where the standard output of the shell command becomes the return value. So, for example, you can get the output of ls by doing
shell_dir_listing = `ls`
But there are many other ways to call shell commands and they all have benefits/drawbacks and work differently. This article explains the choices in detail, but here's a quick summary possibilities:
stdout = %x{cmd} - Alternate syntax for backticks, behind the scenes
it's doing the same thing
exec(cmd) - Completely replace the running process with a new cmd process
success = system(cmd) - Run a subprocess and return true/false
on success/failure (based on cmd exit status)
IO#popen(cmd) { |io| } - Run a subprocess and connect stdout and
stderr to io
stdin, stdout, stderr = Open3.popen3(cmd) - Run a subprocess and
connect to all pipes (in, out, err)
Given that the consensus seems to prefer rake's #sh method, but OP explicitly requests bash, this answer may have some use.
This is relevant since Rake#sh uses the Kernel#system call to run shell commands. Ruby hardcodes that to /bin/sh, ignoring the user's configured shell or $SHELL in the environment.
Here's a workaround which invokes bash from /bin/sh, allowing you to still use the sh method:
task :hello_world do
sh <<-EOS.strip_heredoc, {verbose: false}
/bin/bash -xeu <<'BASH'
echo "Hello, world!"
BASH
EOS
end
class String
def strip_heredoc
gsub(/^#{scan(/^[ \t]*(?=\S)/).min}/, ''.freeze)
end
end
#strip_heredoc is borrowed from rails:
https://github.com/rails/rails/blob/master/activesupport/lib/active_support/core_ext/string/strip.rb
You could probably get it by requiring active_support, or maybe it's autoloaded when you're in a rails project, but I was using this outside rails and so had to def it myself.
There are two heredocs, an outer one with the markers EOS and an inner one with the markers BASH.
The way this works is by feeding the inside heredoc between the BASH markers to bash's stdin. Note that it is running within the context of /bin/sh, so it's a posix heredoc, not a ruby one. Normally that requires the end marker to be in column 1, which isn't the case here because of the indenting.
However, because it's wrapped within a ruby heredoc, the strip_heredoc method applied there de-indents it, placing the entirety of the left side of the inner heredoc in column 1 prior to /bin/sh seeing it.
/bin/sh also would normally expand variables within the heredoc, which could interfere with the script. The single quotes around the start marker, 'BASH', tell /bin/sh not to expand anything inside the heredoc before it is passed to bash.
However /bin/sh does still apply escapes to the string before passing it to bash. That means backslash escapes have to be doubled to make it through /bin/sh to bash, i.e. \ becomes \\.
The bash options -xeu are optional.
The -eu arguments tell bash to run in strict mode, which stops execution upon any failure or reference to an undefined variable. This will return an error to rake, which will stop the rake task. Usually, this is what you want. The arguments can be dropped if you want normal bash behavior.
The -x option to bash and {verbose: false} argument to #sh work in concert so that rake only prints the bash commands which are actually executed. This is useful if your bash script isn't meant to run in its entirety, for example, if it has a test which allows it to exit gracefully early in the script.
Be careful to not set an exit code other than 0 if you don't want the rake task to fail. Usually, that means you don't want to use any || exit constructs without setting the exit code explicitly, i.e. || exit 0.
%{echo "World!"} defines a String. I expect you wanted %x{echo "World!"}.
%x{echo "World!"} executes the command and returns the output (stdout). You will not see the result. But you may do:
puts %x{echo "World!"}
There are more ways to call a system command:
Backticks: `
system( cmd )
popen
Open3#popen3
There are two ways:
sh " expr "
or
%x( expr )
Mind that ( expr ) can be { expr } , | expr | or ` expr `
The difference is, sh "expr" is a ruby method to execute something, and %x( expr ) is the ruby built-in method. The result and action are different. Here is an example
task :default do
value = sh "echo hello"
puts value
value = %x(echo world)
puts value
end
get:
hello # from sh "echo hello"
true # from puts value
world # from puts value
You can see that %x( expr ) will only do the shell expr but the stdout will not show in the screen. So, you'd better use%x( expr ) when you need the command result.
But if you just want to do a shell command, I recommend you use sh "expr". Because sh "irb" will make you go into the irb shell, while %x(irb) will dead.
I have a rake task that runs, quite a lot of code. At the end, I need to use sftp and ssh to do some stuff. At the moment I'm unable to automate it. Is there a way to pass to stdout?
This seems like a simple question but I can't find the answer anywhere
#some ruby code
#more ruby code
sh "sftp myuser#hots" #this opens the sftp console
sh "put file" #this doesn't get run until sftp is exited
sh "put another_file" #neither does this
#more ruby code
sh "ssh host" # opens the ssh console
sh "some_action_on_host" # this doesn't get run until ssh is exited
I know there will be ways of doing sftp and ssh using ruby but ideally I just want to be able to pipe variables and commands into the console
So you want to run sftp and send a series of commands to it? How about something like:
sftp = IO.popen("sftp myuser#hots", "w+")
sftp << "put file\n"
sftp << "put another file\n"
sftp.flush # make sure to include this
If you don't want to use ruby, then you may want to enclose your shell commands into ` (backtick characters). This string will be passed to Kernel.` method. This method execute the text as an OS shell command and returns the command's output as a string, e.g.:
`ls`
Alternative syntax to ` is %x[]. This way you can write any bash script:
%x[sftp myuser#hots <<COMMAND
put #{file}
quit
COMMAND]
Please note that this syntax support ruby expressions interpolation using #{...} syntax (similar to double-quoted string literals).