I'm just learning Python. One thing I hope to use Python for is Linux shell scripting. I've been studying some examples and practicing by converting my bash scripts to Python.
I just spent a few hours converting one of my bash scripts to Python and I was very disappointed with my resulting code. It was much more verbose and syntactically cluttered. It was certainly not concise and tight like the bash script I started with.
Then I stumbled across this (unrelated) Ruby script. Wow. It seems to give direct access to the shell, unlike Python which has a layer or two between the language and the shell.
#! /usr/bin/ruby
MAX_TIME_SINCE_START = 1800
begin
curr_time = Time.new.to_i
kslideshow_processes_list = `pgrep kslideshow.kss`.split("\n")
kslideshow_processes_list.each { |kslideshow_process|
kslideshow_process.strip!
ps_result = `ps -p #{kslideshow_process} -o lstart`
process_start_date = `ps -p #{kslideshow_process} -o
lstart`.split("\n")[1].strip
get_date_command = "date -d \"#{process_start_date}\" +\"%s\""
kslideshow_start_time = `#{get_date_command}`.to_i
time_since_started = curr_time - kslideshow_start_time
if time_since_started MAX_TIME_SINCE_START
system( "kill #{kslideshow_process}" )
end
}
sleep MAX_TIME_SINCE_START
end while true
This is the kind of code I was hoping would result from my switch from bash script to Python. Is it possible in Python to write shell scripts so cleanly?
It would be very educational for me to see the above code converted to Python by someone who knows what they are doing.
I don't intend to start a Ruby vs. Python discussion. I simply want to see how cleanly the above task can be implemented in Python by someone who knows more Python than I do. Is that a fair question for this site? Thanks
You can IMO do this at least as elegant as in Ruby by using the plumbum package. There are ways to optimize away system calls (e.g. using dateutils to convert from a date format, or using psutils), but that would no longer be a comparison of calling system utilities (I hope I got the logic from the Ruby example correct):
from time import time, sleep
from plumbum.cmd import pgrep, ps, date, kill
MAX_TIME_SINCE_START = 5 # 1800
while True:
curr_time = time()
for kslideshow_process in pgrep("gimp").split():
print kslideshow_process
process_start_date = ps("-p", kslideshow_process, "-o", "lstart"
).split("\n")[1].strip()
kslideshow_start_time = int(date("-d", process_start_date, '+%s'))
print kslideshow_start_time
time_since_started = curr_time - kslideshow_start_time
if time_since_started > MAX_TIME_SINCE_START:
kill([kslideshow_process])
sleep(MAX_TIME_SINCE_START)
You can import any commandline program using from plumbum.cmd import ... and it is automatically available as a function that takes the commandline arguments as parameters.
You can also make chains of commands:
chain = ls["-a"] | grep["-v", "\\.py"] | wc["-l"]
result = chain()
so there is far less often need for intermediate variables to carry results from one subshell to another.
Barring the bad ideas and ways you could poke your own eye out.. an example of the more efficient way to open a another process in python is as follows...
process = subprocess.Popen([MyCommand, parameter1, "parameter 2"], shell=False, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, stdin=subprocess.PIPE)
stdout,stderr = process.communicate()
return_code = process.poll()
Doing things with popen is a little bit more overhead than some other languages, but it offers a lot of versatility that no other scripting language I know of offers like getting output from many many processes live.
Related
Is it possible to have (Rakudo) Perl6 execute some code before dropping you into the REPL? Like python does with "python -i ".
For instance, I want to load up some modules and maybe read a side file and build some data structures from that side file before dropping into the REPL and letting the user do the things they need to do on the data structure, using the REPL as a user interface.
This is similar but different than Start REPL with definitions loaded from file though answers to this question might satisfy that one. The basic case is that, at the end of execution of any program, instead of exiting, the interpreter leaves the user at the REPL. Aside from providing a nifty, built-in, Perl6-based user interface for interactive programs, it also provides a good tool from which to debug code that otherwise exits with an error.
edit:
Selecting Zoffix's solution as the correct (so far) one as it is the only one that satisfies all requirements as stated. Here's hoping this capability gets added to the compiler or language spec.
You can load modules with the -M switch.
$ perl6 -MJSON::Tiny
To exit type 'exit' or '^D'
> to-json Array.new: 1,2,3.Str
[ 1, 2, "3" ]
>
If you want to run other code, currently you have to put it into a module first.
$ mkdir lib
$ echo 'our $bar = 42' > lib/foo.pm6
$ perl6 -Ilib -Mfoo
To exit type 'exit' or '^D'
> $bar
42
>
I'd like to provide an answer that Zoffix gave on IRC. It satisfies the basic requirement but is far from pretty and it uses NQP for which there is no user support nor is the NQP API ( "nqp::*" calls ) guaranteed for the future and can change without warning.
replify ｢
say 'Hello to your custom REPL! Type `say $a` to print the secret variable';
my $a = "The value is {rand}";
｣;
sub replify (Str:D \pre-code = '') {
use nqp;
my %adverbs; # command line args like --MFoo
my \r := REPL.new: nqp::getcomp('perl6'), %adverbs;
my \enc := %adverbs<encoding>:v.Str;
enc && enc ne 'fixed_8' && $*IN.set-encoding: enc;
my $*CTXSAVE := r;
my $*MAIN_CTX;
pre-code and r.repl-eval: pre-code, $, :outer_ctx(nqp::getattr(r, REPL, '$!save_ctx')),
|%adverbs;
$*MAIN_CTX and nqp::bindattr(r, REPL, '$!save_ctx', $*MAIN_CTX);
r.repl-loop: :interactive, |%adverbs;
}
I'm trying to execute a Ruby script file.
Assuming the input is a string that contains the file content.
What are the possible ways? taking into considerations that I need to keep the output of the executed file whether stdout or not separated from the Main script.
As an example of what I'm trying to do is have a function called execute(code)
Then calling execute('4 + 5') would return 9 although I can write a whole Ruby script in the place of '4 + 5'.
If anyone can forward me to any related tutorials or books, I'd be thankful :)
You can call shell commands in Ruby, it's as simple and intuitive as surrounding your desired command in backticks.
The output gets returned, so just save it to a variable:
script1.rb:
puts "asdf"
script2.rb:
output = `ruby script1.rb`
puts output
"asdf"
I question what exactly it is you're trying to do, though. Because this is totally unintuitive and roundabout. Are you sure you aren't just looking for a module or something?
My question is a follow up to this question: No return on command line when running Ruby script because the answer doesn't offer an explanation:
Take the following file, script.rb:
def hello(names)
names.map {|name| "#{name} is awesome!"}
end
hello(["mark", "tony", "scott"])
When executing the file on the command line with ruby script.rb the return value of the following function does not appear. However, testing the method in IRB or by dropping into the code with PRY outputs an explicit return value. Why isn't the return value visible via script execution?
P.S. I am aware that using puts will output code into the terminal but I'm interested in why the return value doesn't output.
Because both IRB or Pry are REPL's
REPL stands for: read, evaluate, print and loop. And that's exactly what both Pry and IRB are doing.
They will first read your input, evaluate your code, print the result of the code execution and then start over.
A Ruby script can't return a value directly like you want it to, the Bash shell works in the same way, a Bash function can't return a string directly. You can return a string (with stdout) and assign it to the variable.
~:$~ cat scr.rb
~:$~ puts "Something here"
~:$~ echo $(ruby ./scr.rb)
Something here
~:$~ myvar=$(echo $(ruby ./scr.rb))
~:$~ echo $myvar
Something here
It's really simple: Bash (or whatever shell you are using) and Ruby are different programming languages. Bash doesn't know anything about Ruby. It doesn't know what a " Ruby return" is, it doesn't know what a "Ruby array" is, it doesn't know what a "Ruby string" is. Therefore, you simply cannot possibly return a Ruby object to Bash.
In fact, the shell usually just uses some operating system function to execute the Ruby script (e.g. the classical fork and exec or something like vfork or clone). If you wanted to return values this way, the operating system kernel would have to know about the semantics of every programming language ever invented plus every programming language that is going to be invented in the future. That is just not feasible.
Note that a command can return a value to the shell, namely an integer between 0 and 255 intended as a status code (with 0 meaning "success" and nonzero meaning "error"), and you can set that return value by calling Kernel#exit.
I used to have the same question myself when I started coding. If you have a closer look at your code you can see why it doesn't print anything. You are actually no asking it in your code. Imagine having a huge script of thousands of lines and you want to execute it. You would have millions of pointless outputs if ruby myscript.rb worked the same way as the REPLs.
In addition, if you do want it to work that way, you can just do require the script inside the REPL session ( require_relative 'your_script' ) and then if you call your function hello it will work the way you describe.
I can use the ruval gem. It evaluates each statement and returns its value.
$ ruval names.rb
def hello(names)
names.map {|name| "#{name} is awesome!"}
end
=> hello
hello(["mark", "tony", "scott"])
=> ["mark is awesome!", "tony is awesome!", "scott is awesome!"]
I am trying to use an external program from within Scala that accepts its input from the standard input stream.
Code roughly the equivalent to the following runs on Linux but raises an exception when run on Windows (tested on Windows 7). The exception's description states that the echo command cannot be found.
def invokeProgram(data : String) {
import scala.sys.process._
val cmdEcho = Seq("echo", data)
val cmdProgram = Seq("program")
println((cmdEcho #| cmdProgram).!!)
}
The code works correctly on Linux but fails as described on Windows, yet the echo command is common to both platforms and its usage is syntactically the same for my purposes. Is this as simple as echo not being in PATH? Is there even a separate echo.exe on Windows or is it bundled into something else? The invoked program can be made to accept its input from a temporary file which is what I will fall back to if I cannot resolve this issue.
The difference is this:
dcs#shadowfax:~$ which echo
/bin/echo
That is, on Unix the echo command is actually a binary, though most shells implement it as a builtin. On Windows, on the other hand, there's no binary called echo.exe (or echo.com, etc). It's solely a builtin command of the shell.
You don't need to use echo at all. Instead, use the #< method of ProcessBuilder. Here is an example:
import java.io.ByteArrayInputStream
import scala.sys.process._
val data = "hello"
val is = new ByteArrayInputStream(data.getBytes)
"cat" #< is ! //complicated way to print hello
When writing more than a trivial script in bash, I often wonder how to make the code testable.
It is typically hard to write tests for bash code, due to the fact that it is low on functions that take a value and return a value, and high on functions that check and set some aspect in the environment, modify the file-system, invoke a program, etc. - functions that depend on the environment or have side effects. Thus the setup and test code become much more complicated than the code they test.
For example, consider a simple function to test:
function add_to_file() {
local f=$1
cat >> $f
sort -u $f -o $f
}
Test code for this function might consist of:
add_to_file.before:
foo
bar
baz
add_to_file.after:
bar
baz
foo
qux
And test code:
function test_add_to_file() {
cp add_to_file.{before,tmp}
add_to_file add_to_file.tmp
cmp add_to_file.{tmp,after} && echo pass || echo fail
rm add_to_file.tmp
}
Here 5 lines of code are tested by 6 lines of test code and 7 lines of data.
Now consider a slightly more complicated case:
function distribute() {
local file=$1 ; shift
local hosts=( "$#" )
for host in "${hosts[#]}" ; do
rsync -ae ssh $file $host:$file
done
}
I can't even say how to start write a test for that...
So, is there a good way to do TDD in bash scripts, or should I give up and put my efforts elsewhere?
So here is what I learned:
There are some testing frameworks written in bash and for bash,
however...
It is not so much that Bash is not suitable for TDD (although some
other languages come to mind that are a better fit), but the
typical tasks that Bash is used for (Installation, System
configuration), that are hard to write tests for, and in
particularly hard to setup the test.
The poor data structure support in Bash makes it hard to separate
logic from side-effect, and indeed there is typically little logic
in Bash scripts. That makes it hard to break scripts into
testable chunks. There are some functions that can be tested, but
that is the exception, not the rule.
Function are a good thing (tm), but they can only go so far.
Nested functions can be even better, but they are also limited.
At the end of the day, with major effort some coverage can be
obtained, but it will test the less interesting part of the code,
and will keep the bulk of the testing as a good (or bad) old manual
testing.
Meta: I decided to answer (and accept) my own question, because I was unable to choose between Sinan Ünür's (voted up) and mouviciel's (voted up) answers that where equally useful and insightful. I want to note Stefano Borini's answer, that although not impressed me initially, I learned to appreciate it over time. Also his design patterns or best practices for shell scripts answer (voted up) referred above was useful.
If you are writing code at the same time with tests, try to make it high on functions that don't use anything besides their parameters and don't modify environment. That is, if your function might as well run in a subshell, then it will be easy to test. It takes some arguments and outputs something to stdout, or to a file, or maybe it does something on the system, but caller does not feel side effects.
Yes, you will end up with big chain of functions passing down some WORKING_DIR variable that might as well be global, but this is minor inconvenience comparing to the task of tracking what does each function read and modify. Enabling unit tests is just a free bonus too.
Try to minimize cases where you need output. A little subshell abuse will go long way to keeping things nicely separated (at the expense of performance).
Instead of linear structure, where functions are called, set some environment, then other ones are called, all pretty much on one level, try to go for deep call tree with minimum data going back. Returning stuff in bash is inconvenient if you adopt self-imposed abstinence from global vars...
From an implementation point of view, I suggest shUnit2 or bats.
From a practical point of view, I suggest not to give up. I use TDD on bash scripts and I confirm that it is worth the effort.
Of course, I get about twice as many lines of test than of code but with complex scripts, efforts in testing are a good investment. This is true in particular when your client changes its mind near the end of the project and modifies some requirements. Having a regression test suite is a big aid in changing complex bash code.
If you code a bash program large enough to require TDD, you are using the wrong language.
I suggest you to read my previous post on best practices in bash programming, you will probably find something useful to make your bash program testable, but my statement above stays.
Design patterns or best practices for shell scripts
Writing what Meszaros calls consumer tests is hard in any language. Another approach is to verify the behavior of commands such as rsync manually, then write unit tests to prove specific functionality without hitting the network. In this slightly-modified example, $run is used to print the side-effects if the script is run with the keyword "test"
function distribute {
local file=$1 ; shift
for host in $# ; do
$run rsync -ae ssh $file $host:$file
done
}
if [[ $1 == "test" ]]; then
run="echo"
else
distribute schedule.txt $*
exit 0
fi
#
# Built-in self-tests
#
output=$(mktemp)
expected=$(mktemp)
set -e
trap "rm $got $expected" EXIT
distribute schedule.txt login1 login2 > $output
cat << EOF > $expected
rsync -ae ssh schedule.txt login1:schedule.txt
rsync -ae ssh schedule.txt login2:schedule.txt
EOF
diff $output $expected
echo -n '.'
echo; echo "PASS"
You might want to take a look at cucumber/aruba. Did quite a nice job for me.
Additionally, you can stub just about everything you want by doing something like this:
#
# code.sh
#
some_function_calling_some_external_binary()
{
if ! external_binary action_1; then
# ...
fi
if ! external_binary action_2; then
# ...
fi
}
#
# test.sh
#
# now for the test, simply stub your external binary:
external_binary()
{
if [ "$#" = "action_1" ]; then
# stub action_1
elif [ "$#" = "action_2" ]; then
# stub action_2
else
external_binary $#
fi
}
The advanced bash scripting guide has an example of an assert function but here is a simpler and more flexible assert function - just use eval of $* to test any condition.
assert() {
if ! eval $* ; then
echo
echo "===== Assertion failed: \"$*\" ====="
echo "File \"$0\", line:$LINENO line:${BASH_LINENO[*]}"
echo line:$(caller 0)
exit 99
fi
}
# e.g. USAGE:
assert [[ $r == 42 ]]
assert "((r==42))"
BASH_LINENO and caller bash builtin are bash shell specific.
take a look at Outthentic framework - it is designed to create scenarios which runs any Bash code and then analyze the stdout using formal DSL, it's pretty easy to build any Tdd/blackbox tests suite upon this tool.