It seems to me, there are several ways to return a value from a Bash function.
Approach 1: Use a "local-global" variable, which is defined as local in the caller:
func1() {
a=10
}
parent1() {
local a
func1
a=$(($a + 1))
}
Approach 2: Use command substitution:
func2() {
echo 10
}
parent2() {
a=$(func2)
a=$(($a + 1))
}
How much speedup could one expect from using approach 1 over approach2?
And, I know that it is not good programming practice to use global variables like in approach 1, but could it at some point be justified due to efficiency considerations?
The single most expensive operation in shell scripting is forking. Any operation involving a fork, such as command substitution, will be 1-3 orders of magnitude slower than one that doesn't.
For example, here's a straight forward approach for a loop that reads a bunch of generated files on the form of file-1234 and strips out the file- prefix using sed, requiring a total of three forks (command substitution + two stage pipeline):
$ time printf "file-%s\n" {1..10000} |
while read line; do n=$(echo "$line" | sed -e "s/.*-//"); done
real 0m46.847s
Here's a loop that does the same thing with parameter expansion, requiring no forks:
$ time printf "file-%s\n" {1..10000} |
while read line; do n=${line#*-}; done
real 0m0.150s
The forky version takes 300x longer.
Therefore, the answer to your question is yes: if efficiency matters, you have solid justification for factoring out or replacing forky code.
When the fork count is constant with respect to the input (or it's too messy to make it constant), and the code is still too slow, that's when you should rewrite it in a faster language.
surely approach 1 is much faster than approach 2 because it has not any interrupt (which in turn may need several OS kernel crossing to service) and has only one memory access!!!
Related
I had looked on overflow and exchange, but I can't seem to find an answer for this. I am currently trying to make a recursive fibonacci function. However when I try to test it (using command line entry) it keeps returning the error
fork: retry: Resource temporarily unavailable
I had found a post saying it had to do with shell resource limits, but I felt like that probably wasn't my issue since I've seen separate posts with recursive functions being able to call their function fine enough.
I had broken up the operation into parts to see what exactly was happening, but I can't find a specific problem - it may have to do with how i'm calling my function, but I'm not sure, either.
I can pass the parameter I want just fine, though. It just prints the same error for whatever number I put in. If I enter 5, it echoes the fork error five times. It returns, but doesn't return the values...
For specification, I currently use Bash Version 4.4.20(1)
function fib_r
{
int=$1
for ((i=1; i<=int; i++))
do
f1=$(fib_r $((int-1)))
f2=$(fib_r $((int-2)))
fibo=$((f1+f2))
done
}
What I want to achieve is when you enter a number on command line, It does calculate that number, however it shows the calculated number at each step, rather than return the final value from start to end:
An example output:
1 1
2 1
3 2
4 3
5 5
6 8
7 13
8 21
9 34
10 55
This Shellcheck-clean code fixes a few problems with the code in the question:
function fib_r
{
local -r n=$1
if (( n == 0 )); then
echo 0
elif (( n == 1 )); then
echo 1
else
local -r f1=$(fib_r "$((n-1))")
local -r f2=$(fib_r "$((n-2))")
echo "$((f1+f2))"
fi
return 0
}
It adds code for base cases (0 and 1) to stop the recursion. The original code had no base cases so the recursion continued until resources were exhausted.
The function echos the result so it is available to the caller. Command substitution ($(command)) is usually only useful if the command prints its result(s) to standard output.
The loop was removed because this is supposed to be a recursive function (and the loop wasn't useful).
local is used to make variables local to the function so they won't clash with variables used elsewhere in a program that uses the function. The -r (readonly) option is used because the variables never need to be changed within the function and it prevents them being accidentally changed by other functions.
The variable name int was changed to n because that is more conventional for functions like this (and int seems really strange to people who know C, or related programming languages).
Note that this function is very slow. That is partly because it uses command substitution (which runs an expensive sub-process every time) to return results, but mostly because this particular recursive algorithm is very inefficient (exponential complexity, see Computational complexity of Fibonacci Sequence). Much faster recursive implementations are possible.
The question was updated to request a function that prints all of the Fibonacci numbers up to a given number. This is a recursive function that does that:
function fib_r
{
local -r n=$1
local -r depth=${2-1}
local -r f1=${3-1}
local -r f2=${4-0}
if (( depth <= n )); then
printf '%2d %d\n' "$depth" "$f1"
fib_r "$n" "$((depth+1))" "$((f1+f2))" "$f1"
fi
return 0
}
This uses a much more efficient algorithm (O(n)), so it can calculate all of the Fibonacci numbers that can be represented by a 64-bit integer in a fraction of a second. Run fib_r 92 to do that.
I have learned Bash process substitution from Bash's man page. Unfortunately, my unskilled usage of the feature is ugly.
DEV=<(some commands that produce lines of data) && {
while read -u ${DEV##*/} FIELD1 FIELD2 FIELD3; do
some commands that consume the fields of a single line of data
done
}
Do skilled programmers have other ways to do this?
If an executable sample is desired, try this:
DEV=<(echo -ne "Cincinnati Hamilton Ohio\nAtlanta Fulton Georgia\n") && {
while read -u ${DEV##*/} FIELD1 FIELD2 FIELD3; do
echo "$FIELD1 lies in $FIELD2 County, $FIELD3."
done
}
Sample output:
Cincinnati lies in Hamilton County, Ohio.
Atlanta lies in Fulton County, Georgia.
In my actual application, the "some commands" are more complicated, but the above sample captures the essence of the question.
Process substitution <() is required. Alternatives to process substitution would not help.
Redirect into the loop's stdin with the operator <.
while read city county state; do
echo "$city lies in $county County, $state."
done < <(echo -ne "Cincinnati Hamilton Ohio\nAtlanta Fulton Georgia\n")
Output:
Cincinnati lies in Hamilton County, Ohio.
Atlanta lies in Fulton County, Georgia.
Note that in this example, a pipe works just as well.
echo -ne "Cincinnati Hamilton Ohio\nAtlanta Fulton Georgia\n" |
while read city county state
do
echo "$city lies in $county County, $state."
done
Also, uppercase variable names should be reserved for environment variables (like PATH) and other special variables (like RANDOM). And descriptive variable names are always good.
There are few alternative that will be portable. The 'right' choice depends on the specific case. In particular, it depends on the time to produce the input data, and the size of the input. In particular:
If it takes lot of time to process the data, you want to get parallel processing between the data generation, and the 'while' loop. This will result in incremental processing, and not having to wait for all the input data processing, before starting output data processing.
If the input is very large (and does not fit into a shell variable), you might not have a choice but to force an actual pipe. This is also true when the data is binary, Unicode, or similar - where bash variable will not work.
Mapping to the original question - PRODUCE = echo Cincinnati ..., and CONSUME - echo "$city ..."
For the trivial case (small input, fast produce/consume), the following will work. Bash will run them SEQUNIALLY: PRODUCE then CONSUME.
while read ... ; do
CONSUME
done <<< "$(PRODUCE)"
For the complex case (large input, or slow produce & consume), the following can be use to request PARALLEL execution
while read ... ; do
CONSUME
done < <(PRODUCE)
For the PRODUCE code is complex (loops, conditional, etc), or long (multiple lines), consider moving it into a function, instead of in-lining them into the loop command.
function produce {
PRODUCE
}
while read ... ; do
CONSUME
done < <(produce)
So I have two files in my directory that contain a number in each of them. I want to make a script that calculates the average of these two numbers. How would I write it? Would this be correct?
avg=$((${<file1.txt}-${<file2.txt})/2)
Your example does not work. Furthermore, your formula is probably incorrect. Here are two options without unnecessary cat:
avg=$(( (`<file1.txt` + `<file2.txt`) / 2 ))
or
avg=$(( ($(<file1.txt) + $(<file2.txt)) / 2 ))
I find the first one more readable though. Also be warned: this trivial approach will cause problems when your files contain more than just the plain numbers.
EDIT:
I should have noted that the first syntactical/legacy option which uses the backticks (` `) is no longer recommended and should be avoided. You can read more about the WHY here. Thanks at mklement0 for the link!
EDIT2:
According to Eric, the values are floating point numbers. You can't do this directly in bash because only integer numbers are supported. You have to use a little helper:
avg=$(bc <<< "( $(<file1.txt) + $(<file2.txt) ) / 2")
or maybe easier to understand
avg=$(echo "( $(<file1.txt) + $(<file2.txt) ) / 2" | bc)
For those who might wonder what bc is (see man bc):
bc is a language that supports arbitrary precision numbers with
interactive execution of statements.
Here is another alternative since perl is usually installed by default:
avg=$(perl -e 'print( ($ARGV[0] + $ARGV[1]) / 2 )' -- $(<file1.txt) $(<file2.txt))
You'll want to use a command substitution:
avg=$(($(cat file1.txt)-$(cat file2.txt)/2))
However, Bash is a pretty bad language for doing maths (at least unless it's completely integer maths). You might want to look into bc or a "real" language like Python.
I have a file system containing PNG images. The layout of the filesystem is: ZOOM/X/Y.png where ZOOM, X, and Y are all integers.
I need to change the names of the PNG files. Basically, I need to convert Y from its current value to 2^ZOOM-Y-1. I've written a bash script to accomplish this task. However, I suspect it can be optimized substantially. (I also suspect that I may have been better off writing it in Perl, but that is another story.)
Here is the script. Is this about as good as it gets? Or can the performance be optimized? Are there tools I can use that would profile the script for me and tell me where I'm spending all my execution time?
#!/bin/bash
tiles=`ls -d */*/*`
for oldPath in $tiles
do
oldY=`basename -s .png $oldPath`
zoomX=`dirname $oldPath`
zoom=`echo $zoomX | sed 's#\([^\]\)/.*#\1#'`
newY=`echo 2^$zoom-$oldY-1|bc`
mv ${zoomX}/${oldY}.png ${zoomX}/${newY}.png
done
for oldpath in */*/*
do
x=$(basename "$oldpath" .png)
zoom_y=$(dirname "$oldpath")
y=$(basename "$zoom_y")
ozoom=$(dirname "$zoom_y")
nzoom=$(echo "2^$zoom-$y-1" | bc)
mv "$oldpath" $nzoom/$y/$x.png
done
This avoids using sed. I like basename and dirname. However, you can also use bash (and Korn) shell notations such as:
y=${zoom_y#*/}
ozoom=${zoom_y%/*}
You might be able to do it all without invoking basename or dirname at all.
REWRITE due to misunderstanding of the formula and the updated var names. Still no subprocesses apart from mv and ls.
#!/bin/bash
tiles=`ls -d */*/*`
for thisPath in $tiles
do
thisFile=${thisPath#*/*/}
oldY=${thisFile%.png}
zoomX=${thisPath%/*}
zoom=${thisPath%/*/*}
newY=$(((1<<zoom) - oldY - 1))
mv ${zoomX}/${oldY}.png ${zoomX}/${newY}.png
done
It's likely that the overall throughput of your rename is limited by the filesystem. Choosing the right filesystem and tuning it for this sort of operation would speed up the overall job much more than tweaking the script.
If you optimize the script you'll probably see less CPU consumed but the same total duration. Since forking off the various subprocesses (basename, dirname, sed, bc) are probably more significant than the actual work you are probably right that a perl implementation would use less CPU because it can do all of those operations internally (including the mv).
I see 3 improvements I would do, if it was my script. Whether they have an huge impact - I don't think so.
But you should avoid as hell parsing the output of ls. Maybe this directory is very predictable, from the things found inside, but if I read your script correctly, you can use the globbing with for directly:
for thisPath in */*/*
repeatedly, $(cmd) is better than cmd with the deprecated backticks, which aren't nestable.
thisDir=$(dirname $thisPath)
arithmetic in bash directly:
newTile=$((2**$zoom-$thisTile-1))
as long as you don't need floating point, or output is getting too big.
I don't get the sed-part:
zoom=`echo $zoomX | sed 's#\([^\]\)/.*#\1#'`
Is there something missing after the backslash? A second one? You're searching for something which isn't a backslash, followed by a slash-something? Maybe it could be done purely in bash too.
one precept of computing credited to Donald Knuth is, "don't optimize too early." Scripts run pretty fast and 'mv' operations(as long as they're not going across filesystems where you're really copying it to another disk and then deleting the file) are pretty fast as well, as all the filesystem has to do in most cases is just rename the file or change its parentage.
Probably where it's spending most of its time is in that intial 'ls' operation. I suspect you have ALOT of files. There isn't much that can be done there. Doing it another language like perl or python is going to face the same hurdle. However you might be able to get more INTELLIGENCE and not limit yourself to 3 levels(//*).
Each time I have downloaded a new copy of Rakudo Perl 6, I have run the following expression just to get an idea of its current performance:
say [+] 1 .. 100000;
And the speeds have been increasing, but each time, there is a noticeable delay (several seconds) for the calculation. As a comparison, something like this in Perl 5 (or other interpreted languages) returns almost instantly:
use List::Util 'sum';
print sum(1 .. 100000), "\n";
or in Ruby (also nearly instant):
(1 .. 100000).inject(0) {|sum,x| sum+x}
Rewriting the expression as a Perl6 loop ends up being about twice as fast as reducing the range, but it is still a very noticeable delay (more than a second) for the simple calculation:
my $sum;
loop (my $x = 1; $x <= 100000; $x++) {$sum += $x}
So my question is, what aspects of the Perl6 implementation are causing these performance issues? And should this improve with time, or is this overhead an unfortunate side effect of the "everything is an object" model that Perl6 is using?
And lastly, what about the loop construct is faster than the [+] reduction operator? I would think that the loop would result in more total ops than the reduction.
EDIT:
I'd accept both mortiz's and hobbs's answers if I could. That everything is a being handled as a method call more directly answers why [+] is being slow, so that one gets it.
Another thing you have to understand about the lack of optimization is that it's compounded. A large portion of Rakudo is written in Perl 6. So for example the [+] operator is implemented by the method Any.reduce (called with $expression set to &infix:<+>), which has as its inner loop
for #.list {
#args.push($_);
if (#args == $arity) {
my $res = $expression.(#args[0], #args[1]);
#args = ($res);
}
}
in other words, a pure-perl implementation of reduce, which itself is being run by Rakudo. So not only is the code you can see not getting optimized, the code that you don't see that's making your code run is also not getting
optimized. Even instances of the + operator are actually method calls, since although the + operator on Num is implemented by Parrot, there's nothing yet in Rakudo to recognize that you've got two Nums and optimize away the method call, so there's a full dynamic dispatch before Rakudo finds multi sub infix:<+>(Num $a, Num $b) and realizes that all it's really doing is an 'add' opcode. It's a reasonable excuse for being 100-1000x slower than Perl 5 :)
Update 8/23/2010
More information from Jonathan Worthington on the kinds of changes that need to happen with the Perl 6 object model (or at least Rakudo's conception of it) to make things fast while retaining Perl 6's "everything is method calls" nature.
Update 1/10/2019
Since I can see that this is still getting attention... over the years, Rakudo/MoarVM have gotten JIT, inlining, dynamic specialization, and tons of work by many people optimizing every part of the system. The result is that most of those method calls can be "compiled out" and have nearly zero runtime cost. Perl 6 scores hundreds or thousands of times faster on many benchmarks than it did in 2010, and in some cases it's faster than Perl 5.
In the case of the sum-to-100,000 problem that the question started with, Rakudo 2018.06 is still a bit slower than perl 5.26.2:
$ time perl -e 'use List::Util 'sum'; print sum(1 .. 100000), "\n";' >/dev/null
real 0m0.023s
user 0m0.015s
sys 0m0.008s
$ time perl6 -e 'say [+] 1 .. 100000;' >/dev/null
real 0m0.089s
user 0m0.107s
sys 0m0.022s
But if we amortize out startup cost by running the code 10,000 times, we see a different story:
$ time perl -e 'use List::Util 'sum'; for (1 .. 10000) { print sum(1 .. 100000), "\n"; }' > /dev/null
real 0m16.320s
user 0m16.317s
sys 0m0.004s
$ time perl6 -e 'for 1 .. 10000 { say [+] 1 .. 100000; }' >/dev/null
real 0m0.214s
user 0m0.245s
sys 0m0.021s
perl6 uses a few hundred more milliseconds than perl5 on startup and compilation, but then it figures out how to do the actual summation around 70 times faster.
There are really various reasons why Rakudo is so slow.
The first and maybe most important reason is that Rakudo doesn't do any optimizations yet. The current goals are more explore new features, and to become more robust. You know, they say "first make it run, then make it right, then make it fast".
The second reason is that parrot doesn't offer any JIT compilation yet, and the garbage collector isn't the fastest. There are plans for a JIT compiler, and people are working on it (the previous one was ripped out because it was i386 only and a maintenance nightmare). There are also thoughts of porting Rakudo to other VMs, but that'll surely wait till after end of July.
In the end, nobody can really tell how fast a complete, well-optimized Perl 6 implementation will be until we have one, but I do expect it to be much better than now.
BTW the case you cited [+] 1..$big_number could be made to run in O(1), because 1..$big_number returns a Range, which is introspectable. So you can use a sum formula for the [+] Range case. Again it's something that could be done, but that hasn't been done yet.
It certainly isn't because everything is an object, because that's true in a number of other languages too (like Ruby). There's no reason why Perl 6 would have to be magnitudes slower than other languages like Perl 5 or Ruby, but the fact is that Rakudo is not as mature as perl or CRuby. There hasn't been much speed optimization yet.
Considering that now your test case is optimized to an O(1) algorithm that returns nearly instantly, and that it seems almost like there are several optimizations a week;
I expect quite an performance improvement all around.
$ perl6 -e 'say [+] 1..10**1000; say now - INIT now'
5000000000000000000000000000000000000000000000 ...
0.007447
Even if that wasn't special-cased for ranges it is still quite a bit faster than it was.
It now does your test calculation in less than a fifth of a second.
$ perl6 -e 'say [+] (1..100000).list; say now - INIT now'
5000050000
0.13052975
I submitted these to Fefe's language competition in December 2008. wp.pugs.pl is a literal translation of the Perl 5 example, wp.rakudo.pl is far more sixier. I have two programs because the two implement a different subset of the spec. Build information is outdated meanwhile. The sources:
#!/usr/bin/env pugs
# Pugs: <http://pugs.blogs.com/> <http://pugscode.org/>
# prerequisite: ghc-6.8.x, not 6.10.x
# svn co http://svn.pugscode.org/pugs/
# perl Makefile.PL
# make
# if build stops because of haskeline, do:
# $HOME/.cabal/bin/cabal update ; $HOME/.cabal/bin/cabal install haskeline
# learn more: <http://jnthn.net/papers/2008-tcpw-perl64danoob-slides.pdf>
my %words;
for =<> {
for .split {
%words{$_}++
}
}
for (sort { %words{$^b} <=> %words{$^a} }, %words.keys) {
say "$_ %words{$_}"
}
#!/usr/bin/env perl6
# Rakudo: <http://rakudo.org/> <http://www.parrot.org/download>
# svn co http://svn.perl.org/parrot/trunk parrot
# perl Configure.pl
# make perl6
# Solution contributed by Frank W. & Moritz Lenz
# <http://use.perl.org/~fw/journal/38055>
# learn more: <http://jnthn.net/papers/2008-tcpw-perl64danoob-slides.pdf>
my %words;
$*IN.lines.split(/\s+/).map: { %words{$_}++ };
for %words.pairs.sort: { $^b.value <=> $^a.value } -> $pair {
say $pair
}
These were the results in 2008:
$ time ./wp.pugs.pl < /usr/src/linux/COPYING > foo
real 0m2.529s
user 0m2.464s
sys 0m0.064s
$ time ./wp.rakudo.pl < /usr/src/linux/COPYING > foo
real 0m32.544s
user 0m1.920s
sys 0m0.248s
Today:
$ time ./wp.pugs.pl < /usr/src/linux/COPYING > foo
real 0m5.105s
user 0m4.898s
sys 0m0.096s
$ time ./wp.rakudo.pl < /usr/src/linux/COPYING > foo
Divide by zero
current instr.: '' pc -1 ((unknown file):-1)
Segmentation fault
real 0m3.236s
user 0m0.447s
sys 0m0.080s
Late additions: The crash has been dealt with at Why do I get 'divide by zero` errors when I try to run my script with Rakudo?. The Rakudo program is inefficient, see comments below and http://justrakudoit.wordpress.com/2010/06/30/rakudo-and-speed/.