Assume that I have programs P0, P1, ...P(n-1) for some n > 0. How can I easily redirect the output of program Pi to program P(i+1 mod n) for all i (0 <= i < n)?
For example, let's say I have a program square, which repeatedly reads a number and than prints the square of that number, and a program calc, which sometimes prints a number after which it expects to be able to read the square of it. How do I connect these programs such that whenever calc prints a number, square squares it returns it to calc?
Edit: I should probably clarify what I mean with "easily". The named pipe/fifo solution is one that indeed works (and I have used in the past), but it actually requires quite a bit of work to do properly if you compare it with using a bash pipe. (You need to get a not yet existing filename, make a pipe with that name, run the "pipe loop", clean up the named pipe.) Imagine you could no longer write prog1 | prog2 and would always have to use named pipes to connect programs.
I'm looking for something that is almost as easy as writing a "normal" pipe. For instance something like { prog1 | prog2 } >&0 would be great.
After spending quite some time yesterday trying to redirect stdout to stdin, I ended up with the following method. It isn't really nice, but I think I prefer it over the named pipe/fifo solution.
read | { P0 | ... | P(n-1); } >/dev/fd/0
The { ... } >/dev/fd/0 is to redirect stdout to stdin for the pipe sequence as a whole (i.e. it redirects the output of P(n-1) to the input of P0). Using >&0 or something similar does not work; this is probably because bash assumes 0 is read-only while it doesn't mind writing to /dev/fd/0.
The initial read-pipe is necessary because without it both the input and output file descriptor are the same pts device (at least on my system) and the redirect has no effect. (The pts device doesn't work as a pipe; writing to it puts things on your screen.) By making the input of the { ... } a normal pipe, the redirect has the desired effect.
To illustrate with my calc/square example:
function calc() {
# calculate sum of squares of numbers 0,..,10
sum=0
for ((i=0; i<10; i++)); do
echo $i # "request" the square of i
read ii # read the square of i
echo "got $ii" >&2 # debug message
let sum=$sum+$ii
done
echo "sum $sum" >&2 # output result to stderr
}
function square() {
# square numbers
read j # receive first "request"
while [ "$j" != "" ]; do
let jj=$j*$j
echo "square($j) = $jj" >&2 # debug message
echo $jj # send square
read j # receive next "request"
done
}
read | { calc | square; } >/dev/fd/0
Running the above code gives the following output:
square(0) = 0
got 0
square(1) = 1
got 1
square(2) = 4
got 4
square(3) = 9
got 9
square(4) = 16
got 16
square(5) = 25
got 25
square(6) = 36
got 36
square(7) = 49
got 49
square(8) = 64
got 64
square(9) = 81
got 81
sum 285
Of course, this method is quite a bit of a hack. Especially the read part has an undesired side-effect: termination of the "real" pipe loop does not lead to termination of the whole. I couldn't think of anything better than read as it seems that you can only determine that the pipe loop has terminated by try to writing write something to it.
A named pipe might do it:
$ mkfifo outside
$ <outside calc | square >outside &
$ echo "1" >outside ## Trigger the loop to start
This is a very interesting question. I (vaguely) remember an assignment very similar in college 17 years ago. We had to create an array of pipes, where our code would get filehandles for the input/output of each pipe. Then the code would fork and close the unused filehandles.
I'm thinking you could do something similar with named pipes in bash. Use mknod or mkfifo to create a set of pipes with unique names you can reference then fork your program.
My solutions uses pipexec (Most of the function implementation comes from your answer):
square.sh
function square() {
# square numbers
read j # receive first "request"
while [ "$j" != "" ]; do
let jj=$j*$j
echo "square($j) = $jj" >&2 # debug message
echo $jj # send square
read j # receive next "request"
done
}
square $#
calc.sh
function calc() {
# calculate sum of squares of numbers 0,..,10
sum=0
for ((i=0; i<10; i++)); do
echo $i # "request" the square of i
read ii # read the square of i
echo "got $ii" >&2 # debug message
let sum=$sum+$ii
done
echo "sum $sum" >&2 # output result to stderr
}
calc $#
The command
pipexec [ CALC /bin/bash calc.sh ] [ SQUARE /bin/bash square.sh ] \
"{CALC:1>SQUARE:0}" "{SQUARE:1>CALC:0}"
The output (same as in your answer)
square(0) = 0
got 0
square(1) = 1
got 1
square(2) = 4
got 4
square(3) = 9
got 9
square(4) = 16
got 16
square(5) = 25
got 25
square(6) = 36
got 36
square(7) = 49
got 49
square(8) = 64
got 64
square(9) = 81
got 81
sum 285
Comment: pipexec was designed to start processes and build arbitrary pipes in between. Because bash functions cannot be handled as processes, there is the need to have the functions in separate files and use a separate bash.
Named pipes.
Create a series of fifos, using mkfifo
i.e fifo0, fifo1
Then attach each process in term to the pipes you want:
processn < fifo(n-1) > fifon
I doubt sh/bash can do it.
ZSH would be a better bet, with its MULTIOS and coproc features.
A command stack can be composed as string from an array of arbitrary commands
and evaluated with eval. The following example gives the result 65536.
function square ()
{
read n
echo $((n*n))
} # ---------- end of function square ----------
declare -a commands=( 'echo 4' 'square' 'square' 'square' )
#-------------------------------------------------------------------------------
# build the command stack using pipes
#-------------------------------------------------------------------------------
declare stack=${commands[0]}
for (( COUNTER=1; COUNTER<${#commands[#]}; COUNTER++ )); do
stack="${stack} | ${commands[${COUNTER}]}"
done
#-------------------------------------------------------------------------------
# run the command stack
#-------------------------------------------------------------------------------
eval "$stack"
Related
I have a for loop in bash that writes values to a file. However, because there are a lot of values, the process takes a long time, which I think can be saved by improving the code.
nk=1152
nb=24
for k in $(seq 0 $((nk-1))); do
for i in $(seq 0 $((nb-1))); do
for j in $(seq 0 $((nb-1))); do
echo -e "$k\t$i\t$j"
done
done
done > file.dat
I've moved the output action to after the entire loop is done rather than echo -e "$k\t$i\t$j" >> file.dat to avoid opening and closing the file many times. However, the speed the script writes to the file is still rather slow, ~ 10kbps.
Is there a better way to improve the IO?
Many thanks
Jacek
It looks like the seq calls are fairly punishing since that is a separate process. Try this just using shell math instead:
for ((k=0;k<=$nk-1;k++)); do
for ((i=0;i<=$nb-1;i++)); do
for ((j=0;j<=$nb-1;j++)); do
echo -e "$k\t$i\t$j"
done
done
done > file.dat
It takes just 7.5s on my machine.
Another way is to compute the sequences just once and use them repeatedly, saving a lot of shell calls:
nk=1152
nb=24
kseq=$(seq 0 $((nk-1)))
bseq=$(seq 0 $((nb-1)))
for k in $kseq; do
for i in $bseq; do
for j in $bseq; do
echo -e "$k\t$i\t$j"
done
done
done > file.dat
This is not really "better" than the first option, but it shows how much of the time is spent spinning up instances of seq versus actually getting stuff done.
Bash isn't always the best for this. Consider this Ruby equivalent which runs in 0.5s:
#!/usr/bin/env ruby
nk=1152
nb=24
nk.times do |k|
nb.times do |i|
nb.times do |j|
puts "%d\t%d\t%d" % [ k, i, j ]
end
end
end
What is the most time consuming is calling seq in a nested loop. Keep in mind that each time you call seq it loads command from disk, fork a process to run it, capture the output, and store the whole output sequence into memory.
Instead of calling seq you could use an arithmetic loop:
#!/usr/bin/env bash
declare -i nk=1152
declare -i nb=24
declare -i i j k
for ((k=0; k<nk; k++)); do
for (( i=0; i<nb; i++)); do
for (( j=0; j<nb; j++)); do
printf '%d\t%d\t%d\n' "$k" "$i" "$j"
done
done
done > file.dat
Running seq in a subshell consumes most of the time.
Switch to a different language that provides all the needed features without shelling out. For example, in Perl:
#!/usr/bin/perl
use warnings;
use strict;
use feature qw{ say };
my $nk = 1152;
my $nb = 24;
for my $k (0 .. $nk - 1) {
for my $i (0 .. $nb - 1) {
for my $j (0 .. $nb - 1) {
say "$k\t$i\t$j"
}
}
}
The original bash solution runs for 22 seconds, the Perl one finishes in 0.1 seconds. The output is identical.
#Jacek : I don't think the I/O is the problem, but the number of child processes spawned. I would store the result of the seq 0 $((nb-1)) into an array and loop over the array, i.e.
nb_seq=( $(seq 0 $((nb-1)) )
...
for i in "${nb_seq[#]}"; do
for j in "${nb_seq[#]}"; do
seq is bad) once i've done this function special for this case:
$ que () { printf -v _N %$1s; _N=(${_N// / 1}); printf "${!_N[*]}"; }
$ que 10
0 1 2 3 4 5 6 7 8 9
And you can try to write first all to a var and then whole var into a file:
store+="$k\t$i\t$j\n"
printf "$store" > file
No. it's even worse like that)
I have the following file
Durand 12 9 14
Lucas 8 11 4
Martin 9 12 1
I need to display the name and the average of the three other with a function. The function part is easy.
I thought I could get line by line with:
head -i notes | tail -1
and then put the result of the command in a table in order to access it
table=(head -i notes | tail -1)
echo "${table[0]} averge : moy ${table[1]} ${table[2]} ${table[3]}"
You might use three important concepts to approach a problem like this.
Iterate over a file
Store values as variables
Do math to variables
A good way to read a file line by line is with a while loop:
while read line; do echo $line; done < notes
Notice how we use a file redirect < to treat the file as standard input. read consumes one full line at a time. Let's expand on that in order to store separate variables.
while read name a b c; do echo $name $a $b $c; done < notes
Now let's get math involved. You could use an external program like bc, but that's inefficient if we don't need floating point math (decimals). Bash has math built in!
while read name a b c; do echo $name $(( (a + b + c) / 3 )); done < notes
Like you said, the function part is easy :)
awk one liner:
awk '{print $1, ($2+$3+$4)/3}' notes
Say I want to iterate over two lists of letters and numbers.
A B C D and seq 1 100.
How can I iterate over letters along with numbers but not as in nested for-loop? So it would be A1B2C3D4 A5B6C7D8 ...
What I've tried so far: nested for-loop and & done don't seem to be of any help, since they produce either A1 B1 C1 D1 A2 B2... or inconsistent results of parallel execution.
Also it feels like a very basic parallel loop, so no need for a detailed explanation or actual code: ANY ANSWER mentioning link to docs or the conventional name of such sequence would be immediately accepted.
The following script generates your expected output with a leading space:
Script
for i in {1..100}; do
IFS= read c
printf %s "$c$i"
done < <(yes $' A\nB\nC\n\D')
Output
A1B2C3D4 A5B6C7D8 A9B10C11D12 A13B14C15D16 A17B18C19D20 A21B22C23D24 A25B26C27D28 A29B30C31D32 A33B34C35D36 A37B38C39D40 A41B42C43D44 A45B46C47D48 A49B50C51D52 A53B54C55D56 A57B58C59D60 A61B62C63D64 A65B66C67D68 A69B70C71D72 A73B74C75D76 A77B78C79D80 A81B82C83D84 A85B86C87D88 A89B90C91D92 A93B94C95D96 A97B98C99D100
Explanation
To read the sequence 1 2 3 ... 100 in its full length, we need to repeat the sequence A B C D over and over again. yes is a command that repeats its argument ad infinitum. yes x prints
x
x
x
...
To let yes print something different in every line, we use a trick. $' A\nB\nC\nD' is a string that contains linebreaks ($'' is a so called bash ansi-c quote). yes $' A\nB\nC\nD' will print
A
B
C
D
A
B
...
Instead of printing to the console, we want to consume the text later. To this end, we could write yes ... | someCommand or someCommand < <(yes ...) which has some advantages over a pipe. The latter is called process substitution. Note that for ...; done is also just one command. The redirected stdin can be read from anywhere inside the for loop.
#!/bin/bash
# ASCII code for A
A=65
# Loop from 1 to 100
for ii in $( seq 1 100 )
do
# Compute ASCII code with using modulo
code=$(( (ii-1) % 4 + A ))
# Print letter
printf "\x$(printf %x $code)"
# Print number
echo $ii
done
This question already has answers here:
call bash script and fill input data from another bash script
(2 answers)
Closed 4 years ago.
I have this bash script that runs calculus operations for me. It starts off with "read"...
How can I make Script A enter a value into Script B, read the output and dismiss Script B again?
Example:
#ScriptA.sh
a=12
b=4
[open Script B, enter $a and $b and read the result]
echo "The result is [the result of Script B]."
#ScriptB.sh
read x y
echo $(($x+$y))
Desired Output:
bash ScriptA.sh
The result is 16.
Of course it's about more complex maths but you get the idea. Note that, for convenience purposes, I don't want to change the structure of Script B (read x y). But I hope that there are some guys here that can solve this problem.
Thanks in advance.
You should do something like this:
#!/bin/bash
a=12
b=4
result=$(echo $a $b | ./script_b.sh)
echo "the result is $result"
Script B should work like -bc does.
Example:
echo `echo "4^3.5" | -scriptb.sh`
[result]
Edit: I just came up with a part of the solution by myself and thought I'd share it:
# ScriptA.sh
echo `echo "44 33" | bash ScriptB.sh`
# ScriptB.sh
read x y
echo $(($x+$y))
Output:
bash ScriptA.sh
77
The next problem is my ScriptB.sh looks a little more like this:
# ScriptB.sh
until [[ 1 = 2 ]]; do
echo Enter x and y
read x y
if [[ x = q ]]; then
break 1
fi
echo $(($x+$y))
done
This is in order to allow multiple inputs, if I want to use ScriptB manually. If I let ScriptA use ScriptB in the above mentioned way the output looks like this:
bash ScriptA.sh
b.sh: line 9: +: syntax error: operand expected (error token is "+")
Enter x and y 77 Enter x and y
It seems to be the case that after ScriptA inputs 44 and 33 and hits enter, like it should, but it hits enter right away a second time triggering the syntax error message and ending ScriptB. This is suboptimal, because in the case of the real ScriptB it will enter a "(standard_in) 1: parse error"-chain, resulting in no result at all. The solution to this problem would be by teaching ScriptA to read what ScriptB promts as result and ending it right after this. Or making it enter "q" as a second input instead of just hitting enter.
Edit 2:
Ok. Got it. Script A should look like this in order to work as desired:
e=2.7182818285
pi=3.141
a=$(printf "$e $pi \n q \n" | bash ScriptB.sh)
a=${a:14:20}
echo $a
I'm working with an existing script which was written a bit messily. Setting up a loop with all of the spaghetti code could make a bigger headache than I want to deal with in the near term. Maybe when I have more time I can clean it up but for now, I'm just looking for a simple fix.
The script deals with virtual disks on a xen server. It reads multipath output and asks if particular LUNs should be formatted in any way based on specific criteria. However, rather than taking that disk path and inserting it, already formatted, into a configuration file, it simply presents every line in the format
'phy:/dev/mapper/UUID,xvd?,w',
UUID, of course, is an actual UUID.
The script actually presents each of the found LUNs in this format expecting the user to copy and paste them into the config file replacing each ? with a letter in sequence. This is tedious at best.
There are several ways to increment a number in bash. Among others:
var=$((var+1))
((var+=1))
((var++))
Is there a way to do the same with characters which doesn't involve looping over the entire alphabet such that I could easily "increment" the disk assignment from xvda to xvdb, etc?
To do an "increment" on a letter, define the function:
incr() { LC_CTYPE=C printf "\\$(printf '%03o' "$(($(printf '%d' "'$1")+1))")"; }
Now, observe:
$ echo $(incr a)
b
$ echo $(incr b)
c
$ echo $(incr c)
d
Because, this increments up through ASCII, incr z becomes {.
How it works
The first step is to convert a letter to its ASCII numeric value. For example, a is 97:
$ printf '%d' "'a"
97
The next step is to increment that:
$ echo "$((97+1))"
98
Or:
$ echo "$(($(printf '%d' "'a")+1))"
98
The last step is convert the new incremented number back to a letter:
$ LC_CTYPE=C printf "\\$(printf '%03o' "98")"
b
Or:
$ LC_CTYPE=C printf "\\$(printf '%03o' "$(($(printf '%d' "'a")+1))")"
b
Alternative
With bash, we can define an associative array to hold the next character:
$ declare -A Incr; last=a; for next in {b..z}; do Incr[$last]=$next; last=$next; done; Incr[z]=a
Or, if you prefer code spread out over multiple lines:
declare -A Incr
last=a
for next in {b..z}
do
Incr[$last]=$next
last=$next
done
Incr[z]=a
With this array, characters can be incremented via:
$ echo "${Incr[a]}"
b
$ echo "${Incr[b]}"
c
$ echo "${Incr[c]}"
d
In this version, the increment of z loops back to a:
$ echo "${Incr[z]}"
a
How about an array with entries A-Z assigned to indexes 1-26?
IFS=':' read -r -a alpharray <<< ":A:B:C:D:E:F:G:H:I:J:K:L:M:N:O:P:Q:R:S:T:U:V:W:X:Y:Z"
This has 1=A, 2=B, etc. If you want 0=A, 1=B, and so on, remove the first colon.
IFS=':' read -r -a alpharray <<< "A:B:C:D:E:F:G:H:I:J:K:L:M:N:O:P:Q:R:S:T:U:V:W:X:Y:Z"
Then later, where you actually need the letter;
var=$((var+1))
'phy:/dev/mapper/UUID,xvd${alpharray[$var]},w',
The only problem is that if you end up running past 26 letters, you'll start getting blanks returned from the array.
Use a Bash 4 Range
You can use a Bash 4 feature that lets you specify a range within a sequence expression. For example:
for letter in {a..z}; do
echo "phy:/dev/mapper/UUID,xvd${letter},w"
done
See also Ranges in the Bash Wiki.
Here's a function that will return the next letter in the range a-z. An input of 'z' returns 'a'.
nextl(){
((num=(36#$(printf '%c' $1)-9) % 26+97));
printf '%b\n' '\x'$(printf "%x" $num);
}
It treats the first letter of the input as a base 36 integer, subtracts 9, and returns the character whose ordinal number is 'a' plus that value mod 26.
Use Jot
While the Bash range option uses built-ins, you can also use a utility like the BSD jot utility. This is available on macOS by default, but your mileage may vary on Linux systems. For example, you'll need to install athena-jot on Debian.
More Loops
One trick here is to pre-populate a Bash array and then use an index variable to grab your desired output from the array. For example:
letters=( "" $(jot -w %c 26 a) )
for idx in 1 26; do
echo ${letters[$idx]}
done
A Loop-Free Alternative
Note that you don't have to increment the counter in a loop. You can do it other ways, too. Consider the following, which will increment any letter passed to the function without having to prepopulate an array:
increment_var () {
local new_var=$(jot -nw %c 2 "$1" | tail -1)
if [[ "$new_var" == "{" ]]; then
echo "Error: You can't increment past 'z'" >&2
exit 1
fi
echo -n "$new_var"
}
var="c"
var=$(increment_var "$var")
echo "$var"
This is probably closer to what the OP wants, but it certainly seems more complex and less elegant than the original loop recommended elsewhere. However, your mileage may vary, and it's good to have options!