I have the following bash script and want to run other script from that and capture the results:
#!/bin/bash
while read line; do
echo "exit" | out=`python file.py`
if [[ $out == *"WORD"* ]]; then
echo $line >> out.txt
fi
done<$1
But this is not working for me. In each iteration out wouln't get value...
echo "exit" | out=`python file.py`
Should be something like (send "exit" to the result of assigning the output of file.py to out - seems odd):
echo "exit" && out=`python file.py`
or (send "exit" as input to file.py and assign output to out):
out=`echo "exit" | python file.py`
depends on what you're trying to achieve.
A pipeline runs in a subshell, so variable assignments within it aren't visible in the parent shell. It should be:
out=$(echo exit | python file.py)
Now the whole pipeline is inside the command substitution, but the variable assignment is in the original shell.
Keep python execution outside loop since that is not dependent upon any loop variable:
#!/bin/bash
# initialize output file
> out.txt
# execute python script
out=$(echo "exit" | python file.py)
# loop
while read -r line; do
[[ "$out" == *"WORD"* ]] && echo "$line" >> out.txt
done < "$1"
Also quoting seem to be missing at many point that I have added.
Related
Running another script in bash script while loop runs but the loop breaks!
N.B. The script I mentioned just loops over files in current directory and just run mpirun.
Here's my bash script:
#!/bin/bash
np="$1"
bin="$2"
ref="$3"
query="$4"
word_size="$5"
i=1;
input="$query"
while read line; do
echo $line
if [[ "${line:0:1}" == ">" ]] ; then
header="$line"
echo "$header" >> seq_"${i}".fasta
else
seq="$line"
echo "$seq" >> seq_"${i}".fasta
if ! (( i % 5)) ; then
./run.sh $np $bin $ref $word_size
^^^^^^^^
#for filename in *.fasta; do
# mpirun -np "${np}" "${bin}" -d "${ref}" -ql "${filename}" -k "${word_size}" -b > log
# rm $filename
#done
fi
((i++))
fi
done < $input
The problem is that your run.sh script is passing no parameters to mpirun. That script passes the variables ${np} ${bin} ${ref} ${filename} ${word_size} to mpirun, but those variables are local to your main script and are undefined in run.sh. You could export those variables in the main script so that they are available to all child processes, but a better solution would be to use positional parameters in run.sh:
for filename in *.fasta; do
mpirun -np "${1}" "${2}" -d "${3}" -ql "${4}" -k "${5}" -b > log
rm $filename
done
I don't know about mpirun, but if you have anything inside your loop that reads from stdin, the loop will break.
The following script calls another program reading its output in a while loop (see Bash - How to pipe input to while loop and preserve variables after loop ends):
while read -r col0 col1; do
# [...]
done < <(other_program [args ...])
How can I check for the exit code of other_program to see if the loop was executed properly?
Note: ls -d / /nosuch is used as an example command below, because it fails (exit code 1) while still producing stdout output (/) (in addition to stderr output).
Bash v4.2+ solution:
ccarton's helpful answer works well in principle, but by default the while loop runs in a subshell, which means that any variables created or modified in the loop will not be visible to the current shell.
In Bash v4.2+, you can change this by turning the lastpipe option on, which makes the last segment of a pipeline run in the current shell;
as in ccarton's answer, the pipefail option must be set to have $? reflect the exit code of the first failing command in the pipeline:
shopt -s lastpipe # run the last segment of a pipeline in the current shell
shopt -so pipefail # reflect a pipeline's first failing command's exit code in $?
ls -d / /nosuch | while read -r line; do
result=$line
done
echo "result: [$result]; exit code: $?"
The above yields (stderr output omitted):
result: [/]; exit code: 1
As you can see, the $result variable, set in the while loop, is available, and the ls command's (nonzero) exit code is reflected in $?.
Bash v3+ solution:
ikkachu's helpful answer works well and shows advanced techniques, but it is a bit cumbersome.
Here is a simpler alternative:
while read -r line || { ec=$line && break; }; do # Note the `|| { ...; }` part.
result=$line
done < <(ls -d / /nosuch; printf $?) # Note the `; printf $?` part.
echo "result: [$result]; exit code: $ec"
By appending the value of $?, the ls command's exit code, to the output without a trailing \n (printf $?), read reads it in the last loop operation, but indicates failure (exit code 1), which would normally exit the loop.
We can detect this case with ||, and assign the exit code (that was still read into $line) to variable $ec and exit the loop then.
On the off chance that the command's output doesn't have a trailing \n, more work is needed:
while read -r line ||
{ [[ $line =~ ^(.*)/([0-9]+)$ ]] && ec=${BASH_REMATCH[2]} && line=${BASH_REMATCH[1]};
[[ -n $line ]]; }
do
result=$line
done < <(printf 'no trailing newline'; ls /nosuch; printf "/$?")
echo "result: [$result]; exit code: $ec"
The above yields (stderr output omitted):
result: [no trailing newline]; exit code: 1
At least one way would be to redirect the output of the background process through a named pipe. This would allow to pick up its PID and then get the exit status through waiting on the PID.
#!/bin/bash
mkfifo pipe || exit 1
(echo foo ; exit 19) > pipe &
pid=$!
while read x ; do echo "read: $x" ; done < pipe
wait $pid
echo "exit status of bg process: $?"
rm pipe
If you can use a direct pipe (i.e. don't mind the loop being run in a subshell), you could use Bash's PIPESTATUS, which contains the exit codes of all commands in the pipeline:
(echo foo ; exit 19) | while read x ; do
echo "read: $x" ; done;
echo "status: ${PIPESTATUS[0]}"
A simple way is to use the bash pipefail option to propagate the first error code from a pipeline.
set -o pipefail
other_program | while read x; do
echo "Read: $x"
done || echo "Error: $?"
Another way is to use coproc (requires 4.0+).
coproc other_program [args ...]
while read -r -u ${COPROC[0]} col0 col1; do
# [...]
done
wait $COPROC_PID || echo "Error exit status: $?"
coproc frees you from having to setup asynchronicity and stdin/stdout redirection that you'd otherwise need to do in an equivalent mkfifo.
Please explain to me why the very last echo statement is blank? I expect that XCODE is incremented in the while loop to a value of 1:
#!/bin/bash
OUTPUT="name1 ip ip status" # normally output of another command with multi line output
if [ -z "$OUTPUT" ]
then
echo "Status WARN: No messages from SMcli"
exit $STATE_WARNING
else
echo "$OUTPUT"|while read NAME IP1 IP2 STATUS
do
if [ "$STATUS" != "Optimal" ]
then
echo "CRIT: $NAME - $STATUS"
echo $((++XCODE))
else
echo "OK: $NAME - $STATUS"
fi
done
fi
echo $XCODE
I've tried using the following statement instead of the ++XCODE method
XCODE=`expr $XCODE + 1`
and it too won't print outside of the while statement. I think I'm missing something about variable scope here, but the ol' man page isn't showing it to me.
Because you're piping into the while loop, a sub-shell is created to run the while loop.
Now this child process has its own copy of the environment and can't pass any
variables back to its parent (as in any unix process).
Therefore you'll need to restructure so that you're not piping into the loop.
Alternatively you could run in a function, for example, and echo the value you
want returned from the sub-process.
http://tldp.org/LDP/abs/html/subshells.html#SUBSHELL
The problem is that processes put together with a pipe are executed in subshells (and therefore have their own environment). Whatever happens within the while does not affect anything outside of the pipe.
Your specific example can be solved by rewriting the pipe to
while ... do ... done <<< "$OUTPUT"
or perhaps
while ... do ... done < <(echo "$OUTPUT")
This should work as well (because echo and while are in same subshell):
#!/bin/bash
cat /tmp/randomFile | (while read line
do
LINE="$LINE $line"
done && echo $LINE )
One more option:
#!/bin/bash
cat /some/file | while read line
do
var="abc"
echo $var | xsel -i -p # redirect stdin to the X primary selection
done
var=$(xsel -o -p) # redirect back to stdout
echo $var
EDIT:
Here, xsel is a requirement (install it).
Alternatively, you can use xclip:
xclip -i -selection clipboard
instead of
xsel -i -p
I got around this when I was making my own little du:
ls -l | sed '/total/d ; s/ */\t/g' | cut -f 5 |
( SUM=0; while read SIZE; do SUM=$(($SUM+$SIZE)); done; echo "$(($SUM/1024/1024/1024))GB" )
The point is that I make a subshell with ( ) containing my SUM variable and the while, but I pipe into the whole ( ) instead of into the while itself, which avoids the gotcha.
#!/bin/bash
OUTPUT="name1 ip ip status"
+export XCODE=0;
if [ -z "$OUTPUT" ]
----
echo "CRIT: $NAME - $STATUS"
- echo $((++XCODE))
+ export XCODE=$(( $XCODE + 1 ))
else
echo $XCODE
see if those changes help
Another option is to output the results into a file from the subshell and then read it in the parent shell. something like
#!/bin/bash
EXPORTFILE=/tmp/exportfile${RANDOM}
cat /tmp/randomFile | while read line
do
LINE="$LINE $line"
echo $LINE > $EXPORTFILE
done
LINE=$(cat $EXPORTFILE)
I read a lot about passing piping stdin to bash read function, but nothing seems to work for my bash version!!
GNU bash, version 3.2.51(1)-release (x86_64-suse-linux-gnu)
I have a bash script that in some point asks the user "yes/no" with variable CONTINUEQUESTION:
echo "Do you want to continue? (yes/no):"
read CONTINUEQUESTION
tmp=$(tr '[:upper:]' '[:lower:]' <<<$CONTINUEQUESTION)
if [[ "$tmp" != 'y' && "$tmp" != 'yes' ]]; then
echo "Aborting because of input '$CONTINUEQUESTION'"
exit
fi
I would like to pipe a "yes or no" to this question without user input!
Yes i know i could use expect, but i don't prefer it in this case.
So i tried several things:
CONTINUEQUESTION='yes'
echo $CONTINUEQUESTION | ./myscript.sh
Aborting because of input ''
./myscript.sh <<< "$CONTINUEQUESTION"
Aborting because of input ''
...and many other, nothing worked!?
O.k. now I did a bit revers thinking and find out that the below line causes the problem with the pipe...because when i remarked it out all the below answers are working just fine, but not when this line is executed:
running=`ssh root#${HOSTNAME} 'su - root -c "/bin/tools list | grep \"system running\"" 2>&1'`
But, i need this line before the read! What do i need to reverse the 2>&1????
My script look like this and is working without this try to over come the user intervantion:
LIST_FILE_NAME=$1
STILL_RUNNING=0
running=`ssh root#${HOSTNAME} 'su - root -c "cat '$LIST_FILE_NAME' | grep \"system running\"" 2>&1'`
if [[ $running =~ .*running.* ]]; then
STILL_RUNNING=1
echo "NODE $NODE running stop before continuing."
fi
if [ $STILL_RUNNING -eq 1 ]; then
echo "Aborting system was still running!"
exit 1
fi
echo "Do you want to continue? (yes/no):"
read CONTINUEQUESTION
tmp=$(tr '[:upper:]' '[:lower:]' <<<$CONTINUEQUESTION)
if [[ "$tmp" != 'y' && "$tmp" != 'yes' ]]; then
echo "Aborting because of input '$CONTINUEQUESTION'"
exit
fi
echo "o.k."
4 points:
list.log can have a line with "system running" or "system notrunning"
if list.log has a line with "system notrunning" than the bash script continue towards the question
at the question i never got it right to inject the 'y' or 'yes' so the bash aborts because of input ''
i execute this like: ./myscript.sh list list.log (normal way)
This bash runs well if the user interacts at the question!
Thanks for you time!!!
Consider this variation as well:
#!/bin/bash
read -p "Do you want to continue? (yes/no): " CONTINUEQUESTION
if [[ $CONTINUEQUESTION != [Yy] && $CONTINUEQUESTION != [Yy][Ee][Ss] ]]; then
echo "Aborting because of input '$CONTINUEQUESTION'."
exit
fi
Tested with:
bash script.sh <<< yes
If it doesn't work, show the output of:
bash -x script.sh <<< yes
Your line
$CONTINUEQUESTION='yes'
shoul really be
CONTINUEQUESTION='yes'
I am not sure then that your are feeding stdin with the word 'yes'. You could add an echo after the read to be sure.
You can use heredoc:
bash -ex ./myscript.sh << 'EOF'
yes
EOF
Search for Here Documents in man bash.
EDIT: Based on comments you can use this ssh command:
running=$(ssh -t -t root#${HOSTNAME} "grep 'system running' \"$LIST_FILE_NAME\"")
Please explain to me why the very last echo statement is blank? I expect that XCODE is incremented in the while loop to a value of 1:
#!/bin/bash
OUTPUT="name1 ip ip status" # normally output of another command with multi line output
if [ -z "$OUTPUT" ]
then
echo "Status WARN: No messages from SMcli"
exit $STATE_WARNING
else
echo "$OUTPUT"|while read NAME IP1 IP2 STATUS
do
if [ "$STATUS" != "Optimal" ]
then
echo "CRIT: $NAME - $STATUS"
echo $((++XCODE))
else
echo "OK: $NAME - $STATUS"
fi
done
fi
echo $XCODE
I've tried using the following statement instead of the ++XCODE method
XCODE=`expr $XCODE + 1`
and it too won't print outside of the while statement. I think I'm missing something about variable scope here, but the ol' man page isn't showing it to me.
Because you're piping into the while loop, a sub-shell is created to run the while loop.
Now this child process has its own copy of the environment and can't pass any
variables back to its parent (as in any unix process).
Therefore you'll need to restructure so that you're not piping into the loop.
Alternatively you could run in a function, for example, and echo the value you
want returned from the sub-process.
http://tldp.org/LDP/abs/html/subshells.html#SUBSHELL
The problem is that processes put together with a pipe are executed in subshells (and therefore have their own environment). Whatever happens within the while does not affect anything outside of the pipe.
Your specific example can be solved by rewriting the pipe to
while ... do ... done <<< "$OUTPUT"
or perhaps
while ... do ... done < <(echo "$OUTPUT")
This should work as well (because echo and while are in same subshell):
#!/bin/bash
cat /tmp/randomFile | (while read line
do
LINE="$LINE $line"
done && echo $LINE )
One more option:
#!/bin/bash
cat /some/file | while read line
do
var="abc"
echo $var | xsel -i -p # redirect stdin to the X primary selection
done
var=$(xsel -o -p) # redirect back to stdout
echo $var
EDIT:
Here, xsel is a requirement (install it).
Alternatively, you can use xclip:
xclip -i -selection clipboard
instead of
xsel -i -p
I got around this when I was making my own little du:
ls -l | sed '/total/d ; s/ */\t/g' | cut -f 5 |
( SUM=0; while read SIZE; do SUM=$(($SUM+$SIZE)); done; echo "$(($SUM/1024/1024/1024))GB" )
The point is that I make a subshell with ( ) containing my SUM variable and the while, but I pipe into the whole ( ) instead of into the while itself, which avoids the gotcha.
#!/bin/bash
OUTPUT="name1 ip ip status"
+export XCODE=0;
if [ -z "$OUTPUT" ]
----
echo "CRIT: $NAME - $STATUS"
- echo $((++XCODE))
+ export XCODE=$(( $XCODE + 1 ))
else
echo $XCODE
see if those changes help
Another option is to output the results into a file from the subshell and then read it in the parent shell. something like
#!/bin/bash
EXPORTFILE=/tmp/exportfile${RANDOM}
cat /tmp/randomFile | while read line
do
LINE="$LINE $line"
echo $LINE > $EXPORTFILE
done
LINE=$(cat $EXPORTFILE)