Why is exit my status valid in command line but not within bash script? (Bash) - bash

There are a few layers here, so bear with me.
My docker-container ssh -c"echo 'YAY!'; exit 25;" command executes echo 'YAY!'; exit 25; in my docker container. It returns:
YAY
error: message=YAY!
, code=25
I need to know if the command within the container was successful, so I append the following to the command:
docker-container ssh -c"echo 'YAY!'; exit 25;" >&1 2>/tmp/stderr; cat /tmp/stderr | grep 'code=' | cut -d'=' -f2 | { read exitStatus; echo $exitStatus; }
This sends the stderr to /tmp/stderr and, with the echo $exitStatus returns:
YAY!
25
So, this is exactly what I want. I want the $exitStatus saved to a variable. My problem is, I am placing this into a bash script (GIT pre-commit) and when this exact code is executed, the exit status is null.
Here is my bash script:
# .git/hooks/pre-commit
if [ -z ${DOCKER_MOUNT+x} ];
then
docker-container ssh -c"echo 'YAY!'; exit 25;" >&1 2>/tmp/stderr; cat /tmp/stderr | grep 'code=' | cut -d'=' -f2 | { read exitStatus; echo $exitStatus; }
exit $exitStatus;
else
echo "Container detected!"
fi;

That's because you're setting the variable in a pipeline. Each command in the pipeline is run in a subshell, and when the subshell exits the variable are no longer available.
bash allows you to run the pipeline's last command in the current shell, but you also have to turn off job control
An example
# default bash
$ echo foo | { read x; echo x=$x; } ; echo x=$x
x=foo
x=
# with "lastpipe" configuration
$ set +m; shopt -s lastpipe
$ echo foo | { read x; echo x=$x; } ; echo x=$x
x=foo
x=foo
Add set +m; shopt -s lastpipe to your script and you should be good.
And as Charles comments, there are more efficient ways to do it. Like this:
source <(docker-container ssh -c "echo 'YAY!'; exit 25;" 2>&1 1>/dev/null | awk -F= '/code=/ {print "exitStatus=" $2}')
echo $exitStatus

Related

heredoc failing when piped into ssh

I am trying to do commands via ssh using a heredoc. when I use the 'EOF' style:
result=$(ssh $quiet -T ${G_IPAddressBase}${G_TGP} <<- 'EOF'
set pid=`ps -ef | grep TestPatterns | egrep -v 'grep|tail|vi|csh' | gawk '{print $2}'`
if ( "$pid" != "" ) then
echo "TestPatterns ($pid) terminated"
kill $pid
else
echo "TestPatterns down already"
endif
EOF
)
everything works as excepted because the $ and qrave marks are not expanded out.
But if I want to include local variables and use \ to stop the variable from being resolved, the \ does not stop the $ variable from resolving as expected.
result=$(ssh $quiet -T ${G_IPAddressBase}${G_TGP} <<- EOF
set pid=\`ps -ef | grep $mytaskName | grep '\/bin\/bash' | gawk '{print \$2}'\`
if ( "\$pid" == "" ) then
echo "ERROR: $mytaskName cored"
else
if ( -f "$mytaskLogpath" ) then
set agent1Done=1
else
echo "ERROR: $mytaskName didn't make a report file"
endif
endif
EOF
)
The pid is being set to blank because its looks like this on the remote side:
set pid=`ps -ef | grep TestPatterns | egrep -v 'grep|tail|vi|csh' | gawk '{print }'`
Everything that I have read says this should not be happening. If I cat << EOF > /tmp/command.csh the heredoc to a file the commands match what I expect on the remote box and work as expected.
This script is in bash, but the default shell is tcsh and that's what the remote is running so its not because the csh commands are running in a bash window on the remote login.
Any ideas what I am doing wrong?

Bash Script can run php script manually but cannot work in Cron

I have a bash script like this:
#!/bin/bash
log_file=/home/michael/bash/test.log
checkalive=checkalive.php
#declare
needRestart=0
#Check checkalive.php
is_checkalive=`ps aux | grep -v grep| grep -v "$0" | grep $checkalive| wc -l | awk '{print $1}'`
if [ $is_checkalive != "0" ] ;
then
checkaliveId=$(ps -ef | grep $checkalive | grep -v 'grep' | awk '{ printf $2 }')
echo "Service $checkalive is running. $checkaliveId"
else
echo "$checkalive OFF"
needRestart=1
fi
#NEED needRestart
if [ $needRestart == "1" ];
then
#START SERVICE
echo "Restarting services..."
/usr/bin/php5.6 /home/michael/bash/$checkalive >/dev/null 2>&1 &
echo "$checkalive..."
echo `date '+%Y-%m-%d %H:%M:%S'` " Start /home/michael/bash/$checkalive" >> $log_file
fi
I can run it manually but when I try to run it in Cron, it doesn't work for some reasons. Apparently the command:
/usr/bin/php5.6 /home/michael/bash/$checkalive >/dev/null 2>&1 &
does not work.
All of file permissions are already set to executable. Any advice?
Thank you
You have run into one of cron's most common mistakes, trying to use it like an arbitrary shell script. Cron is not a shell script and you can't do everything you can do in one, like dereferencing variables or setting arbitrary new variables.
I suggest you replace your values into the cron line and avoid usage of variables
/usr/bin/php5.6 /home/michael/bash/checkalive.php >/dev/null 2>&1 &
Also, consider removing the trailing & as it is not necessary.

How to run commands off of a pipe

I would like to run commands such as "history" or "!23" off of a pipe.
How might I achieve this?
Why does the following command not work?
echo "history" | xargs eval $1
To answer (2) first:
history and eval are both bash builtins. So xargs cannot run either of them.
xargs does not use $1 arguments. man xargs for the correct syntax.
For (1), it doesn't really make much sense to do what you are attempting because shell history is not likely to be synchronised between invocations, but you could try something like:
{ echo 'history'; echo '!23'; } | bash -i
or:
{ echo 'history'; echo '!23'; } | while read -r cmd; do eval "$cmd"; done
Note that pipelines run inside subshells. Environment changes are not retained:
x=1; echo "x=2" | while read -r cmd; do eval "$cmd"; done; echo "$x"
You can try like this
First redirect the history commands to a file (cut out the line numbers)
history | cut -c 8- > cmd.txt
Now Create this script hcmd.sh(Referred to this Read a file line by line assigning the value to a variable)
#!/bin/bash
while IFS='' read -r line || [[ -n "$line" ]]; do
echo "Text read from file: $line"
$line
done < "cmd.txt"
Run it like this
./hcmd.sh

snakemake rule calls a shell script but exits after first command

I have a shell script that works well if I just run it from command line. When I call it from a rule within snakemake it fails.
The script runs a for loop over a file of identifiers and uses those to grep the sequences from a fastq file followed by multiple sequence alignment and makes a consensus.
Here is the script. I placed some echo statements in there and for some reason it doesn't call the commands. It stops at the grep statement.
I have tried adding set +o pipefail; in the rule but that doesn't work either.
#!/bin/bash
function Usage(){
echo -e "\
Usage: $(basename $0) -r|--read2 -l|--umi-list -f|--outfile \n\
where: ... \n\
" >&2
exit 1
}
# Check argument count
[[ "$#" -lt 2 ]] && Usage
# parse arguments
while [[ "$#" -gt 1 ]];do
case "$1" in
-r|--read2)
READ2="$2"
shift
;;
-l|--umi-list)
UMI="$2"
shift
;;
-f|--outfile)
OUTFILE="$2"
shift
;;
*)
Usage
;;
esac
shift
done
# Set defaults
# Check arguments
[[ -f "${READ2}" ]] || (echo "Cannot find input file ${READ2}, exiting..." >&2; exit 1)
[[ -f "${UMI}" ]] || (echo "Cannot find input file ${UMI}, exiting..." >&2; exit 1)
#Create output directory
OUTDIR=$(dirname "${OUTFILE}")
[[ -d "${OUTDIR}" ]] || (set -x; mkdir -p "${OUTDIR}")
# Make temporary directories
TEMP_DIR="${OUTDIR}/temp"
[[ -d "${TEMP_DIR}" ]] || (set -x; mkdir -p "${TEMP_DIR}")
#RUN consensus script
for f in $( more "${UMI}" | cut -f1);do
NAME=$(echo $f)
grep "${NAME}" "${READ2}" | cut -f1 -d ' ' | sed 's/#M/M/' > "${TEMP_DIR}/${NAME}.name"
echo subsetting reads
seqtk subseq "${READ2}" "${TEMP_DIR}/${NAME}.name" | seqtk seq -A > "${TEMP_DIR}/${NAME}.fasta"
~/software/muscle3.8.31_i86linux64 -msf -in "${TEMP_DIR}/${NAME}.fasta" -out "${TEMP_DIR}/${NAME}.muscle.fasta"
echo make consensus
~/software/EMBOSS-6.6.0/emboss/cons -sequence "${TEMP_DIR}/${NAME}.muscle.fasta" -outseq "${TEMP_DIR}/${NAME}.cons.fasta"
sed -i 's/n//g' "${TEMP_DIR}/${NAME}.cons.fasta"
sed -i "s/EMBOSS_001/${NAME}.cons/" "${TEMP_DIR}/${NAME}.cons.fasta"
done
cat "${TEMP_DIR}/*.cons.fasta" > "${OUTFILE}"
Snakemake rule:
rule make_consensus:
input:
r2=get_extracted,
lst="{prefix}/{sample}/reads/cell_barcode_umi.count"
output:
fasta="{prefix}/{sample}/reads/fasta/{sample}.R2.consensus.fa"
shell:
"sh ./scripts/make_consensus.sh -r {input.r2} -l {input.lst} -f {output.fasta}"
Edit Snakemake error messages I changed some of the paths to a neutral filepath
RuleException:
CalledProcessError in line 29 of ~/user/scripts/consensus.smk:
Command ' set -euo pipefail; sh ./scripts/make_consensus.sh -r ~/user/file.extracted.fastq -l ~/user/cell_barcode_umi
.count -f ~/user/file.consensus.fa ' returned non-zero exit status 1.
File "~/user/scripts/consensus.smk", line 29, in __rule
_make_consensus
File "~/user/miniconda3/lib/python3.6/concurrent/futures/thread.py", line 56, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
If there are better ways to do this than using a shell for loop please let me know!
thanks!
Edit
Script ran as standalone: first grep
grep AGGCCGTTCT_TGTGGATG R_extracted/wgs_5_OL_debug.R2.extracted.fastq | cut -f1 -d ' ' | sed 's/#M/M/' > ./fasta/temp/AGGCCGTTCT_TGTGGATG.name
Script ran through snakemake: first 2 grep statements
grep :::::::::::::: R_extracted/wgs_5_OL_debug.R2.extracted.fastq | cut -f1 -d ' ' | sed 's/#M/M/' > ./fasta/temp/::::::::::::::.name
I'm now trying to figure out where those :::: in snakemake are coming from. All ideas welcome
It stops at the grep statement
My guess is that the grep command in make_consensus.sh doesn't capture anything. grep returns exit code 1 in such cases and the non-zero exit status propagates to snakemake. (see also Handling SIGPIPE error in snakemake)
Loosely related... There is an inconsistency between the shebang of make_consensus.sh that says the script should be executed with bash (#!/bin/bash) and the actual execution using sh (sh ./scripts/make_consensus.sh). (In practice it shouldn't make any difference since sh is probably redirected to bash anyway)

execute conky with a cron job and bash

for my script in bash, I'd like to start conky if it's not running and pick a random wallpaper
#! /bin/bash
## dependances : randomize-lines
# otherwise wont work with cron
export DISPLAY=0
while read line ; do
echo $line | grep -vqe "^#"
if [ $? -eq 0 ]; then export $line; fi
done < ~/.dbus/session-bus/$(cat /var/lib/dbus/machine-id)-$DISPLAY
# random background
pathToImage="$HOME/Images/wallpaper/"
img="`find $pathToImage -name \*.jpg | rl | tail -n 1`"
/usr/bin/gconftool -t str -s /desktop/gnome/background/picture_filename $img
# test if conky is running
if ps ax | grep -v grep | grep conky > /dev/null
then
echo "conky running"
else
echo "conky is not running"
conky
fi
if I try to run the script within a terminal
$ ~/script/wallpaper/random-background.sh
conky is not running
Conky: can't open display: 0
if I put the test before the DISPLAY=0, it'll works in a terminal but not with cron
thank you
I think that should be
export DISPLAY=:0
but that won't work for
~/.dbus/session-bus/$(cat /var/lib/dbus/machine-id)-$DISPLAY
but you could change that to
~/.dbus/session-bus/$(cat /var/lib/dbus/machine-id)-0
You missed a ":":
export DISPLAY=:0

Resources