Trying to create submit script to SGE - bash

I'm trying to edit my working bash script to an SGE script in order to submit it as a job to the cluster.
Currently I have:
#!/bin/bash
# Perform fastqc on files in a specified directory.
for ((j=1; j <=17; j++))
do
directory=/data4/una/batch"$j"/
files=$""$directory"/*.fastq.gz"
batch=$"batch_"$j""
outfile=$""$batch"_submit_script.sh"
echo "#!/bin/bash">>$outfile;
echo "# Your job name">>$outfile;
echo "# -N $batch">>$outfile;
echo "# The job should be placed into the queue 'all.q'">>$outfile;
echo "#$ -q all.q">>$outfile;
echo "# Running in the current working directory">>$outfile;
echo "#$ -cwd">>$outfile;
echo "">>$outfile;
echo "# Export some necessary environment variables">>$outfile;
echo "#$ -S /bin/bash">>$outfile;
echo "#$ -v PATH">>$outfile;
echo "#$ -v LD_LIBRARY_PATH">>$outfile;
echo "#$ -v PYTHONPATH">>$outfile;
echo "# Finally, put your command here">>$outfile;
echo "">>$outfile;
echo "#$ for i in $files;">>$outfile;
echo "#$ do;">>$outfile;
echo "#$ fastqc -f fastq -o /data4/una/test/fastq/$i;">>$outfile;
echo "#$done">>$outfile;
echo "">>$outfile;
qsub $outfile;
done
But I'm getting an error:
Unable to read script file because of error: ERROR! invalid option argument "-f"
But
fastqc -f fastq -o /data4/una/test/fastq/$i
is a totally valid line in my bash script.
Thoughts?
Thanks!

It actually was poor formatting for my loop that was causing this error. I didn't need to start those lines with #$ at all, so those lines become:
echo "for i in $files;">>$outfile;
echo "do">>$outfile;
echo " fastqc -f fastq -o /data4/una/test/fastqc $i">>$outfile;
echo "done">>$outfile;
echo "">>$outfile;
qsub $outfile;

Related

Bash script while loop breaks after running another script

Running another script in bash script while loop runs but the loop breaks!
N.B. The script I mentioned just loops over files in current directory and just run mpirun.
Here's my bash script:
#!/bin/bash
np="$1"
bin="$2"
ref="$3"
query="$4"
word_size="$5"
i=1;
input="$query"
while read line; do
echo $line
if [[ "${line:0:1}" == ">" ]] ; then
header="$line"
echo "$header" >> seq_"${i}".fasta
else
seq="$line"
echo "$seq" >> seq_"${i}".fasta
if ! (( i % 5)) ; then
./run.sh $np $bin $ref $word_size
^^^^^^^^
#for filename in *.fasta; do
# mpirun -np "${np}" "${bin}" -d "${ref}" -ql "${filename}" -k "${word_size}" -b > log
# rm $filename
#done
fi
((i++))
fi
done < $input
The problem is that your run.sh script is passing no parameters to mpirun. That script passes the variables ${np} ${bin} ${ref} ${filename} ${word_size} to mpirun, but those variables are local to your main script and are undefined in run.sh. You could export those variables in the main script so that they are available to all child processes, but a better solution would be to use positional parameters in run.sh:
for filename in *.fasta; do
mpirun -np "${1}" "${2}" -d "${3}" -ql "${4}" -k "${5}" -b > log
rm $filename
done
I don't know about mpirun, but if you have anything inside your loop that reads from stdin, the loop will break.

using cronic to reduce email notifications in a wrapper script for successful backups

I've been using cronic to silence emails from cron jobs when the job is successful. I'm trying to customize it so when a response code is 0 and the error output matches a string of "mount: /VessRAID/RH: /dev/sde1 already mounted on /VessRAID/RH.", to not send an email. Below is the script, then the contents of the email then my attempt at trying to suppress the email which is not working. Any idea what I may be doing wrong?
#!/bin/bash
# Cronic v3 - cron job report wrapper
# Copyright 2007-2016 Chuck Houpt. No rights reserved, whatsoever.
# Public Domain CC0: http://creativecommons.org/publicdomain/zero/1.0/
set -eu
TMP=$(mktemp -d)
OUT=$TMP/cronic.out
ERR=$TMP/cronic.err
TRACE=$TMP/cronic.trace
set +e
"$#" >$OUT 2>$TRACE
RESULT=$?
set -e
PATTERN="^${PS4:0:1}\\+${PS4:1}"
if grep -aq "$PATTERN" $TRACE
then
! grep -av "$PATTERN" $TRACE > $ERR
else
ERR=$TRACE
fi
if [ $RESULT -ne 0 -o -s "$ERR" ]
then
echo "Cronic detected failure or error output for the command:"
echo "$#"
echo
echo "RESULT CODE: $RESULT"
echo
echo "ERROR OUTPUT:"
cat "$ERR"
echo
echo "STANDARD OUTPUT:"
cat "$OUT"
if [ $TRACE != $ERR ]
then
echo
echo "TRACE-ERROR OUTPUT:"
cat "$TRACE"
fi
fi
rm -rf "$TMP"
Here is what the email notification looks like:
Cronic detected failure or error output for the command:
/usr/local/sbin/reg-backup-cronic.sh daily
RESULT CODE: 0
ERROR OUTPUT:
mount: /VessRAID/RH: /dev/sde1 already mounted on /VessRAID/RH.
STANDARD OUTPUT:
/dev/sde1 on /VessRAID/RH type ext4 (rw,relatime)
Here is my attempt at a wrapper script:
#!/bin/bash
/usr/local/sbin/reg-backup.sh $1
CODE=$?
err=$TRACE
if [[ $CODE -eq 0 && $err = "mount: /VessRAID/RH: /dev/sde1 already mounted on /VessRAID/RH." ]]
then
exit $CODE
fi
Alas the emails continue.
Hat tip to the creator of cronic, Chuck Houpt, for cluing me in to an answer, which was to look at the original script and why the error is happening. Case-sensitivity got the best of me:
if mount | grep Vessraid; then
echo starting $1 backup >> /var/log/vessraid.log
Notice the case in VessRAID should have been:
if mount | grep VessRAID; then
echo starting $1 backup >> /var/log/vessraid.log
Now emails only happen when there really is an error.

I'm not able to run the MVN goals through shell script

I just used the below shell script in Post Build activity in JENKINS "Execute Shell" cmd
#!/bin/bash
mvn --version
export M2_HOME=/opt/maven/maven-3.3.3 # your Mavan home path
export PATH=$PATH:$M2_HOME/bin
mvn --version
echo $HOME
echo $WORKSPACE
file=$WORKSPACE/XXXX/XXXX-reports
cd $file
cp XXXX-1.html test.html
fail=`grep "test-method.*FAIL" results.xml | sed -e 's/^.*test-
instance-name="\(.*\) ' | tr '\n' ','`
echo $fail
count=0
while [ $count -lt 5 ]
do
if [ ! -z "$fail" -a "$fail" != " " ];
then
echo `$M2_HOME/bin/mvn clean test -f ../../pom.xml -DInclude=${fail}`
cp XXXX-report.html ReReport_$count.html
retry=`expr $count + 1`
fi
done
In the above shell script I'm trying to run the mvn command and continuing to run 5 times to increase pass count in UNIT script execution.
But It's not working. It's failed to execute the maven goals. So, I just comment the executable and added echo to debug.
But no luck.
Any leads
I suggest that you use this following script that modifies yours. Add your parameters there.
The product log file: mvn_jenkins.log will allow you to easily find problems as suggested by mjuarez:
#!/bin/bash
MVNCMD=$(command -v mvn)
MVNPARAMS=" -version" # replace with your appropriate params
LOGFILE="mvn_jenkins.log"
COUNT=0
while [ $COUNT -lt 5 ]; do
echo "Execute: $MVNCMD $MVNPARAMS" >> $LOGFILE
echo "-------" >> $LOGFILE
$MVNCMD $MVNPARAMS 2>&1>> $LOGFILE
echo "" >> $LOGFILE
COUNT=$((COUNT + 1))
done
Explanation:
MVNCMD=$(command -v mvn): put the full qualified name of maven execute file in MVNCMD var
$MVNCMD $MVNPARAMS 2>&1>> $LOGFILE: Execute mvn command with parameters and redirect error output messages (2) and regular output messages (1) to the log file.
About your maven parameters I don't understand why you use -DskipTests=true.

Center Justify/align ASCII art in bash

I was just working on my bash project and
I want the header ascii art to automatically adjust at the center whenever the terminal executes the script in whatever resolution. Is it possible mates?
Following is my code:
#!/bin/bash
clear
echo
echo -e "\t\t1▄██████▄#0000▄████████0000▄████████11▄██████▄ ";
echo -e "\t\t███0000███111███0110███111███1011███1███#0000██";
echo -e "\t\t███0001███111███0111███111███1100█▀11███#ffff██";
echo -e "\t\t███0010███11▄███▄▄▄▄██▀11▄███▄▄▄11111███#0000██";
echo -e "\t\t███0011███1▀▀███▀▀▀▀▀111▀▀███▀▀▀11111███#ffff██";
echo -e "\t\t███0100███1▀███████████111███1101█▄11███#0000██";
echo -e "\t\t███0101███111███1000███111███1110███1███#ffff██";
echo -e "\t\t1▀██████▀ffff███1001███111██████████11▀██████▀1";
echo -e "\n\n"
You can use the COLUMNS environment variable that returns the width of the terminal.
banner_width=46
indent=$(( (COLUMNS - banner_width) / 2 ))
prefix=''
for ((i=1; i<=indent; i++)) ; do
prefix+=' '
done
echo
echo -e "${prefix}1▄██████▄#0000▄████████0000▄████████11▄██████▄ ";

Curl command in bash script error

I'm trying to run a curl command within a bash script and I'm coming across the following error:
curl: (26) failed creating formpost data
When I echo the curl command and run that from the command line, that has no problems running. I've ensured there are read permissions for everyone on the file and the folder it's in, as well as using relative and absolute paths.
Here's a sample of the script:
#!/bin/bash
usage(){
echo "usage: <command> options: <u|p|f> "
}
while getopts u:p:f: option
do
case "${option}"
in
u) USERNAME=${OPTARG};;
p) PORT=${OPTARG};;
f) FILE=$OPTARG;;
esac
done
if [ $OPTIND -eq 1 ]; then
#echo "No options were passed";
usage
exit
fi
echo "Please enter your password: "
read PASSWORD
TEST="curl -u $USERNAME:$PASSWORD -F file=#\"$FILE\" -F force=true -F install=true http://localhost:$PORT/test.jsp"
echo $TEST
$TEST
echo $?
Thanks!

Resources