How to process multiple command line arguments in Bash? [duplicate] - bash

This question already has answers here:
How to iterate over arguments in a Bash script
(9 answers)
How do I parse command line arguments in Bash?
(40 answers)
Closed 5 years ago.
I am having problem allowing my script to take more than three arguments. My script will take commands like this, for example:
./myscript.sh -i -v -r filename
so far if it only takes two arguments plus filename:
./myscript.sh -i -v filename
If I run the full commands, [-i] [-v] [-r] it gives this errors...
"mv: invalid option -- 'r'
Try 'mv --help' for more information."
here is my code so far....
#!/bin/bash
dirfile='.trash'
verbose=
function helpfunction()
{
echo "=============== HELP COMMAND ============="
echo ""
echo "-h | --help"
echo "-i | --interactive"
echo "-v | --verbose"
echo "-r | --recursive"
echo ""
}
if [ $# -eq 0 ]; then
echo "no commands"
exit 1
fi
option="${1}"
case ${option} in
-h | --help)
helpfunction
exit
;;
#interactive -i or --interactive
-i) FILE="${*}"
echo "File name is $FILE"
echo -n "Do you want to remove this file (y/n)? "
read answer
if echo "$answer" | grep -iq "^y" ;then
mv $FILE $dirfile
fi
;;
#verbose -v or --verbose
-v) FILE="${*}"
if [ -f "$FILE" ]; then
-v $FILE
fi
;;
#recursive -r or --recursive
-r) FILE="${*}"
if [ -d "${*}" ]; then
rm -r "$FILE"
fi
;;
#unremove -u or --unremove
-u) FILE="${*}"
for file in $dirfile*;
do
if [[ -f $file ]]; then
echo "file found"
else
echo "File not found"
fi
done
;;
#list -l or --list
-l) FILE="${*}"
for entry in "$dirfile"/*
do
echo "$entry"
done
;;
*)
echo "`basename ${0}`:usage: [-f file] | [-d directory]"
exit 1 # Command to come out of the program with status 1
;;
esac

When you say
FILE="${*}"
You're taking all the remaining arguments as one string, and assigning that to FILE. You can see this by adding 'set -x' to the top of your script:
opus$ ./myscript.sh -i -v -r filename
+ '[' 4 -eq 0 ']'
+ option=-i
+ case ${option} in
+ FILE='-i -v -r filename'
+ echo 'File name is -i -v -r filename'
File name is -i -v -r filename
+ echo -n 'Do you want to remove this file (y/n)? '
Do you want to remove this file (y/n)? + read answer
y
+ grep -iq '^y'
+ echo y
+ mv -i -v -r filename
mv: invalid option -- 'r'
Try 'mv --help' for more information.
Saying $1 gives the first argument. After you discover that, you can use shift to load the next argument as $1, and cycle through them like that.
Alternatively, you could just use the builtin getops argument parser.
Joe

You are getting error at this line
mv $FILE $dirfile
Here is the execution of your script in debug mode.
bash -x ./myscript.sh -i -v -r test.txt
+ dirfile=.trash
+ verbose=
+ '[' 4 -eq 0 ']'
+ option=-i
+ case ${option} in
+ FILE='-i -v -r test.txt'
+ echo 'File name is -i -v -r test.txt'
File name is -i -v -r test.txt
+ echo -n 'Do you want to remove this file (y/n)? '
Do you want to remove this file (y/n)? + read answer
y
+ echo y
+ grep -iq '^y'
+ mv -i -v -r test.txt .trash
mv: invalid option -- 'r'
Try 'mv --help' for more information.
-r is not a valid argument for mv.
Update your script like FILE=$4
if you want -i -v -r as optional, you can extract the last argument following this Getting the last argument passed to a shell script
You can also use ${!#} for getting the last parameter.

Related

snakemake rule calls a shell script but exits after first command

I have a shell script that works well if I just run it from command line. When I call it from a rule within snakemake it fails.
The script runs a for loop over a file of identifiers and uses those to grep the sequences from a fastq file followed by multiple sequence alignment and makes a consensus.
Here is the script. I placed some echo statements in there and for some reason it doesn't call the commands. It stops at the grep statement.
I have tried adding set +o pipefail; in the rule but that doesn't work either.
#!/bin/bash
function Usage(){
echo -e "\
Usage: $(basename $0) -r|--read2 -l|--umi-list -f|--outfile \n\
where: ... \n\
" >&2
exit 1
}
# Check argument count
[[ "$#" -lt 2 ]] && Usage
# parse arguments
while [[ "$#" -gt 1 ]];do
case "$1" in
-r|--read2)
READ2="$2"
shift
;;
-l|--umi-list)
UMI="$2"
shift
;;
-f|--outfile)
OUTFILE="$2"
shift
;;
*)
Usage
;;
esac
shift
done
# Set defaults
# Check arguments
[[ -f "${READ2}" ]] || (echo "Cannot find input file ${READ2}, exiting..." >&2; exit 1)
[[ -f "${UMI}" ]] || (echo "Cannot find input file ${UMI}, exiting..." >&2; exit 1)
#Create output directory
OUTDIR=$(dirname "${OUTFILE}")
[[ -d "${OUTDIR}" ]] || (set -x; mkdir -p "${OUTDIR}")
# Make temporary directories
TEMP_DIR="${OUTDIR}/temp"
[[ -d "${TEMP_DIR}" ]] || (set -x; mkdir -p "${TEMP_DIR}")
#RUN consensus script
for f in $( more "${UMI}" | cut -f1);do
NAME=$(echo $f)
grep "${NAME}" "${READ2}" | cut -f1 -d ' ' | sed 's/#M/M/' > "${TEMP_DIR}/${NAME}.name"
echo subsetting reads
seqtk subseq "${READ2}" "${TEMP_DIR}/${NAME}.name" | seqtk seq -A > "${TEMP_DIR}/${NAME}.fasta"
~/software/muscle3.8.31_i86linux64 -msf -in "${TEMP_DIR}/${NAME}.fasta" -out "${TEMP_DIR}/${NAME}.muscle.fasta"
echo make consensus
~/software/EMBOSS-6.6.0/emboss/cons -sequence "${TEMP_DIR}/${NAME}.muscle.fasta" -outseq "${TEMP_DIR}/${NAME}.cons.fasta"
sed -i 's/n//g' "${TEMP_DIR}/${NAME}.cons.fasta"
sed -i "s/EMBOSS_001/${NAME}.cons/" "${TEMP_DIR}/${NAME}.cons.fasta"
done
cat "${TEMP_DIR}/*.cons.fasta" > "${OUTFILE}"
Snakemake rule:
rule make_consensus:
input:
r2=get_extracted,
lst="{prefix}/{sample}/reads/cell_barcode_umi.count"
output:
fasta="{prefix}/{sample}/reads/fasta/{sample}.R2.consensus.fa"
shell:
"sh ./scripts/make_consensus.sh -r {input.r2} -l {input.lst} -f {output.fasta}"
Edit Snakemake error messages I changed some of the paths to a neutral filepath
RuleException:
CalledProcessError in line 29 of ~/user/scripts/consensus.smk:
Command ' set -euo pipefail; sh ./scripts/make_consensus.sh -r ~/user/file.extracted.fastq -l ~/user/cell_barcode_umi
.count -f ~/user/file.consensus.fa ' returned non-zero exit status 1.
File "~/user/scripts/consensus.smk", line 29, in __rule
_make_consensus
File "~/user/miniconda3/lib/python3.6/concurrent/futures/thread.py", line 56, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
If there are better ways to do this than using a shell for loop please let me know!
thanks!
Edit
Script ran as standalone: first grep
grep AGGCCGTTCT_TGTGGATG R_extracted/wgs_5_OL_debug.R2.extracted.fastq | cut -f1 -d ' ' | sed 's/#M/M/' > ./fasta/temp/AGGCCGTTCT_TGTGGATG.name
Script ran through snakemake: first 2 grep statements
grep :::::::::::::: R_extracted/wgs_5_OL_debug.R2.extracted.fastq | cut -f1 -d ' ' | sed 's/#M/M/' > ./fasta/temp/::::::::::::::.name
I'm now trying to figure out where those :::: in snakemake are coming from. All ideas welcome
It stops at the grep statement
My guess is that the grep command in make_consensus.sh doesn't capture anything. grep returns exit code 1 in such cases and the non-zero exit status propagates to snakemake. (see also Handling SIGPIPE error in snakemake)
Loosely related... There is an inconsistency between the shebang of make_consensus.sh that says the script should be executed with bash (#!/bin/bash) and the actual execution using sh (sh ./scripts/make_consensus.sh). (In practice it shouldn't make any difference since sh is probably redirected to bash anyway)

Grep is not showing results even i used fgrep and -f options

I have used the below content to fetch some values .
But the grep in the code is not showing any results.
#!/bin/bash
file=test.txt
while IFS= read -r cmd;
do
check_address=`grep -c $cmd music.cpp`
if [ $check_address -ge 1 ]; then
echo
else
grep -i -n "$cmd" music.cpp
echo $cmd found
fi
done < "$file"
Note : there are no carriage return in my text file or .sh file.
i checked using
bash -x check.sh
It is just showing
+grep -i -n "$cmd" music.cpp

Grep inside bash script not finding item

I have a script which is checking a key in one file against a key in another to see if it exists in both. However in the script the grep never returns anything has been found but on the command line it does.
#!/bin/bash
# First arg is the csv file of repo keys separated by line and in
# this manner 'customername,REPOKEY'
# Second arg is the log file to search through
log_file=$2
csv_file=$1
while read line;
do
customer=`echo "$line" | cut -d ',' -f 1`
repo_key=`echo "$line" | cut -d ',' -f 2`
if [ `grep "$repo_key" $log_file` ]; then
echo "1"
else
echo "0"
fi
done < $csv_file
The CSV file is formatted as follows:
customername,REPOKEY
and the log file is as follows:
REPOKEY
REPOKEY
REPOKEY
etc
I call the script by doing ./script csvfile.csv logfile.txt
Rather then checking output of grep command use grep -q to check its return status:
if grep -q "$repo_key" "$log_file"; then
echo "1"
else
echo "0"
fi
Also your script can be simplified to:
log_file=$2
csv_file=$1
while IFS=, read -r customer repo_key; do
if grep -q "$repo_key" "$log_file"; then
echo "1"
else
echo "0"
fi
done < "$csv_file"
use the exit status of the grep command to print 1 or 0
repo_key=`echo "$line" | cut -d ',' -f 2`
grep -q "$repo_key" $log_file
if [ $? -eq 1 ]; then
echo "1"
else
echo "0"
fi
-q supresses the output so that no output is printed
$? is the exit status of grep command 1 on successfull match and 0 on unsuccessfull
you can have a much simpler version as
grep -q "$repo_key" $log_file
echo $?
which will produce the same output

How to replace or escape <tab> characters with \t in bash script and being able to output single quotes?

In the goal to create a file from a one line (bash) command, the goal is to output the contents of any text file - in this example a bash script - and wrap each line inside a command that is able to output that same line when pasted in a Terminal window.
Example source input file:
Line 1
Line 2
Line 3
Example desired output:
echo 'Line 1';echo 'Line 2';echo 'Line 3';
Note: whether printf, echo or another command is used to create the output, doesn't matter as long as the source is human readable.
One hurdle were the single quotes, that would not be recreated. Therefore use the form $'string', which are treated specially. The word expands to string, with backslash-escaped characters replaced as specified by the ANSI C standard.
Another requirement is to re-create tab characters from the old file in the new file. Therefore the wish is to replace <\tab> characters with \t.
Our tries to do this with sed or tr fail. How to replace tabs with their escape \t counterpart and still being able to output lines with original quotes?
Input file /Library/Scripts/BootRepairMount.sh contains:
$ cat /Library/Scripts/BootRepairMount.sh
#!/bin/bash
sleep 18
for OUTPUT in $(diskutil list | grep ': Apple_HFS' | awk '{ print $NF }')
do
if [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then
echo "$OUTPUT is not mounted, repair and mount"
diskutil repairVolume $OUTPUT
diskutil mount $OUTPUT
fi
done
The best shell one line command we could create is:
$ oldifs=$IFS;printf '\n';printf '{';while IFS= read -r p;do [[ "$p" == *"'"* ]] && echo -n "echo $'$p';" || echo -n "echo '$p';"; done < /Library/Scripts/BootRepairMount.sh | tr '\t' '\134\164';printf '}';printf '\n\n';IFS=$oldifs
Which returns this faulty output:
{echo '#!/bin/bash';echo 'sleep 18';echo $'for OUTPUT in $(diskutil list | grep ': Apple_HFS' | awk '{ print $NF }')';echo 'do';echo '\if [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then';echo '\\echo "$OUTPUT is not mounted, repair and mount"';echo '\\diskutil repairVolume $OUTPUT';echo '\\diskutil mount $OUTPUT';echo '\fi';echo 'done';}
Desired output is:
{echo '#!/bin/bash';echo 'sleep 18';echo $'for OUTPUT in $(diskutil list | grep ': Apple_HFS' | awk '{ print $NF }')';echo 'do';echo '\tif [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then';echo '\t\techo "$OUTPUT is not mounted, repair and mount"';echo '\t\tdiskutil repairVolume $OUTPUT';echo '\t\tdiskutil mount $OUTPUT';echo '\tfi';echo 'done';}
Bash one line command version 2
$ oldifs=$IFS;printf '\n';printf '{';while IFS= read -r p;do [[ "$p" == *"'"* ]] && printf 'printf $'\''%q'\'';' "$p" || printf 'printf '\''%q'\'';' "$p"; done < /Library/Scripts/BootRepairMount.sh;printf '}';printf '\n\n';IFS=$oldifs
returns output that is heavy escaped:
{printf '\#\!/bin/bash';printf 'sleep\ 18';printf $'for\ OUTPUT\ in\ \$\(diskutil\ list\ \|\ grep\ \':\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Apple_HFS\'\ \|\ awk\ \'\{\ print\ \$NF\ \}\'\)';printf 'do';printf '$'\tif [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then'';printf '$'\t\techo "$OUTPUT is not mounted, repair and mount"'';printf '$'\t\tdiskutil repairVolume $OUTPUT'';printf '$'\t\tdiskutil mount $OUTPUT'';printf '$'\tfi'';printf 'done';}
that never gets unescaped back to its original values in Mac OS X 10.7.5.
printf '\#\!/bin/bash';
outputs:
\#\!/bin/bash
As well as:
echo -e '\#\!/bin/bash'
does output the unescaped value
\#\!/bin/bash
-e is not a valid command switch for the Mac OS X 10.7.5 echo command, according to its man page.
bash's builtin command printf has %q format code that handles this:
printf '\n{ '; while IFS= read -r p; do printf "echo %q; " "$p"; done < /Library/Scripts/BootRepairMount.sh; printf '}\n\n'
Unfortunately, it doesn't always choose quoting/escaping modes that're easy to read. Specifically, it tends to prefer escaping individual metacharacters (e.g. spaces) rather than enclosing them in quotes:
{ echo \#\!/bin/bash; echo sleep\ 18; echo for\ OUTPUT\ in\ \$(diskutil\ list\ \|\ grep\ \':\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Apple_HFS\'\ \|\ awk\ \'{\ print\ \$NF\ }\'); echo do; echo $'\tif [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then'; echo $'\t\techo "$OUTPUT is not mounted, repair and mount"'; echo $'\t\tdiskutil repairVolume $OUTPUT'; echo $'\t\tdiskutil mount $OUTPUT'; echo $'\tfi'; echo done; }
If I understand right you want paste one long line to the Terminal.app and want get the "source code" of original script. So, need a script what will generate the one-line script.
Maybe a bit unusual solution, but it is easy and simple.
here is the test script called test.sh (instead of your BootReapirMount.sh)
for i in {1..10}
do
date
done
Here is the generator script mkecho.sh
#!/bin/bash
[[ ! -f "$1" ]] && echo "Need filename" && exit 1
asc=$(gzip < "$1" | base64)
echo "base64 -D <<<'$asc'| gzip -d"
Now, run:
bash mkecho.sh test.sh
you will get the next:
base64 -D <<<'H4sIAASwqFEAA0vLL1LIVMjMU6g21NMzNKjlSsnn4kxJLEkFMvJSuQBZFmY0HwAAAA=='| gzip -d
If you copy and paste the above into the terminal, it will will display the original test.sh
Variant2
If you want directly execute the script, you should modify the mkecho.sh to the next mkeval.sh
#!/bin/bash
[[ ! -f "$1" ]] && echo "Need filename" && exit 1
asc=$(gzip < "$1" | base64)
echo -n 'eval "$(base64 -D <<<"'
echo -n $asc
echo -n '" | gzip -d)"'
echo
When run
bash mkeval.sh test.sh
will get
eval "$(base64 -D <<<"H4sIAASwqFEAA0vLL1LIVMjMU6g21NMzNKjlSsnn4kxJLEkFMvJSuQBZFmY0HwAAAA==" | gzip -d)"
and finally when you copy and paste it into the terminal, you run the test.sh and will get:
Fri May 31 16:25:08 CEST 2013
... 8 lined deleted...
Fri May 31 16:25:08 CEST 2013
Warning: because the script is NOT TESTED for every possible conditions, nor for redirects and so on - I really don't recommending using the eval verision.
sed 's/\\t/\\/g'
$ echo 'ffsd \tif [[ -z $' | sed 's/\\t/\\/g'
ffsd \if [[ -z $

In a unix box, I am taking a list of files as input. If it is found, return the path otherwise return a message "filename file not found"

I have used the find command for this, but it doesnt return any message when a file is not found.
And I want the search to be recursive and return a message "not found" when a file is not found.
Here's the code I have done so far. Here "input.txt" contains the list of files to be searched.
set `cat input.txt`
echo $#
for i in $#
do
find $HOME -name $i
done
Try this:
listfile=input.txt
exec 3>&1
find | \
grep -f <( sed 's|.*|/&$|' "$listfile" ) | \
tee /dev/fd/3 | \
sed 's|.*/\([^/]*\)$|\1|' | \
grep -v -f - "$listfile" | \
sed 's/$/ Not found/'
exec 3>&-
open file descriptor 3
find the files
see if they're on the list (use sed to
send a copy of the found ones to file descriptor 3
strip off the directory name
get a list of the ones that don't appear
add the "Not found" message
close file descriptor 3
Output looks like:
/path/to/file1
/path/somewhere/file2
foo Not found
bar Not found
No loops necessary.
Whats wrong with using a script. I hope this will do.
#!/bin/bash -f
for i in $#
do
var=`find $HOME -name $i`
if [ -z "$var"]
then
var="File not found"
fi
echo $var
done
You can use the shell builtin 'test' to test the existence of a file. There is also an alternative syntax using square brackets:
if [ -f $a ]; then # Don't forget the semicolon.
echo $a
else
echo 'Not Found'
fi
Here is one way - create a list of all the files to grep against. If your implementation supports
grep -q otherwise use grep [pattern] 2&>1 >/dev/null....
find $HOME -type f |
while read fname
do
echo "$(basename $fname) $fname"
done > /tmp/chk.lis
while read fname
do
grep -q "^$fname" /tmp/chk.lis
[ $? -eq 0 ] && echo "$fname found" || echo "$fname not found"
done < /tmp/chk.lis
All of this is needed because POSIX find does not return an error when a file is not found
perl -nlE'say-f$_?$_:"not found: $_"' file

Resources