I'm a terrible beginner at bash scripting, and am hoping someone can help me out with this issue.
Having a problem with the Prey project standalone scripts. There's a line that's supposed to send an email, and apparently its not formatted correctly.
response=`mailsender -f "$mail_from" -t "$mail_to" -u "$complete_subject" \
-s $smtp_server -a $file_list -o message-file="$trace_file.msg" \
tls=auto username=$smtp_username \
password=\`decrypt \"$smtp_password\"\``
Where mailsender is an alias to Brandon Zehm's PERL sendEmail script, $smtp_password is a pointless base64 encoding of the password, and decrypt is:
decrypt() {
echo "$1" | openssl enc -base64 -d
}
So can anyone tell me what's wrong with the script? For reference, if I just replace the entire decrypt part with the plaintext password, it works fine. i.e.:
response=`mailsender -f "$mail_from" -t "$mail_to" -u "$complete_subject" \
-s $smtp_server -a $file_list -o message-file="$trace_file.msg" \
tls=auto username=$smtp_username password=actual_password`
The simplest thing to do is avoid backticks, and use $() instead -- they nest cleanly, with no special escaping needed:
response=$(Documents/Projects/Shell\ Scripting/printargs -f "$mail_from" \
-t "$mail_to" -u "$complete_subject" -s $smtp_server -a $file_list \
-o message-file="$trace_file.msg" tls=auto username=$smtp_username \
password="$(decrypt "$smtp_password")")
I think this script is isomorphic with yours:
decrypt()
{
echo "$1" | tr 'a-z' 'A-Z'
}
xxx=`echo xxx=yyy pass=\`decrypt \"xyz abc\"\``
echo "$xxx"
When run with 'sh -x xxx' (where 'sh' is 'bash' in disguise):
$ sh -x xxx
+++ decrypt '"xyz' 'abc"'
+++ echo '"xyz'
+++ tr a-z A-Z
++ echo xxx=yyy 'pass="XYZ'
+ xxx='xxx=yyy pass="XYZ'
+ echo 'xxx=yyy pass="XYZ'
xxx=yyy pass="XYZ
$
You can see where there are problems - if you know how to look. The decrypt command line has two arguments where the intention was to have just one, and the arguments include a double quote before the first and another at the end of the second.
So, in your script, the argument passed to decrypt includes a pair of double quotes, which probably isn't what you wanted.
If we rewrite the script using the '$(...)' notation, which nests much more neatly, then we get:
decrypt()
{
echo "$1" | tr 'a-z' 'A-Z'
}
yyy=$(echo zzz=yyy pass=$(decrypt "xyz abc"))
echo "$yyy"
The trace from this looks like:
$ sh -x xxx
+++ decrypt 'xyz abc'
+++ echo 'xyz abc'
+++ tr a-z A-Z
++ echo zzz=yyy pass=XYZ ABC
+ yyy='zzz=yyy pass=XYZ ABC'
+ echo 'zzz=yyy pass=XYZ ABC'
zzz=yyy pass=XYZ ABC
$
I'm one of the guys from Prey. This bug was confimed yesterday and a fix has already been commited.
I do agree that $() is much easier to read than backticking -- and specially back-backticking --, and actually that's one of the things we're working on (big code refactoring).
Lately I've been working on a Bash framework called Skull which provides a much nicer interface for writing shell scripts. Hopefully Prey 0.6 will be based completely on it, and excessive backticking will be replaced with $() to make it easier for everyone to read.
Related
A bash function, prepend_line, takes two parameters: a string and a fully-qualified path to a file. It's used for logging, inserting the current date/time and the string at the top of the log file.
Standalone use works fine: prepend_line "test string" "$log_file"
How can I get the output from a command, e.g. mv -fv "$fileOne" "$fileTwo" to be used as the first parameter for prepend_line?
I've tried various combinations of piping to xargs, but I don't understand how it works and I'm not convinced it's the best way in any case.
If you really have to:
export -f prepend_line
mv -fv "$fileOne" "$fileTwo" |
xargs -0 bash -c 'prepend_line "$1" "$log_file"' --
The -0 parses the line as beeing zero delimetered. As there should be no zeros in mv -v output, as filenames can't have a zero byte, you will get only a single element. This element/line will be passed as the first argument to the bash subshell.
Tested with:
prepend_line() {
printf "%s\n" "$#" | xxd -p
}
fileOne=$'1\x01\x02\x031234566\n\t\e'
fileTwo=$'2\x01\x02\x031234566\n\t\e \n\n\n'
export -f prepend_line
printf "%s\n" "$fileOne -> $fileTwo" |
xargs -0 bash -c 'prepend_line "$1" "$log_file"' --
The script will output (output from the xxd -p inside prepend_line):
31010203313233343536360a091b202d3e2032010203313233343536360a
091b200a0a0a0a0a0a
Same hex output with some extra newlines and comments:
# first filename $'1\x01\x02\x031234566\n\t\e'
31010203313233343536360a091b
# the string: space + '->' + space
202d3e20
# second filename $'2\x01\x02\x031234566\n\t\e \n\n\n'
32010203313233343536360a091b200a0a0a0a0a0a
If you really have to parse some strange input's you can convert your string to hex with xxd -p. Then, later, convert it back to machine representation with xxd -r -p and streaming right into the output:
prepend_line() {
# some work
# append the output of the "$1" command to the log_file
<<<"$1" xxd -p -r >> "$2"
# some other work
}
prepend_line "$(mv -fv "$fileOne" "$fileTwo" | xxd -p)" "$log_file"
But I doubt you will ever need to handle such cases. Who names filenames using $'\x01' and suffixes with empty newlines 'great_script.sh'$'\n\n'?
Anyway, objectively I would rather see the interface as using a stream:
mv -fv "$fileOne" "$fileTwo" | prepend_line "$log_file"
It needs set -o pipefail to propagate errors correctly. Inside prepend_line I would just redirect the output to the log file or some temporary file, sparing the need of parsing and corner cases.
I'm trying to kick off multiple processes to work through some test suites. In my bash script I have the following
printf "%s\0" "${SUITE_ARRAY[#]}" | xargs -P 2 -0 bash -c 'run_test_suite "$#" ${EXTRA_ARG}'
Below is the defined script, cut down to it's basics.
SUITE_ARRAY will be a list of suites that may have 1 or more, {Suite 1, Suite 2, ..., Suite n}
EXTRA_ARG will be like a specific name to store values in another script
#!/bin/bash
run_test_suite(){
suite=$1
someArg=$2
someSaveDir=someArg"/"suite
# some preprocess work happens here, but isn't relevant to running
runSomeScript.sh suite someSaveDir
}
export -f run_test_suite
SUITES=$1
EXTRA_ARG=$2
IFS=','
SUITECOUNT=0
for csuite in ${SUITES}; do
SUITE_ARRAY[$SUITECOUNT]=$csuite
SUITECOUNT=$(($SUITECOUNT+1))
done
unset IFS
printf "%s\0" "${SUITE_ARRAY[#]}" | xargs -P 2 -0 bash -c 'run_test_suite "$#" ${EXTRA_ARG}'
The issue I'm having is how to get the ${EXTRA_ARG} passed into xargs. From how I've come to understand it, xargs will take whatever is piped into it, so the way I have it doesn't seem correct.
Any suggestions on how to correctly pass the values? Thanks in advance
If you want EXTRA_ARG to be available to the subshell, you need to export it. You can do that either explicitly, with the export keyword, or by putting the var=value assignment in the same simple command as xargs itself:
#!/bin/bash
run_test_suite(){
suite=$1
someArg=$2
someSaveDir=someArg"/"suite
# some preprocess work happens here, but isn't relevant to running
runSomeScript.sh suite someSaveDir
}
export -f run_test_suite
# assuming that the "array" in $1 is comma-separated:
IFS=, read -r -a suite_array <<<"$1"
# see the EXTRA_ARG="$2" just before xargs on the same line; this exports the variable
printf "%s\0" "${suite_array[#]}" | \
EXTRA_ARG="$2" xargs -P 2 -0 bash -c 'run_test_suite "$#" "${EXTRA_ARG}"' _
The _ prevents the first argument passed from xargs to bash from becoming $0, and thus not included in "$#".
Note also that I changed "${suite_array[#]}" to be assigned by splitting $1 on commas. This or something like it (you could use IFS=$'\n' to split on newlines instead, for example) is necessary, as $1 cannot contain a literal array; every shell command-line argument is only a single string.
This is something of a guess:
#!/bin/bash
run_test_suite(){
suite="$1"
someArg="$2"
someSaveDir="${someArg}/${suite}"
# some preprocess work happens here, but isn't relevant to running
runSomeScript.sh "${suite}" "${someSaveDir}"
}
export -f run_test_suite
SUITE_ARRAY="$1"
EXTRA_ARG="$2"
printf "%s\0" "${SUITE_ARRAY[#]}" |
xargs -n 1 -I '{}' -P 2 -0 bash -c 'run_test_suite {} '"${EXTRA_ARG}"
Using GNU Parallel it looks like this:
#!/bin/bash
run_test_suite(){
suite="$1"
someArg="$2"
someSaveDir="$someArg"/"$suite"
# some preprocess work happens here, but isn't relevant to running
echo runSomeScript.sh "$suite" "$someSaveDir"
}
export -f run_test_suite
EXTRA_ARG="$2"
parallel -d, -q run_test_suite {} "$EXTRA_ARG" ::: "$1"
Called as:
mytester 'suite 1,suite 2,suite "three"' 'extra "quoted" args here'
If you have the suites in an array:
parallel -q run_test_suite {} "$EXTRA_ARG" ::: "${SUITE_ARRAY[#]}"
Added bonus: Any output from the jobs will not be mixed, so you will not have to deal with http://mywiki.wooledge.org/BashPitfalls#Using_output_from_xargs_-P
I'd like to be able to comment out a single flag in a one-line command. Bash only seems to have from # till end-of-line comments. I'm looking at tricks like:
ls -l $([ ] && -F is turned off) -a /etc
It's ugly, but better than nothing. Is there a better way?
The following seems to work, but I'm not sure whether it is portable:
ls -l `# -F is turned off` -a /etc
My preferred is:
Commenting in a Bash script
This will have some overhead, but technically it does answer your question
echo abc `#put your comment here` \
def `#another chance for a comment` \
xyz etc
And for pipelines specifically, there is a cleaner solution with no overhead
echo abc | # normal comment OK here
tr a-z A-Z | # another normal comment OK here
sort | # the pipelines are automatically continued
uniq # final comment
How to put a line comment for a multi-line command
I find it easiest (and most readable) to just copy the line and comment out the original version:
#Old version of ls:
#ls -l $([ ] && -F is turned off) -a /etc
ls -l -a /etc
$(: ...) is a little less ugly, but still not good.
Here's my solution for inline comments in between multiple piped commands.
Example uncommented code:
#!/bin/sh
cat input.txt \
| grep something \
| sort -r
Solution for a pipe comment (using a helper function):
#!/bin/sh
pipe_comment() {
cat -
}
cat input.txt \
| pipe_comment "filter down to lines that contain the word: something" \
| grep something \
| pipe_comment "reverse sort what is left" \
| sort -r
Or if you prefer, here's the same solution without the helper function, but it's a little messier:
#!/bin/sh
cat input.txt \
| cat - `: filter down to lines that contain the word: something` \
| grep something \
| cat - `: reverse sort what is left` \
| sort -r
Most commands allow args to come in any order. Just move the commented flags to the end of the line:
ls -l -a /etc # -F is turned off
Then to turn it back on, just uncomment and remove the text:
ls -l -a /etc -F
How about storing it in a variable?
#extraargs=-F
ls -l $extraargs -a /etc
If you know a variable is empty, you could use it as a comment. Of course if it is not empty it will mess up your command.
ls -l ${1# -F is turned off} -a /etc
ยง 10.2. Parameter Substitution
For disabling a part of a command like a && b, I simply created an empty script x which is on path, so I can do things like:
mvn install && runProject
when I need to build, and
x mvn install && runProject
when not (using Ctrl + A and Ctrl + E to move to the beginning and end).
As noted in comments, another way to do that is Bash built-in : instead of x:
$ : Hello world, how are you? && echo "Fine."
Fine.
It seems that $(...) doesn't survive from ps -ef.
My scenario is that I want to have a dummy param that can be used to identify the very process. Mostly I use this method, but the method is not workable everywhere. For example, python program.py would be like
mkdir -p MyProgramTag;python MyProgramTag/../program.py
The MyProgramTag would be the tag for identifying the process started.
Another one I can't find an answer for, and it feels like I've gone mad.
I have a BASH script using a for loop to run a complex command (many protein sequence alignments) on a lot of files (~5000). The loop produces statements that will execute when given alone (i.e. copy-pasted from the error message to the command prompt), but which return "no such file or directory" inside the loop. Script below; there are actually several more arguments but this includes some representative ones and the file arguments.
#!/bin/bash
# Pass directory with targets as FASTA sequences as argument.
# Arguments to psiblast
# Common
db=local/db/nr/nr
outfile="/mnt/scratch/psi-blast"
e=0.001
threads=8
itnum=5
pssm="/mnt/scratch/psi-blast/pssm."
pssm_txt="/mnt/scratch/psi-blast/pssm."
pseudo=0
pwa_inclusion=0.002
for i in ${1}/*
do
filename=$(basename $i)
"local/ncbi-blast-2.2.23+/bin/psiblast\
-query ${i}\
-db $db\
-out ${outfile}/${filename}.out\
-evalue $e\
-num_threads $threads\
-num_iterations $itnum\
-out_pssm ${pssm}$filename\
-out_ascii_pssm ${pssm_txt}${filename}.txt\
-pseudocount $pseudo\
-inclusion_ethresh $pwa_inclusion"
done
Running this scripts gives "<scriptname> line <last line before 'done'>: <attempted command> : No such file or directory. If I then paste the attempted command onto the prompt it will run.
Each of these commands takes a couple of minutes to run.
try without the quotes. and you forgot some slashes.
for i in ${1}/*
do
filename=$(basename $i)
local/ncbi-blast-2.2.23+/bin/psiblast \
-query "${i}" \
-db "$db" \
-out "${outfile}/${filename}.out" \
-evalue "$e" \
-num_threads "$threads" \
-num_iterations "$itnum" \
-out_pssm "${pssm}/$filename" \
-out_ascii_pssm "${pssm_txt}/${filename}.txt" \
-pseudocount "$pseudo" \
-inclusion_ethresh "$pwa_inclusion"
done
The behavior you're observing will occur if there are spaces in the filenames you're iterating over. For this reason, you'll want to properly quote your filenames, as in the following minimal example:
#!/bin/bash
for i in *
do
filename="$(basename "$i")"
command="ls -lah '$filename'"
echo "filename=$filename"
echo "Command = $command"
eval "$command"
done
Adding quotes to filenames will not help when using a for loop. To overcome this, I've always done something similar to the following example whenever I needed to loop over filenames:
ls -1 directory | { while read line; do echo $line; done; }
I want to create a shellscript that reads files from a .diz file, where information about various source files are stored, that are needed to compile a certain piece of software (imagemagick in this case). i am using Mac OSX Leopard 10.5 for this examples.
Basically i want to have an easy way to maintain these .diz files that hold the information for up-to-date source packages. i would just need to update these .diz files with urls, version information and file checksums.
Example line:
libpng:1.2.42:libpng-1.2.42.tar.bz2?use_mirror=biznetnetworks:http://downloads.sourceforge.net/project/libpng/00-libpng-stable/1.2.42/libpng-1.2.42.tar.bz2?use_mirror=biznetnetworks:9a5cbe9798927fdf528f3186a8840ebe
script part:
while IFS=: read app version file url md5
do
echo "Downloading $app Version: $version"
curl -L -v -O $url 2>> logfile.txt
$calculated_md5=`/sbin/md5 $file | /usr/bin/cut -f 2 -d "="`
echo $calculated_md5
done < "files.diz"
Actually I have more than just one question concerning this.
how to calculate and compare the checksums the best? i wanted to store md5 checksums in the .diz file and compare it with string comparison with "cut"ting out the string
is there a way to tell curl another filename to save to? (in my case the filename gets ugly libpng-1.2.42.tar.bz2?use_mirror=biznetnetworks)
i seem to have issues with the backticks that should direct the output of the piped md5 and cut into the variable $calculated_md5. is the syntax wrong?
Thanks!
The following is a practical one-liner:
curl -s -L <url> | tee <destination-file> |
sha256sum -c <(echo "a748a107dd0c6146e7f8a40f9d0fde29e19b3e8234d2de7e522a1fea15048e70 -") ||
rm -f <destination-file>
wrapping it up in a function taking 3 arguments:
- the url
- the destination
- the sha256
download() {
curl -s -L $1 | tee $2 | sha256sum -c <(echo "$3 -") || rm -f $2
}
while IFS=: read app version file url md5
do
echo "Downloading $app Version: $version"
#use -o for output file. define $outputfile yourself
curl -L -v $url -o $outputfile 2>> logfile.txt
# use $(..) instead of backticks.
calculated_md5=$(/sbin/md5 "$file" | /usr/bin/cut -f 2 -d "=")
# compare md5
case "$calculated_md5" in
"$md5" )
echo "md5 ok"
echo "do something else here";;
esac
done < "files.diz"
My curl has a -o (--output) option to specify an output file. There's also a problem with your assignment to $calculated_md5. It shouldn't have the dollar sign at the front when you assign to it. I don't have /sbin/md5 here so I can't comment on that. What I do have is md5sum. If you have it too, you might consider it as an alternative. In particular, it has a --check option that works from a file listing of md5sums that might be handy for your situation. HTH.