Some unexpected thing with my shell script - shell

This script is used for detecting USB drive input any copy some log files to USB.
on line 47,
cp "/home/root/io/log/${end}.txt" "${USB}/log" && let sfile++ || let err3=1
three commands are all executed, whether the cp command is succeed or not. And if I swap this line with line 49
cp "/home/root/io/log/${begin}.txt" "${USB}/log" && let sfile++ || let err3=1
, only the upper line of codes didn't work as expected.
echo $USB > /dev/null
echo $FILE > /dev/null
echo $begin > /dev/null
echo $end > /dev/null
echo $destfile > /dev/null
echo $tmp > /dev/null
echo $err > /dev/null
echo $sfile > /dev/null
echo $DATE > /dev/null
echo 84 > /sys/class/gpio/export
echo out > /sys/class/gpio/gpio84/direction
while true
do
USB=""
FILE=""
begin=""
end=""
destfile=""
DATE=""
let err1=0
let err2=0
let err3=0
let err4=0
let err=0
let sfile=0
ls /dev/sda1 > /dev/null 2>&1 || rm -rf /media/sda1
ls /media/sda1 > /dev/null 2>&1 && USB="/media/sda1"
if [[ "$USB" != "" ]]; then
ls $USB | grep 'cp.txt' > /dev/null && FILE="cp.txt"
if [[ "$FILE" != "" ]]; then
ls "${USB}/log" > /dev/null 2>&1 && rm -rf "${USB}/log"
mkdir "${USB}/log" && echo 0 >> /dev/null || let err1=1
begin=$(cat "${USB}/${FILE}" | cut -d 'd' -f 2)
end=$(cat "${USB}/${FILE}" | cut -d 'd' -f 3)
cp "/home/root/io/log/${end}.txt" "${USB}/log" && let sfile++ || let err3=1
cp "/home/root/io/log/${begin}.txt" "${USB}/log" && let sfile++ || let err2=1
destfile=$(ls /home/root/io/log)
destfile=${destfile#*${begin}.txt}
destfile=${destfile%${end}.txt*}
echo $destfile
while [[ "$(echo $destfile | cut -d ' ' -f 1)" != "" ]]; do
tmp=""
tmp=$(echo $destfile | cut -d ' ' -f 1)
destfile=${destfile#*$tmp}
cp "/home/root/io/log/${tmp}" "${USB}/log" && let sfile++ || let err4=1
done
DATE=$(date +%Y%m%d-%H%M%S)
LOG="${DATE} Requested: from-${begin} to-${end}. Succeed:${sfile} err1:${err1}err2:${err2}err3:${err3}err4:${err4}"
echo $LOG >> /home/root/io/cplog/cplog.txt
echo $LOG >> /media/sda1/cplog.txt
if [[ err -eq 0 ]]; then
echo 1 > /sys/class/gpio/gpio84/value
sleep 1
echo 0 > /sys/class/gpio/gpio84/value
else
for (( i = 0; i < 10; i++ )); do
echo 1 > /sys/class/gpio/gpio84/value
sleep 0.05
echo 0 > /sys/class/gpio/gpio84/value
sleep 0.05
done
fi
umount "${USB}"
fi
fi
sleep 2
done

I guess you are using bash, but unless you stick with POSIX-compatibility, always indicate the shell you are using. Assuming bash, if you have a command sequence X && Y || Z, and if X returns a non-zero exit status, Y won't be executed.
Aside from this, you do have a logical flaw in your script. Note that let also sets an exit status. The exit status is 1 if the value of the expression to be evaluated, is 0, and the exit status is 0 otherwise. Now regarding your let sfile++, the value of the expression is the value which sfile had before incrementing.
In the first iteration of your loop, sfile is zero. Assuming that cp returns exit code 0, sfile will be incremented, the let command returns exit code 1, and the let err4=1 will also be executed.
In the subsequent iterations, this won't happen anymore, because let sfile++ returns exit code 0.
My advice is to not be too fancy with command sequences mixing && and ||. In this case, a simple if would be the better choice:
if cp ...
then
let ...
else
let
fi

Related

Bash conditional commands

I have a string of commands I want to run in succession, if the previous command has been successful. Here is an example:
echo "hello" && sleep 5 && cd /tmp && rm -r * && echo "googbye"
If any part fails, I need to know which part failed. Also, I am trying to implement a spinner while the commands are running. Is there a way to do this?
In terms of detecting which bit failed, you could use:
bad=0
[[ $bad -eq 0 ]] && { echo hello || bad=1; }
[[ $bad -eq 0 ]] && { sleep 5 || bad=2; }
[[ $bad -eq 0 ]] && { cd /tmp || bad=3; }
[[ $bad -eq 0 ]] && { rm -r * || bad=4; }
[[ $bad -eq 0 ]] && { echo goodbye || bad=5; }
The variable bad would then be set based on which bit failed (if any).
Alternatively, if that's too verbose, you can still do it in a single line, albeit a longer one:
bad=1 && echo hello && bad=2 && sleep 5 && bad=3 && cd /tmp && bad=4 && echo goodbye && bad=0
In terms of a spinner, that may not work so well if you actually have output in the job you're doing but you can do it with a simple background function:
spinner() {
while : ; do
echo -ne '\b|' ; sleep 1
echo -ne '\b/' ; sleep 1
echo -ne '\b-' ; sleep 1
echo -ne '\b\\' ; sleep 1
done
}
echo -n 'Starting: .'
spinner & pid=$!
sleep 13 ; kill $pid
echo
echo Done.
With regard to putting it all together based on your question and comments, I'd probably have two functions, the first to run the spinner in the background and the second to do the payload, including capturing its output so it doesn't affect the spinner.
To that end, have a look at the following demo code:
spinner() {
echo -n 'Starting: .'
chars='\-/|'
while [[ -f "$1" ]] ; do
echo -ne "\b${chars:3}" ; sleep 1
chars="${chars:3}${chars:0:3}"
done
echo -e ' .. done.'
}
payload() {
echo Running task 1
sleep 3
true || return 1
echo Running task 2
sleep 3
false || return 2
echo Running task 3
sleep 3
true || return 3
return 0
}
touch /tmp/sentinel.$$
spinner /tmp/sentinel.$$ & pid=$!
payload >/tmp/out.$$ 2>&1 ; rc=$?
rm /tmp/sentinel.$$ ; wait
echo "Return code was $rc, output was:"
cat /tmp/out.$$
rm /tmp/out.$$
In the payload function, simply replace each of the steps with whatever you wish to actually do (such as shut down a systemd job or delete some files).
The code as it stands will fail step 2 (false) but, for your own code, this would be running actual commands and evaluating their return value.
I've also used a sentinel file to terminate the spinner function so that it's not shut down "violently".

Complement the exit status in if with -e set

Is there a simple way to do something like:
if ! some_command; then
some_commands;
fi
or
if [[ com1 && com2 ]]; then
something;
fi
where it's the exit status of com1 and com2 that are used.
I realize that I can do things like get the exit status even with -e set by using || and check that, etc. Just wondering if there was something simpler that I am missing.
ADDENDUM: I also realize that I could do:
if some_command; then
else
some_commands;
fi
! can be used outside of an if statement; it's the general exit-status inverter, not part of the if syntax.
! some_command && { some_commands; }
and
some_command || { some_commands; }
are equivalent. You can also use
com1 && com2 && { some_commands; }
for first: you can write something like: where <command1/2> is/are some command(s)/script(s) with/without parameter
$ <command1> 2>/dev/null || <command2> 2>/dev/null
for second: you can write something like:
$ ( <command1> && <command2> ) 2>/dev/null && <command3> 2>/dev/null
ex:
$ echo 1 2>/dev/null || echo 0 2>/dev/null
1
$ ech1o 1 2>/dev/null|| echo 0 2>/dev/null
0
AND
$ ( echo 1 && echo 2 ) 2>/dev/null && echo 3
1
2
3
The following won't print anything as echo 3 will never be executed (due to echo1 is not a valid command)
$ ( echo1 1 && echo 2 ) 2>/dev/null && echo 3

grep-ing a variable vs. a file - execution time

made an interesting observation - I was storing the output of a cURL statement in a text file and then grep-ing it for certain strings. Later I changed my code to store the output to a variable instead. Turns out, this change caused my script to run much slower. This was really counter intuitive for me since I always thought I/O operations would be more expensive than in-memory operations. Here is the code:
#!/bin/bash
URL="http://m.cnbc.com"
while read line; do
UA=$line
curl -s --location --user-agent "$UA" $URL > RAW.txt
#RAW=`curl --location --user-agent "$UA" $URL`
L=`grep -c -e "Advertise With Us" RAW.txt`
#L=`echo $RAW | grep -c -e "Advertise With Us"`
M=`grep -c -e "id='menu'><button>Menu</button>" RAW.txt`
#M=`echo $RAW | grep -c -e "id='menu'><button>Menu</button>"`
D=`grep -c -e "Careers" RAW.txt`
#D=`echo $RAW | grep -c -e "Careers"`
if [[ ( $L == 1 && $M == 0 ) && ( $D == 0) ]]
then
AC="Legacy"
elif [[ ( $L == 0 && $M == 1 ) && ( $D == 0) ]]
then
AC="Modern"
elif [[ ( $L == 0 && $M == 0 ) && ( $D == 1) ]]
then
AC="Desktop"
else
AC="Unable to Determine"
fi
echo $AC >> Results.txt
done < UserAgents.txt
The commented lines represent the storing-in-variable approach. Any ideas why would this be happening? Also are there any ways to further speed-up this script? Right now it takes about 8 minutes to process 2000 input entries.
Chepner is correct. Read each call to cURL just once, flagging each of the three desired strings. Here's some example code using awk. Completely untested:
URL="http://m.cnbc.com"
while IFS= read -r line; do
RAW=$(curl --location --user-agent "$line" $URL)
awk '
/Advertise With Us/ {
L=1
}
/id='\''menu'\''><button>Menu<\/button>/ {
M=1
}
/Careers/ {
D=1
}
END {
if (L==1 && M==0 && D==0) {
s = "Legacy"
}
else if (L==0 && M==1 && D==0) {
s = "Modern"
}
else if (L==0 && M==0 && D==1) {
s = "Desktop"
}
else {
s = "Unable to Determine"
}
print s >> "Results.txt"
}' "$RAW"
done < UserAgents.txt
Do you really need to count the number of matches with grep -c? It looks like you just need to know if a match was found or not. If so, you can simply use bash's in-built string comparison.
Also, it will be faster if you write to the results file outside the loop.
Try the following:
#!/bin/bash
URL="http://m.cnbc.com"
while read line
do
UA="$line"
RAW=$(curl -s --location --user-agent "$UA" "$URL")
[[ $RAW == *"Advertise With Us"* ]] && L=1 || L=0
[[ $RAW == *"id='menu'><button>Menu</button>"* ]] && M=1 || M=0
[[ $RAW == *Careers* ]] && D=1 || D=0
if (( L==1 && M==0 && D==0 ))
then
AC="Legacy"
elif (( L==1 && M==1 && D==0 ))
then
AC="Modern"
elif (( L==1 && M==0 && D==1 ))
then
AC="Desktop"
else
AC="Unable to Determine"
fi
echo "$AC"
done < UserAgents.txt > Results.txt

Bash Loop and exit status check

An array holds the files accessed, and the archive files are split into smaller sizes in preparation for online backup. I am attempting to retrieve the exit code for each iteration through the loop of the split command. However, it is returning Exit Code 1, yet it says that the operation was successful. Why?
#!/bin/bash
declare -a SplitDirs
declare -a CFiles
CDIR=/mnt/Net_Pics/Working/Compressed/
SDIR=/mnt/Net_Pics/Working/Split/
Err=/mnt/Net_Pics/Working
SplitDirs=(`ls -l "$CDIR" --time-style="long-iso" | egrep '^d' | awk '{print $8}'`)
for dir in "${SplitDirs[#]}"
do
if [ ! -d "$SDIR""$dir" ]; then
mkdir "$SDIR""$dir"
else continue
fi
CFiles=(`ls -l "$CDIR$dir" --time-style="long-iso" | awk '{print $8}'`)
for f in "${CFiles[#]}"
do
if [ ! -e "$SDIR""$dir"/"$f" ]; then
split -d -a 4 -b 1992295 "$CDIR""$dir"/"$f" "$SDIR""$dir"/"$f" --verbose
if [[ "$?" == 1 ]]
then
rm -rf "$SDIR""$dir" && echo "$SDIR""$dir" "Removed due to Error code" "$?""." "Testing Archives and Retrying..." 2>&1 | tee "$Err"/Split_Err.log
7z t "$CDIR""$dir"/"$f" >> tee stdout.log 2>> "$Err"/"$dir"/7z_Err.log >&2
mkdir "$SDIR""$dir" && split -d -a 4 -b 1992295 "$CDIR""$dir"/"$f" "$SDIR""$dir"/"$f" --verbose
if [[ "$?" == 1 ]]
then
rm -rf "$SDIR""$dir" && echo "$SDIR""$dir" "Removed a second time due to Error code "$?". Skipping..." 2>&1 | tee "$Err"/Split_Err.log
continue
else
echo "Split Success:" "$SDIR""$dir"/"$f" "ended with Exit status" "$?" && continue
fi
else
echo "Split Success:" "$SDIR""$dir" "ended with Exit status" "$?" && continue
fi
else
echo "$SDIR""$dir"/"$f" "Exists... Skipping Operation" 2>&1 | tee "$Err"/"$dir"/Split_Err.log
continue
fi
done
(The echo piping in a previous revision of the question was misplaced code, and thank you for pointing that out. The exit code remains the same, though. Overall,the script does what I want it to except for the exit code portion.)
Remove | echo $?. you are processing the return code of echo command(last command).

Test multiple file conditions in one swoop (BASH)?

Often when writing for the bash shell, one needs to test if a file (or Directory) exists (or doesn't exist) and take appropriate action. Most common amongst these test are...
-e - file exists, -f - file is a regular file (not a directory or device file), -s - file is not zero size, -d - file is a directory, -r - file has read permission, -w - file has write, or -x execute permission (for the user running the test)
This is easily confirmed as demonstrated on this user-writable directory....
#/bin/bash
if [ -f "/Library/Application Support" ]; then
echo 'YES SIR -f is fine'
else echo 'no -f for you'
fi
if [ -w "/Library/Application Support" ]; then
echo 'YES SIR -w is fine'
else echo 'no -w for you'
fi
if [ -d "/Library/Application Support" ]; then
echo 'YES SIR -d is fine'
else echo 'no -d for you'
fi
➝ no -f for you ✓
➝ YES SIR -w is fine ✓
➝ YES SIR -d is fine ✓
My question, although seemingly obvious, and unlikely to be impossible - is how to simply combine these tests, without having to perform them separately for each condition... Unfortunately...
if [ -wd "/Library/Application Support" ]
▶ -wd: unary operator expected
if [ -w | -d "/Library/Application Support" ]
▶ [: missing `]'
▶ -d: command not found
if [ -w [ -d "/Library.... ]] & if [ -w && -d "/Library.... ]
▶ [: missing `]'
➝ no -wd for you ✖
➝ no -w | -d for you ✖
➝ no [ -w [ -d .. ]] for you ✖
➝ no -w && -d for you ✖
What am I missing here?
You can use logical operators to multiple conditions, e.g. -a for AND:
MYFILE=/tmp/data.bin
if [ -f "$MYFILE" -a -r "$MYFILE" -a -w "$MYFILE" ]; then
#do stuff
fi
unset MYFILE
Of course, you need to use AND somehow as Kerrek(+1) and Ben(+1) pointed it out. You can do in in few different ways. Here is an ala-microbenchmark results for few methods:
Most portable and readable way:
$ time for i in $(seq 100000); do [ 1 = 1 ] && [ 2 = 2 ] && [ 3 = 3 ]; done
real 0m2.583s
still portable, less readable, faster:
$ time for i in $(seq 100000); do [ 1 = 1 -a 2 = 2 -a 3 = 3 ]; done
real 0m1.681s
bashism, but readable and faster
$ time for i in $(seq 100000); do [[ 1 = 1 ]] && [[ 2 = 2 ]] && [[ 3 = 3 ]]; done
real 0m1.285s
bashism, but quite readable, and fastest.
$ time for i in $(seq 100000); do [[ 1 = 1 && 2 = 2 && 3 = 3 ]]; done
real 0m0.934s
Note, that in bash, "[" is a builtin, so bash is using internal command not a symlink to /usr/bin/test exacutable. The "[[" is a bash keyword. So the slowest possible way will be:
time for i in $(seq 100000); do /usr/bin/\[ 1 = 1 ] && /usr/bin/\[ 2 = 2 ] && /usr/bin/\[ 3 = 3 ]; done
real 14m8.678s
You want -a as in -f foo -a -d foo (actually that test would be false, but you get the idea).
You were close with & you just needed && as in [ -f foo ] && [ -d foo ] although that runs multiple commands rather than one.
Here is a manual page for test which is the command that [ is a link to. Modern implementations of test have a lot more features (along with the shell-builtin version [[ which is documented in your shell's manpage).
check-file(){
while [[ ${#} -gt 0 ]]; do
case $1 in
fxrsw) [[ -f "$2" && -x "$2" && -r "$2" && -s "$2" && -w "$2" ]] || return 1 ;;
fxrs) [[ -f "$2" && -x "$2" && -r "$2" && -s "$2" ]] || return 1 ;;
fxr) [[ -f "$2" && -x "$2" && -r "$2" ]] || return 1 ;;
fr) [[ -f "$2" && -r "$2" ]] || return 1 ;;
fx) [[ -f "$2" && -x "$2" ]] || return 1 ;;
fe) [[ -f "$2" && -e "$2" ]] || return 1 ;;
hf) [[ -h "$2" && -f "$2" ]] || return 1 ;;
*) [[ -e "$1" ]] || return 1 ;;
esac
shift
done
}
check-file fxr "/path/file" && echo "is valid"
check-file hf "/path/folder/symlink" || { echo "Fatal error cant validate symlink"; exit 1; }
check-file fe "file.txt" || touch "file.txt" && ln -s "${HOME}/file.txt" "/docs/file.txt" && check-file hf "/docs/file.txt" || exit 1
if check-file fxrsw "${HOME}"; then
echo "Your home is your home from the looks of it."
else
echo "You infected your own home."
fi
Why not write a function to do it?
check_file () {
local FLAGS=$1
local PATH=$2
if [ -z "$PATH" ] ; then
if [ -z "$FLAGS" ] ; then
echo "check_file: must specify at least a path" >&2
exit 1
fi
PATH=$FLAGS
FLAGS=-e
fi
FLAGS=${FLAGS#-}
while [ -n "$FLAGS" ] ; do
local FLAG=`printf "%c" "$FLAGS"`
if [ ! -$FLAG $PATH ] ; then false; return; fi
FLAGS=${FLAGS#?}
done
true
}
Then just use it like:
for path in / /etc /etc/passwd /bin/bash
{
if check_file -dx $path ; then
echo "$path is a directory and executable"
else
echo "$path is not a directory or not executable"
fi
}
And you should get:
/ is a directory and executable
/etc is a directory and executable
/etc/passwd is not a directory or not executable
/bin/bash is not a directory or not executable
This seems to work (notice the double brackets):
#!/bin/bash
if [[ -fwd "/Library/Application Support" ]]
then
echo 'YES SIR -f -w -d are fine'
else
echo 'no -f or -w or -d for you'
fi

Resources