How to break pipe if stdin is empty? - bash

I want to break the whole pipe if the stdin is empty. I try to combined xargs -r and tee, which means not print and write if stdin is empty, but it failed
...| upstream commands | xargs -r tee output.txt | downstream commands | ...
Any feedback appreciated.

There is no way you can actually terminate a bash pipe conditionally. All commands in a pipeline are started simultaneously. There is however a tool available that would assist you with creating a conditional pipeline. In moreutils you can find the tool ifne which executes a command if and only if the input /dev/stdin is not empty. So you could write something like:
$ command1 | ifne command2 | ifne command3 | ifne command4
Here all commands ifne and command1 are started simultaniously. Only if ifne receives input via /dev/stdin, it will start its respective commandx

Pipe'll break if command failed. You can add grep in between to achieve this. An example:
$ echo ok | awk '{print $0,"1"}' | awk '{print $0,"2"}' | awk '{print $0,"3"}'
ok 1 2 3
Now add grep:
$ echo ok | grep -Ei '^.+$' | awk '{print $0,"1"}' | awk '{print $0,"2"}' | awk '{print $0,"3"}'
ok 1 2 3
And test empty echo:
$ echo | awk '{print $0,"1"}' | awk '{print $0,"2"}' | awk '{print $0,"3"}'
1 2 3
$ echo | grep -Ei '^.+$' | awk '{print $0,"1"}' | awk '{print $0,"2"}' | awk '{print $0,"3"}'
Looks like this works but it doesn't, interesting indeed, well then obvy pipes don't fit here, try this approach:
#!/bin/bash
set -x
fun(){
data=$(echo "$1"); [[ $data ]] && data=$(awk '{print $0,1}' <<< "$data") || return 1; [[ $data ]] && data=$(awk '{print $0,2}' <<< "$data") || return 1; [[ $data ]] && data=$(awk '{print $0,3}' <<< "$data") || return 1; echo "$data"
}
fun ok
fun
Testing:
$ ./test
+ fun ok
++ echo ok
+ data=ok
+ [[ -n ok ]]
++ awk '{print $0,1}'
+ data='ok 1'
+ [[ -n ok 1 ]]
++ awk '{print $0,2}'
+ data='ok 1 2'
+ [[ -n ok 1 2 ]]
++ awk '{print $0,3}'
+ data='ok 1 2 3'
+ echo 'ok 1 2 3'
ok 1 2 3
+ fun
++ echo ''
+ data=
+ [[ -n '' ]]
+ return 1
More readable variant:
#!/bin/bash
set -x
fun(){
data=$(echo "$1")
[[ $data ]] && data=$(awk '{print $0,1}' <<< "$data") || return 1
[[ $data ]] && data=$(awk '{print $0,2}' <<< "$data") || return 1
[[ $data ]] && data=$(awk '{print $0,3}' <<< "$data") || return 1
echo "$data"
}
fun ok
fun

Related

GETOPTS command does not work in my shell script

Can anybody help me please?
i wrote a script and in my script i used GETOPTS to make options but it does not work
it had some error and i check it in shellcheck.net and fixed them but it's not working
#!/bin/bash
while getopts 'n:c2rFt' option; do
case "$option" in
n) export Field="$OPTARG"
;;
c) #Question 1
cat "$1" | awk '{print $1}' | sort | uniq -c | sort -nrk 1,1 > file1
awk 'NR=="$Field" {print}' file1
;;
2) #Question 2
cat "$1" | awk '{ if($9 == 200) print $1,$9 }' | sort | uniq -c | sort -nrk 1,1 > file1
awk 'NR=="$Field" {print}' file1
;;
r) #Question 3
cat "$1" | awk '{print $1,$9}' | sort | uniq -c | sort -nrk 1,1 > file1
awk 'NR=="$Field" {print}' file1
;;
F) #Question 4
cat "$1" | awk '{if($9 >= 400 && $9 <= 451)} {print $1,$9}' | sort | uniq -c | sort -nrk 1,1 > file1
awk 'NR=="$Field" {print}' file1
;;
t) #Question 5
cat "$1" | awk '{print $1,$10}' | sort | uniq -c | sort -nrk 3,3 > file1
awk 'NR=="$Field" {print}' file1
;;
?)
echo "You used wrong option"
echo "USAGE: log_sum.sh [-n N] (-c|-2|-r|-F|-t|-f) <filename>"
echo " -n: Limit the number of results to N"
echo " -c: shows th IP address makes the most number of connection attempts"
echo " -2: shows th most number of seccessful attempts "
echo " -r: shows th most common result codes and their IP addresses"
echo " -F: shows the most common result codes that indicate failure"
echo " -t: shows the IP addresses that get the most bytes sent to them"
exit 1
;;
esac
done
You have your "business logic" in the wrong place: your code assumes that the user will provide the -n option first. That's not required by getopts. You have to write this kind of program with 3 stages: option parsing, validation and actions:
#!/bin/bash
usage() {
local program=$(basename "$0")
cat <<END_USAGE >&2
USAGE: $program -n N (-c|-2|-r|-F|-t|-f) <filename>
-n: Limit the number of results to N
-c: shows th IP address makes the most number of connection attempts
-2: shows th most number of seccessful attempts
-r: shows th most common result codes and their IP addresses
-F: shows the most common result codes that indicate failure
-t: shows the IP addresses that get the most bytes sent to them
END_USAGE
}
# Option parsing
while getopts ':n:c2rFt' option; do
case "$option" in
n) num_results=$OPTARG ;;
c) show_connections=yes ;;
2) show_successful=yes ;;
r) show_common_results=yes ;;
F) show_common_failures=yes ;;
t) show_most_bytes=yes ;;
?) echo "Error: unknown option $OPTARG"; usage; exit 1 ;;
esac
done
shift $((OPTIND - 1))
filename=$1
# Validation
if [[ -z $num_results ]]; then
echo "Error: you must provide the -n option" >&2
usage >&2
exit 1
fi
if [[ -z $filename ]]; then
echo "Error: you must provide a filename" >&2
usage >&2
exit 1
fi
# Actions
# helper function to encapsulate repeated code
top_results() { sort | uniq -c | sort -nrk 1,1 | sed "${num_results}q"; }
if [[ $show_connections == yes ]]; then
awk '{print $1}' "$filename" | top_results
fi
if [[ $show_successful == yes ]]; then
awk '$9 == 200 {print $1,$9}' "$filename" | top_results
fi
if [[ $show_common_results == yes ]]; then
awk '{print $1,$9}' "$filename" | top_results
fi
if [[ $show_common_failures == yes ]]; then
awk '$9 >= 400 && $9 <= 451 {print $1,$9}' "$filename" | top_results
fi
if [[ $show_most_bytes == yes ]]; then
awk '{print $1,$10}' "$filename" | top_results
fi

Tail recursion in Bash

I've tried to write a script to verify that all the stats of a metrics are positive before I make any further changes using the service. The part I'm stuck at is thinking over how to tail the recursion for the following use-case :
function load_cache() {
cacheStat=( $(curl -s -X GET "http://localhost:${MET_PORT}/metrics" | sed 's/\\\\\//\//g' | sed 's/[{}]//g' | awk -v k="cacheSize" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | sed 's/\"\:\"/\|/g' | sed 's/[\,]/ /g' | sed 's/\"//g' | grep -w "cacheSize" | cut -d ':' -f 2) )
# the above gives me the ouput(cacheStat) as -
# 2.0
# 311.0
# 102.0
count=0
for index in ${!cacheStat[*]}
do
if [[ ${cacheStat[$index]} -le 0 ] && [ $count -lt 3 ]]; then
sleep .5
count=$[$count +1];
load_cache
#Wouldn't the above initialise `count` to 0 again.
fi
done
}
What I am trying to do is if any of the elements in the cacheStat is less than or equal to 0, then sleep for .5 secs and query the cacheStat again and perform the check on all its elements again. Though not do this more than 3 times for which I am trying to use `count.
Open to any suggestion to improve the script.
Update -
On modifying the scripts as suggested by #Inian to
RETRY_COUNT=0
function load_cache() {
cacheStat=( $(curl -s -X GET "http://localhost:${MET_PORT}/metrics" | sed 's/\\\\\//\//g' | sed 's/[{}]//g' | awk -v k="cacheSize" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | sed 's/\"\:\"/\|/g' | sed 's/[\,]/ /g' | sed 's/\"//g' | grep -w "cacheSize" | cut -d ':' -f 2) );
for index in ${!cacheStat[*]}
do
echo "Stat - ${cacheStat[$index]}"
if (( ${cacheStat[$index]} <= 0 )) && (( $RETRY_COUNT < 3 )); then
echo "Attempt count - ${RETRY_COUNT}"
sleep .5s
RETRY_COUNT=$((RETRY_COUNT +1));
load_cache
fi
done
}
The logs read -
> > + cacheStat=($(curl -s -X GET "http://localhost:${MET_PORT}/metrics" | sed 's/\\\\\//\//g' | sed
> 's/[{}]//g' | awk -v k="cacheSize"
> > '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | sed
> > 's/\"\:\"/\|/g' | sed 's/[\,]/ /g' | sed 's/\"//g' | grep -w
> > "cacheSize" | cut -d ':' -f 2))
> > ++ curl -s -X GET http://localhost:8181/metrics
> > ++ sed 's/\\\\\//\//g'
> > ++ sed 's/[{}]//g'
> > ++ sed 's/[\,]/ /g'
> > ++ awk -v k=cacheSize '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}'
> > ++ sed 's/\"\:\"/\|/g'
> > ++ cut -d : -f 2
> > ++ sed 's/\"//g'
> > ++ grep -w cacheSize
It doesn't even iterate I guess.
Remove the infinite recursion by moving the count=0 outside the function body.
Also your script has couple of issues, a syntax violation and an outdated construct, lines 12-14 should have been,
if [[ ${cacheStat[$index]} -le 0 ]] && [[ $count -lt 3 ]]; then
sleep .5s
count=$((count +1));
load_cache
fi
or) use a more readable arithmetic operator, (()) in the if-clause as
if (( ${cacheStat[$index]} <= 0 )) && (( $count < 3 )); then
bash does not inherently support floating point arithmetic (comparison in your case), use a third party tool like bc, awk for this,
if (( $(echo "${cacheStat[$index]} <= 0" | bc -l) )) && (( $count < 3 )); then
You can avoid all that ad-hoc JSON parsing by using a JSON parser.
# Avoid using Bash-only "function" keyword
load_cache () {
local try
for try in 1 2 3; do
# Suction: jq doesn't return non-zero exit code for no match
# work around that by piping to grep .
if curl -s -X GET "http://localhost:${MET_PORT}/metrics" |
jq '.[] | select(cacheSize < 0)' |
grep .
then
# Notice also redirection to stderr for diagnostic messages
echo "$0: Attempt $try failed, sleeping before retrying" >&2
sleep 0.5
else
# Return with success, we are done, exit function
return 0
fi
done
# Return failure
return 1
}
I see no reason to prefer recursion over a straightforward for loop for controlling the number of retries.
If you never want to see the offending values, you can use grep -q in the conditional. I'm expecting you would do load_cache >/dev/null if you don't want the output.
If you want to see the non-offending values, the code will need some refactoring, but I'm focusing on getting the central job done elegantly and succinctly. Here's a sketch, mainly to show you the jq syntax for that.
load_cache () {
local try
local results
for try in 1 2 3; do
results=$(curl -s -X GET "http://localhost:${MET_PORT}/metrics" |
jq '.[] | .cacheSize' | tr '\n' ' ')
echo "$0: try $try: cacheSize $results" >&2
# Funky: massage the expression we test againt into a normalized form
# so that we know that the value will always be preceded by a space
case " $results " in
*" 0 "* | *" -"* )
case $try in
3) echo "$0: try $try failed; aborting" >&2 ;;
*) echo "$0: try $try failed; sleeping before retrying" >&2
sleep 0.5 ;;
esac;;
*) return 0
esac
done
return 1
}
The nested case to avoid sleeping on the final iteration isn't particularly elegant, but at least it should ensure that the reader is awake. /-8

Converting multiple lines of bash in to a single line

Is there any short and easy way to convert multiple lines of script in to a single line to be parsed in a eval command?
ie
getent group | cut -f3 -d":" | sort -n | uniq -c |\
while read x ; do
[ -z "${x}" ] && break
set - $x ; if [ $1 -gt 1 ]; then
grps=`getent group | nawk -F: '($3 == n) { print $1 }' n=$2 | xargs` ; echo "Duplicate GID ($2): ${grps}" ; fi done
one_line=`cat your_script_file | sed ":a s/[\]$//; N; s/[\]$//; s/\n/ /; t a ;"`
echo $one_line

BASH: Remove newline for multiple commands

I need some help . I want the result will be
UP:N%:N%
but the current result is
UP:N%
:N%
this is the code.
#!/bin/bash
UP=$(pgrep mysql | wc -l);
if [ "$UP" -ne 1 ];
then
echo -n "DOWN"
else
echo -n "UP:"
fi
df -hl | grep 'sda1' | awk ' {percent+=$5;} END{print percent"%"}'| column -t && echo -n ":"
top -bn2 | grep "Cpu(s)" | \sed "s/.*, *\([0-9.]*\)%* id.*/\1/" | \awk 'END{print 100 - $1"%"}'
You can use command substitution in your first sentence (notice you're creating a subshell in this way):
echo -n $(df -hl | grep 'sda1' | awk ' {percent+=$5;} END{print percent"%"}'| column -t ):

Variable loss in redirected bash while loop

I have the following code
for ip in $(ifconfig | awk -F ":" '/inet addr/{split($2,a," ");print a[1]}')
do
bytesin=0; bytesout=0;
while read line
do
if [[ $(echo ${line} | awk '{print $1}') == ${ip} ]]
then
increment=$(echo ${line} | awk '{print $4}')
bytesout=$((${bytesout} + ${increment}))
else
increment=$(echo ${line} | awk '{print $4}')
bytesin=$((${bytesin} + ${increment}))
fi
done < <(pmacct -s | grep ${ip})
echo "${ip} ${bytesin} ${bytesout}" >> /tmp/bwacct.txt
done
Which I would like to print the incremented values to bwacct.txt, but instead the file is full of zeroes:
91.227.223.66 0 0
91.227.221.126 0 0
127.0.0.1 0 0
My understanding of Bash is that a redirected for loop should preserve variables. What am I doing wrong?
First of all, simplify your script! Usually there are many better ways in bash. Also most of the time you can rely on pure bash solutions instead of running awk or other tools.
Then add some debbuging!
Here is a bit refactored script with debugging
#!/bin/bash
for ip in "$(ifconfig | grep -oP 'inet addr:\K[0-9.]+')"
do
bytesin=0
bytesout=0
while read -r line
do
read -r subIp _ _ increment _ <<< "$line"
if [[ $subIp == "$ip" ]]
then
((bytesout+=increment))
else
((bytesin+=increment))
fi
# some debugging
echo "line: $line"
echo "subIp: $subIp"
echo "bytesin: $bytesin"
echo "bytesout: $bytesout"
done <<< "$(pmacct -s | grep "$ip")"
echo "$ip $bytesin $bytesout" >> /tmp/bwacct.txt
done
Much clearer now, huh? :)

Resources