Should be fairly simple to answer:
Let's say I wanted to execute a command determined by the output of a previous one in Bash:
curl http://website.com 2> /dev/null | grep -i "test" --count | <MY-COMMAND>
What I need: <MY-COMMAND> should only execute if grep had some matches (at least 1).
How can I achieve that?
Also, please feel free to add matching tags, I couldn't come up with any
ifne utility ("run a program if the standard input is not empty") from Jeoy Hess's moreutils package will serve you.
A description of it:
a command that would run the following
command if and only if the standard
input is not empty. I often want this
in crontabs, as in:
find . -name core | ifne mail -s "Core files found" root
Do you need the output of grep to be piped to your command? The answer is simpler if you do not. In that case since grep's return code is success only if it finds a match, you can use && or if:
curl http://website.com 2> /dev/null | grep -q -i "test" && <MY-COMMAND>
if curl http://website.com 2> /dev/null | grep -q -i "test"; then
<MY-COMMAND>
fi
The && operator is a shorthand way of performing an if-else check. It is a short-circuiting operator, which means that the right hand side will only be executed if the left hand side fails.
If you need to pipe the output to your command then you'll need to save the output to a temporary file, test for a match, and then execute your command:
if curl http://website.com 2> /dev/null | grep -i "test" > /tmp/grep.txt; then
<MY-COMMAND> < /tmp/grep.txt
fi
curl http://website.com 2> /dev/null | grep -i "test" && <MY-COMMAND>
From the grep man page: "the exit status is 0 if selected lines are found and 1 otherwise"
The command after && is executed only if the previous command returned exit status 0.
curl http://www.google.com 2>/dev/null | grep window -i -c && echo "this is a success"
Related
I have a bash/zsh command with multiple pipes | that fails when using set -o pipefail. For simplicity assume the command is
set -o pipefail; echo "123456" | head -c2 | grep 5 | cat
How do I quickly find out which command is the first to fail and why? I know I can check the exit code, but that doesn't show which part of the pipeline failed first.
Is there something simpler than the rather verbose check of building up the pipeline one by one and checking for the first failing exit code?
Edit: I removed the contrived code example I made up as it confused people about my purpose of asking. The actual command that prompted this question was:
zstdcat metadata.tsv.zst | \
tsv-summarize -H --group-by Nextclade_pango --count | \
tsv-filter -H --ge 'count:2' | \
tsv-select -H -f1 >open_lineages.txt
In bash, use echo "${PIPESTATUS[#]}" right after the command to get the exit status for each component in a space separated list:
#!/bin/bash
$ set -o pipefail; echo "123456" | head -c2 | grep 5 | cat
$ echo ${PIPESTATUS[#]}
0 0 1 0
Beware zsh users, you need to use the lower case pipestatus instead:
#!/bin/zsh
$ set -o pipefail; echo "123456" | head -c2 | grep 5 | cat
$ echo $pipestatus
0 0 1 0
In fish you can also simply use echo $pipestatus for the same output.
${PIPESTATUS[#]} right after is the answer you were looking for. However, I want to advise on the first example. It's a good habit to anticipate error, so instead of testing after you should have check the path prior everything.
if [ -d "/nonexistent_directory" ]; then
# here pipe shouldn't fail to grep
# ...unless there's something wrong with "foo"
# ...but "grep" may be a failure if the pattern isn't found
if ! ls -1 "/nonexistent_directory" | grep 'foo' ; then
echo "The command 'grep foo' failed."
# else
# echo "The pipeline succeeded."
fi
else
echo "The command 'ls /nonexistent_directory' failed."
fi
Whenever possible, avoid greping ls output in script, that' fragile...
I'm currently monitoring a log file and my ultimate goal is to write a script that uses tail -n0 -f and execute a certain command once grep finds a correspondence. My current code:
tail -n 0 -f $logfile | grep -q $pattern && echo $warning > $anotherlogfile
This works but only once, since grep -q stops when it finds a match. The script must keep searching and running the command, so I can update a status log and run another script to automatically fix the problem. Can you give me a hint?
Thanks
use a while loop
tail -n 0 -f "$logfile" | while read LINE; do
echo "$LINE" | grep -q "$pattern" && echo "$warning" > "$anotherlogfile"
done
awk will let us continue to process lines and take actions when a pattern is found. Something like:
tail -n0 -f "$logfile" | awk -v pattern="$pattern" '$0 ~ pattern {print "WARN" >> "anotherLogFile"}'
If you need to pass in the warning message and path to anotherLogFile you can use more -v flags to awk. Also, you could have awk take the action you want instead. It can run commands via the system() function where you pass the shell command to run
I wrote a small bash script in this post: How to search for a string in a text file and perform a specific action based on the result
I noticed that when I ran the script and check the logs, everything appears to be working but when I look at the Nagios UI, almost half of the servers listed in my text file did not get their notifications disabled. A revised version of the script is below:
host=/Users/bob/wsus.txt
password="P#assw0rd123"
while read -r host; do
region=$(echo "$host" | cut -f1 -d-)
if [[ $region == *sea1* ]]
then
echo "Disabling host notifications for: $host"
curl -vs -o /dev/null -d "cmd_mod=2&cmd_typ=25&host=$host&btnSubmit=Commit" https://nagios.$region.blah.com/nagios/cgi-bin/cmd.cgi" -u "bob:$password" -k 2>&1
else
echo "Disabling host notifications for: $host"
curl -vs -o /dev/null -d "cmd_mod=2&cmd_typ=25&host=$host&btnSubmit=Commit" https://nagios.$region.blah02.com/nagios/cgi-bin/cmd.cgi" -u "bob:$password" -k 2>&1
fi
done < wsus.txt >> /Users/bob/disable.log 2>&1
If i run the command against the servers having the issue manually, it does get disabled in the Nagios UI, so I'm a bit confused. FYI, I'm not well versed in Bash either so this was my attempt at trying to automate this process a bit.
1 - There is a missing double-quote before the first https occurence:
You have:
curl -vs -o /dev/null -d "cmd_mod=2&cmd_typ=25&host=$host&btnSubmit=Commit" https://nagios.$region.blah.com/nagios/cgi-bin/cmd.cgi" -u "bob:$password" -k 2>&1
Should be:
curl -vs -o /dev/null -d "cmd_mod=2&cmd_typ=25&host=$host&btnSubmit=Commit" "https://nagios.$region.blah.com/nagios/cgi-bin/cmd.cgi" -u "bob:$password" -k 2>&1
2 - Your first variable host is never used (overwritten inside the while loop).
I'm guessing what you were trying to do was something like:
hosts_file="/Users/bob/wsus.txt"
log_file="/Users/bob/disable.log"
# ...
while read -r host; do
# Do stuff with $host
done < $hosts_file >> $log_file 2>&1
3 - This looks suspicious to me:
if [[ $region == *sea1* ]]
Note: I haven't tested it yet, so this is my general feeling about this, might be wrong.
The $region isn't double-quoted, so make sure there could be no spaces / funny stuff happening there (but this should not be a problem inside a double-bracket test [[).
The *sea* looks like it would be expanded to match your current directory files matching this globbing. If you want to test this as a regular expression, you should use ~= operator or (my favorite for some reason) grep command:
if grep -q ".*sea.*" <<< "$region"; then
# Your code if match
else
# Your code if no match
fi
The -q keeps grep quiet
There is no need for test like [ or [[ because the return code of grep is already 0 if any match
The <<< simply redirects the right strings as the standard input of the left command (avoid useless piping like echo "$region" | grep -q ".*sea.*").
If this doesn't solve your problem, please provide a sample of your input file hosts_file as well as some output logs.
You could also try to see what's really going on under the hood by enclosing your script with set -x and set +x to activate debug/trace mode.
Given the following 2 lines in a bash script,
LOCATION=$(curl -i -H "Windmill-Name: $APPLICATION_NAME" -H "Windmill-Identifier: $CFBundleIdentifier" -F "ipa=#$IPA" -F "plist=#$PLIST" $WINDMILL_BASE_URL/windmill/rest/windmill/$USER | grep ^Location | awk '{print $2}')
echo "[windmill] Use $LOCATION for accessing '$APPLICATION_NAME'"
In some cases, the echo string appears like below.
for accessing 'MultiPartIOSDemo'[a truncated $LOCATION]
The behaviour is not consistent but when it reproduces, the malformed output is consistent (i.e. a truncated $LOCATION at some range).
It looks like echo outputs the string to the buffer but the piping of curl isn't yet done and ends up writing its output on top.
Can't quite tell.
Update
Tried all of your suggestions, but the same problem occurs.
Have now dropped grep and the script looks like so
LOCATION=$(curl -i -H "Windmill-Name: $APPLICATION_NAME" -H "Windmill-Identifier: $CFBundleIdentifier" -F "ipa=#$IPA" -F "plist=#$PLIST" "$WINDMILL_BASE_URL/windmill/rest/windmill/$USER" | awk -W '/^Location/ {print $2}')
echo "[windmill] Use $LOCATION for accessing '$APPLICATION_NAME'"
Here are some more details.
The script that includes the above lines is wrapped in a
(
(
# bash script
) 2>&1 | tee $HOME/.windmill/$PROJECT_NAME.log
) 2>&1 | tee $HOME/.windmill/windmill.log
Hence the output of the echo is on both logs.
Just noticed that the above behaviour occurs while tailing, e.g.
tail -fn 20 ~/.windmill/windmill.log
However if I do a
more ~/.windmill/windmill.log
I can see that the echo message appears correctly. Notice the newline character "^M". Wondering if it has anything to do with the way tail parses the log.
[windmill] Use [correct $LOCATION] ^M for accessing 'MultiPartIOSDemo'
Clarified Question
Guess after all the above, there are really 2 questions.
Under what circumstances does the ^M appear in the log?
Why is tail parsing the log wrong, i.e. parsing ^M in such a way that it first outputs the "for accessing 'MultiPartIOSDemo'" then the "Use $LOCATION" on top.
I would (1) change
$WINDMILL_BASE_URL/windmill/rest/windmill/$USER
to
"$WINDMILL_BASE_URL/windmill/rest/windmill/$USER"
(2) use tee to debug
and (3) use awk to match "^location" instead of grep
LOCATION=$(curl -i -H "Windmill-Name: $APPLICATION_NAME" -H "Windmill-Identifier: $CFBundleIdentifier" -F "ipa=#$IPA" -F "plist=#$PLIST" "$WINDMILL_BASE_URL/windmill/rest/windmill/$USER" | tee /tmp/debug.$$ | awk ' /^Location/ {print $2}')
echo "[windmill] Use $LOCATION for accessing '$APPLICATION_NAME'"
Perhaps curl is returing an error. You can check the results and only parse it if no error occured
# Create a trace file
tfile=/tmp/trace.$$
# uncomment this line if you want to delete the trace file at the end of the script
#trap "/bin/rm $tfile" 0 1 15
curl -i -H "Windmill-Name: $APPLICATION_NAME" -H "Windmill-Identifier: $CFBundleIdentifier" -F "ipa=#$IPA" -F "plist=#$PLIST" "$WINDMILL_BASE_URL/windmill/rest/windmill/$USER" >$tfile
status=$?
if [ $status -eg 0 ]
then
echo curl was successful
LOCATION=$(awk '/^Location/ {print $2}' <$tfile)
echo "[windmill] Use $LOCATION for accessing '$APPLICATION_NAME'"
else
echo curl exited with status $status
fi
I'm trying write simple notify app in bash. I want to read output from mplayer, parse it and display through notify-send.
I can get desired info from mplayer using this:
mplayer <url> | grep ICY
and then parse in using sed.
I create named pipe, tell mplayer to write it and then I'm reading from it. Unfortunately, it doesn't work. Here's my script:
$fifo=~/.rp/fifo
mkfifo $fifo
mplayer <url> 2>/dev/null | grep ICY 1> $fifo &
while read line < $fifo; do
echo $line
done
wait
Program keeps waiting to input from $fifo. I tried following in other terminal, while this script is running:
Run
echo "Test" > .rp/fifo
Terminal with running script shows "Test"
Run
echo "ICY" | grep ICY > .rp/fifo
also works.
Run
mplayer <url> | grep ICY > .rp/fifo
and it doesn't work.
Is I said above, the combination of mplayer | grep works fine. grep > $fifo works fine. I don't understand why mplayer | grep > $fifo doesn't work.
I suspect you might be experiencing the C library's fully buffered mode for streams. You don't say that you're running the GNU userspace, but if you are, you can look into stdbuf(1) to modify the buffering regime.
You might try first running just grep as a child of stdbuf(1), like this:
mplayer <url> | stdbuf -o L grep ICY > .rp/fifo
If that doesn't work, moar is bettar!
stdbuf -o 0 mplayer <url> | stdbuf -o L grep ICY > .rp/fifo
And if that still doesn't work, then it's possible that mplayer isn't writing to stdout, but directly to /dev/tty. In which case, you will need to read up on expect(1).
You could do unbuffered grep with:
$ mplayer ... 2>&1 | grep --line-buffered "ICY"
or better:
$ mplayer ... 2>&1 | sed -une 's/^.*ICY[^:]*: //p'
or even, why not (sed is very nice for grep and formatting),
this will grep ICY lines and even split line containing - in a first field of 30 chars length separed by a : from a second field:
$ mplayer ... 2>&1 |
sed -une "
/ICY/{
s/^.*ICY[^:]*:.*'\([^']*\)';/\1/;
s/^\(.*\) - /\1 - /;
s/^\(.\{30\}\) *- /\1: /;
p;
}"
could give something like:
Artist name : Song title
Other artist : Other song
Unsplited line
Artist : Title
I start mplayer in slave mode, using FIFO file.
mkfifo /tmp/mpfifo
mplayer -slave -input file=/tmp/mpfifo video.mp4
I am able to control the video player from another terminal.
echo "pause" >> /tmp/mpfifo
echo "volume 50" > /tmp/mpfifo
I want to get value (for example current position of playing video). So I tried:
echo "get_time_pos" > /tmp/mpfifo
But no value returned.
I searched for hours, but no success.
Then I thought to redirect mplayer output to a file:
mplayer -slave -input file=/tmp/mpfifo video.mp4 > /tmp/mpout.txt
After that when like following commands executed:
echo "get_time_pos" > /tmp/mpfifo
echo "get_property length" > /tmp/mpfifo
The outputs in /tmp/mpout.txt was like:
.......
.......
ANS_TIME_POSITION=113.6
ANS_length=2534.602031
If the result of each command would return to the command line would be very nice. Even it may need some works, the output file can be parsed, though.