I wrote a small bash script in this post: How to search for a string in a text file and perform a specific action based on the result
I noticed that when I ran the script and check the logs, everything appears to be working but when I look at the Nagios UI, almost half of the servers listed in my text file did not get their notifications disabled. A revised version of the script is below:
host=/Users/bob/wsus.txt
password="P#assw0rd123"
while read -r host; do
region=$(echo "$host" | cut -f1 -d-)
if [[ $region == *sea1* ]]
then
echo "Disabling host notifications for: $host"
curl -vs -o /dev/null -d "cmd_mod=2&cmd_typ=25&host=$host&btnSubmit=Commit" https://nagios.$region.blah.com/nagios/cgi-bin/cmd.cgi" -u "bob:$password" -k 2>&1
else
echo "Disabling host notifications for: $host"
curl -vs -o /dev/null -d "cmd_mod=2&cmd_typ=25&host=$host&btnSubmit=Commit" https://nagios.$region.blah02.com/nagios/cgi-bin/cmd.cgi" -u "bob:$password" -k 2>&1
fi
done < wsus.txt >> /Users/bob/disable.log 2>&1
If i run the command against the servers having the issue manually, it does get disabled in the Nagios UI, so I'm a bit confused. FYI, I'm not well versed in Bash either so this was my attempt at trying to automate this process a bit.
1 - There is a missing double-quote before the first https occurence:
You have:
curl -vs -o /dev/null -d "cmd_mod=2&cmd_typ=25&host=$host&btnSubmit=Commit" https://nagios.$region.blah.com/nagios/cgi-bin/cmd.cgi" -u "bob:$password" -k 2>&1
Should be:
curl -vs -o /dev/null -d "cmd_mod=2&cmd_typ=25&host=$host&btnSubmit=Commit" "https://nagios.$region.blah.com/nagios/cgi-bin/cmd.cgi" -u "bob:$password" -k 2>&1
2 - Your first variable host is never used (overwritten inside the while loop).
I'm guessing what you were trying to do was something like:
hosts_file="/Users/bob/wsus.txt"
log_file="/Users/bob/disable.log"
# ...
while read -r host; do
# Do stuff with $host
done < $hosts_file >> $log_file 2>&1
3 - This looks suspicious to me:
if [[ $region == *sea1* ]]
Note: I haven't tested it yet, so this is my general feeling about this, might be wrong.
The $region isn't double-quoted, so make sure there could be no spaces / funny stuff happening there (but this should not be a problem inside a double-bracket test [[).
The *sea* looks like it would be expanded to match your current directory files matching this globbing. If you want to test this as a regular expression, you should use ~= operator or (my favorite for some reason) grep command:
if grep -q ".*sea.*" <<< "$region"; then
# Your code if match
else
# Your code if no match
fi
The -q keeps grep quiet
There is no need for test like [ or [[ because the return code of grep is already 0 if any match
The <<< simply redirects the right strings as the standard input of the left command (avoid useless piping like echo "$region" | grep -q ".*sea.*").
If this doesn't solve your problem, please provide a sample of your input file hosts_file as well as some output logs.
You could also try to see what's really going on under the hood by enclosing your script with set -x and set +x to activate debug/trace mode.
Related
I have made a script below. The script is checking responde code of every URL listed in the for example .csv file in column A. Everything works as I planed but after checking all URL`s the script is freezed. I have to do ctrl+c combination to stop it. How can I make script automaticly end the run after all URL's are checked.
#!/bin/bash
for link in `cat $1` $2;
do
response=`curl --output /dev/null --silent --write-out %{http_code} $link`;
if [ "$response" == "$2" ]; then
echo "$link";
fi
done
Your script hangs due to $2 in the for link line (when it hangs, check ps aux | grep curl and you'll find a curl process with the response code as the last argument). Also, for link in `cat $1` $2 is not how you should read and process lines from a file.
Assuming your example.csv file only contains one URL per row and nothing else (which basically makes it a plain text file), this code should do what you want:
#!/usr/bin/env bash
while read -r link; do
response=$(curl --output /dev/null --silent --write-out %{http_code} "$link")
if [[ "$response" == "$2" ]]; then
echo "$link"
fi
done < "$1"
Copying your code verbatim and fudging a test file with a few urls separated by whitespace to test with, it does indeed hang. However, removing the $2 from the end of the for allows the script to finish.
for link in `cat $1`;
Here is the chunk of code for reference:-
Output:
I have checked the variable values using echo and those looks fine.
But what I want do achieve is searching logs on remote hosts using grep which does not give any output.
for dir in ${log_path}
do
for host in ${Host}
do
if [[ "${userinputserverhost}" == "${host}" ]]
then
ssh -q -T username#userinputserverhost "bash -s" <<-'EOF' 2>&1 | tee -a ${LogFile}
echo -e "Fetching details: \n"
`\$(grep -A 5 -s "\${ID}" "\${dir}"/archive/*.log)`
EOF
fi
break
done
done
First, remove all the crap around the grep.
Second, you're overquoting your vars.
Third, skip the "bash -s" if you can.
ssh -q -T username#userinputserverhost <<-'EOF' 2>&1 | tee -a ${LogFile}
echo -e "Fetching details: \n"
grep -A 5 -s "${ID}" "${dir}"/archive/*.log
EOF
Fourth, I don't see where $ID is set...so if that's being loaded on the remote system by the login or something, then that one would need the dollar sign backslashed.
Finally, be aware that here-docs are great, but sometimes here-strings are simpler if you can spare the quotes.
$: ssh 2>&1 dudeling#sandbox-server '
> date
> whoami
> ' | tee -a foo.txt
Fri Apr 30 09:23:09 EDT 2021
dudeling
$: cat foo.txt
Fri Apr 30 09:23:09 EDT 2021
dudeling
That one is more a matter of taste. Even better, if you can, write your remote-script to a local file & use that. And of course, you can always add set -vx into the script to see what gets remotely executed.
cat >tmpScript <<-'EOF'
echo -e "Fetching details: \n"
set -vx
grep -A 5 -s "${ID}" "${dir}"/archive/*.log
EOF
ssh <tmpScript 2>&1 -q -T username#userinputserverhost | tee -a ${LogFile}
Now you have an exact copy of what was issued for debugging.
Thanks Paul for spending time and coming up with suggestions/solutions.
I have managed to get it working couple of days back. Would have felt happy to say that your solution worked 100% but even satisfied that I got it sorted on my own as it helped me learn some new stuff.
FYI - grep -A 5 -s "${ID}" "${dir}"/archive/*.log - this will work but only by using shell built-in 'declare -p' to declare the variables within EOF. Also, I read somewhere and it is recommended to use EOF unqouted as it caters variable expansion to remote hosts without any trouble.
Below piece of code is working for me in bash:
ssh -q -T username#userinputserverhost <<-EOF 2>&1 | tee -a ${LogFile}
echo -e "Fetching details: \n"
$(declare -p ID)
$(declare -p dir)
grep -A 5 -s "${ID}" "${dir}"/archive/*.log
EOF
I'm creating a bash script to check the HTTP headers on remote hosts, I'm doing this via cURL and have noted that appending http://{host} will only work for services running on tcp\80, and not tcp\443. For example for HTTPS services, you require curl -I -k {host}, as opposed to HTTP services which only required curl -I {host}. This is my script:
for host in $(cat file.txt); do
echo " "
echo "Current host: "${host}
curl -I -k https://${host}
echo " "
echo "=============================================="
done
Now what I'm wanting is some condition operator to check that if the output is "Could not resolve host" then the script should run "curl -I http://{host}" on those hosts which the stdout contained the str value "Could not resolve host".
How can I achieve this in bash?
stdout will not contain Could not resolve host though, that's output to stderr. While you could capture stderr and then do string matching, there is a much, much simpler solution: exit code.
You can see here that curl will always exit with code 6 when it fails to resolve host. Thus, simply testing the exit code is sufficient:
curl -i -k http://nowaythisthingexists.test
if [[ $? -eq 6 ]]
then
echo "oopsie, couldn't resolve host!"
fi
Alternately, if you really want to do it by matching strings, make sure to redirect stderr to stdout (and possibly also kill stdout so it doesn't interfere):
output=$(curl -i -k http://nowaythisthingexists.test 2>&1 >/dev/null)
if [[ "$output" = *"Could not resolve host"* ]]
then
echo "oopsie, couldn't resolve host!"
fi
Obviously, you are not getting the output of your request this way, so you'd need to redirect it somewhere more useful than /dev/null — a file, or a Unix pipe. Now it's getting more complicated than it needs to be.
Given the following 2 lines in a bash script,
LOCATION=$(curl -i -H "Windmill-Name: $APPLICATION_NAME" -H "Windmill-Identifier: $CFBundleIdentifier" -F "ipa=#$IPA" -F "plist=#$PLIST" $WINDMILL_BASE_URL/windmill/rest/windmill/$USER | grep ^Location | awk '{print $2}')
echo "[windmill] Use $LOCATION for accessing '$APPLICATION_NAME'"
In some cases, the echo string appears like below.
for accessing 'MultiPartIOSDemo'[a truncated $LOCATION]
The behaviour is not consistent but when it reproduces, the malformed output is consistent (i.e. a truncated $LOCATION at some range).
It looks like echo outputs the string to the buffer but the piping of curl isn't yet done and ends up writing its output on top.
Can't quite tell.
Update
Tried all of your suggestions, but the same problem occurs.
Have now dropped grep and the script looks like so
LOCATION=$(curl -i -H "Windmill-Name: $APPLICATION_NAME" -H "Windmill-Identifier: $CFBundleIdentifier" -F "ipa=#$IPA" -F "plist=#$PLIST" "$WINDMILL_BASE_URL/windmill/rest/windmill/$USER" | awk -W '/^Location/ {print $2}')
echo "[windmill] Use $LOCATION for accessing '$APPLICATION_NAME'"
Here are some more details.
The script that includes the above lines is wrapped in a
(
(
# bash script
) 2>&1 | tee $HOME/.windmill/$PROJECT_NAME.log
) 2>&1 | tee $HOME/.windmill/windmill.log
Hence the output of the echo is on both logs.
Just noticed that the above behaviour occurs while tailing, e.g.
tail -fn 20 ~/.windmill/windmill.log
However if I do a
more ~/.windmill/windmill.log
I can see that the echo message appears correctly. Notice the newline character "^M". Wondering if it has anything to do with the way tail parses the log.
[windmill] Use [correct $LOCATION] ^M for accessing 'MultiPartIOSDemo'
Clarified Question
Guess after all the above, there are really 2 questions.
Under what circumstances does the ^M appear in the log?
Why is tail parsing the log wrong, i.e. parsing ^M in such a way that it first outputs the "for accessing 'MultiPartIOSDemo'" then the "Use $LOCATION" on top.
I would (1) change
$WINDMILL_BASE_URL/windmill/rest/windmill/$USER
to
"$WINDMILL_BASE_URL/windmill/rest/windmill/$USER"
(2) use tee to debug
and (3) use awk to match "^location" instead of grep
LOCATION=$(curl -i -H "Windmill-Name: $APPLICATION_NAME" -H "Windmill-Identifier: $CFBundleIdentifier" -F "ipa=#$IPA" -F "plist=#$PLIST" "$WINDMILL_BASE_URL/windmill/rest/windmill/$USER" | tee /tmp/debug.$$ | awk ' /^Location/ {print $2}')
echo "[windmill] Use $LOCATION for accessing '$APPLICATION_NAME'"
Perhaps curl is returing an error. You can check the results and only parse it if no error occured
# Create a trace file
tfile=/tmp/trace.$$
# uncomment this line if you want to delete the trace file at the end of the script
#trap "/bin/rm $tfile" 0 1 15
curl -i -H "Windmill-Name: $APPLICATION_NAME" -H "Windmill-Identifier: $CFBundleIdentifier" -F "ipa=#$IPA" -F "plist=#$PLIST" "$WINDMILL_BASE_URL/windmill/rest/windmill/$USER" >$tfile
status=$?
if [ $status -eg 0 ]
then
echo curl was successful
LOCATION=$(awk '/^Location/ {print $2}' <$tfile)
echo "[windmill] Use $LOCATION for accessing '$APPLICATION_NAME'"
else
echo curl exited with status $status
fi
Should be fairly simple to answer:
Let's say I wanted to execute a command determined by the output of a previous one in Bash:
curl http://website.com 2> /dev/null | grep -i "test" --count | <MY-COMMAND>
What I need: <MY-COMMAND> should only execute if grep had some matches (at least 1).
How can I achieve that?
Also, please feel free to add matching tags, I couldn't come up with any
ifne utility ("run a program if the standard input is not empty") from Jeoy Hess's moreutils package will serve you.
A description of it:
a command that would run the following
command if and only if the standard
input is not empty. I often want this
in crontabs, as in:
find . -name core | ifne mail -s "Core files found" root
Do you need the output of grep to be piped to your command? The answer is simpler if you do not. In that case since grep's return code is success only if it finds a match, you can use && or if:
curl http://website.com 2> /dev/null | grep -q -i "test" && <MY-COMMAND>
if curl http://website.com 2> /dev/null | grep -q -i "test"; then
<MY-COMMAND>
fi
The && operator is a shorthand way of performing an if-else check. It is a short-circuiting operator, which means that the right hand side will only be executed if the left hand side fails.
If you need to pipe the output to your command then you'll need to save the output to a temporary file, test for a match, and then execute your command:
if curl http://website.com 2> /dev/null | grep -i "test" > /tmp/grep.txt; then
<MY-COMMAND> < /tmp/grep.txt
fi
curl http://website.com 2> /dev/null | grep -i "test" && <MY-COMMAND>
From the grep man page: "the exit status is 0 if selected lines are found and 1 otherwise"
The command after && is executed only if the previous command returned exit status 0.
curl http://www.google.com 2>/dev/null | grep window -i -c && echo "this is a success"