Shell scripting, store command output in variable and preserve formatting [duplicate] - bash

This question already has answers here:
Closed 11 years ago.
Possible Duplicate:
Capturing multiple line output to a bash variable
I have what is probably a basic scripting question, but I haven't been able to find an answer anywhere that I've looked.
I have an awk script that processes a file, and spits out a list of disabled systems. When I call it manually from the command line, I get formatted output back:
$awk -f awkscript datafile
The following notifications are currently disabled:
host: bar
host: foo
I am writing a wrapper script to call from my crontab, which will run the awk script, determine if there is any output, and email me if there is. It looks like (simplified):
BODY=`awk -f awkscript datafile`
if [ -n "$BODY" ]
then
echo $BODY | mailx -s "Disabled notifications" me#mail.address
else
echo "Nothing is disabled"
fi
When run this way, and confirmed by adding an echo $BODY into the script, the output is stripped of the formatting (newlines are mainly what I'm concerned with), so I get output that looks like:
The following notitifications are currently disabled: host: bar host: foo
I'm trying to figure out how to preserve the formatting that is present if I run the command manually.
Things I've tried so far:
echo -e `cat datafile | awkscript` > /tmp/tmpfile
echo -e /tmp/tmpfile
I tried this because on my system (Solaris 5.10), using echo without the -e ignores standard escape sequences like \n . Didn't work. I checked the tmpfile, and it doesn't have any formatting in it, so the problem is happening when storing the output, not when printing it out.
BODY="$(awk -f awkscript datafile)"
echo -e "$BODY"
I tried this because everything I could find, including some other questions here on stackoverflow said that the problem was that the shell would replace whitespace codes with spaces if it wasn't quoted. Didn't work.
I've tried using printf instead of echo, using $(command) instead of `command`, and using a tempfile instead of a variable to store the output, but nothing seems to retain the formatting.
What am I missing, or is there another way to do this which avoids this problem all together?

BODY=`awk -f awkscript datafile`
if [ -n "$BODY" ]
then echo "$BODY" | mailx -s "Disabled notifications" me#mail.address
else echo "Nothing is disabled"
fi
Note the double quotes in the echo.
You can simplify this version, too:
echo -e `cat datafile | awkscript` > /tmp/tmpfile
echo -e /tmp/tmpfile
to just:
tmpfile=/tmp/tmpfile.$$
awkscript > $tmpfile
if [ -s $tmpfile ]
then mailx -s "Disabled notifications" me#mail.address < $tmpfile
else echo "Nothing is disabled"
fi
Backquotes are useful (but better written as $(cmd args)) but do not have to be used everywhere.

Using quotes should work, and does for me:
$ cat test.sh
#!/bin/bash -e
BODY=`cat test.in`
if [ -n "$BODY" ]; then
echo "$BODY" | mailx -s "test" username
else
echo "Nothing"
fi
$ cat test.in
The following notifications are currently disabled:
host: bar
host: foo
$ ./test.sh
$
And this sends me a properly formatted email.

Related

Cat command with pipe working in bash but not in script

Pretty simple script, but I am stuck.
It connects to a battery balancer, spits out the info into a json formatted file. I then have a pipe the output into jq to obtain the info I need.
It works in the bash shell, but not in the script:
Here is the script:
echo "Checking battery voltages"
jkbms -p 3C:A5:19:7B:28:09 -o json > /home/bms/batt.log
echo cat /home/bms/batt.log | jq -r '.highest_cell_voltage'
echo "done"
The cat line shows this in the script output:
Checking battery voltages
parse error: Invalid numeric literal at line 1, column 4
done
From the shell it works as expected:
cat /home/bms/batt.log | jq -r '.highest_cell_voltage'
4.152044773101807
I have tried enclosing the whole cat command in quotes etc, but I am at a loss.
This, however, works:
echo "Checking battery voltages"
jkbms -p 3C:A5:19:7B:28:09 -o json > /home/bms/batt.log
batt=$(cat /home/bms/batt.log)
echo $batt | jq -r '.highest_cell_voltage'
#echo /usr/bin/cat /home/bms/batt.log
echo "done"
jkbms -p 3C:A5:19:7B:28:09 -o json > /home/bms/batt.log
echo cat /home/bms/batt.log | jq -r '.highest_cell_voltage'
The echo here is wrong. By the way, you can simplify the above to:
jkbms -p 3C:A5:19:7B:28:09 -o json|tee /home/bms/batt.log|jq -r '.highest_cell_voltage'
If I need to print the output of the comand on the screen, how do i do it without using echo?
If you want the saved output in /home/bms/batt.log, you can cat /home/bms/batt.log anytime.
If you want to print the output of the comand on the screen only at the time of execution, you can tee /dev/tty instead of tee /home/bms/batt.log.
If at the time of execution you want the output on screen as well as in the log file, you can tee /home/bms/batt.log /dev/tty at once.

While loop to execute nagios commands not working properly

I wrote a small bash script in this post: How to search for a string in a text file and perform a specific action based on the result
I noticed that when I ran the script and check the logs, everything appears to be working but when I look at the Nagios UI, almost half of the servers listed in my text file did not get their notifications disabled. A revised version of the script is below:
host=/Users/bob/wsus.txt
password="P#assw0rd123"
while read -r host; do
region=$(echo "$host" | cut -f1 -d-)
if [[ $region == *sea1* ]]
then
echo "Disabling host notifications for: $host"
curl -vs -o /dev/null -d "cmd_mod=2&cmd_typ=25&host=$host&btnSubmit=Commit" https://nagios.$region.blah.com/nagios/cgi-bin/cmd.cgi" -u "bob:$password" -k 2>&1
else
echo "Disabling host notifications for: $host"
curl -vs -o /dev/null -d "cmd_mod=2&cmd_typ=25&host=$host&btnSubmit=Commit" https://nagios.$region.blah02.com/nagios/cgi-bin/cmd.cgi" -u "bob:$password" -k 2>&1
fi
done < wsus.txt >> /Users/bob/disable.log 2>&1
If i run the command against the servers having the issue manually, it does get disabled in the Nagios UI, so I'm a bit confused. FYI, I'm not well versed in Bash either so this was my attempt at trying to automate this process a bit.
1 - There is a missing double-quote before the first https occurence:
You have:
curl -vs -o /dev/null -d "cmd_mod=2&cmd_typ=25&host=$host&btnSubmit=Commit" https://nagios.$region.blah.com/nagios/cgi-bin/cmd.cgi" -u "bob:$password" -k 2>&1
Should be:
curl -vs -o /dev/null -d "cmd_mod=2&cmd_typ=25&host=$host&btnSubmit=Commit" "https://nagios.$region.blah.com/nagios/cgi-bin/cmd.cgi" -u "bob:$password" -k 2>&1
2 - Your first variable host is never used (overwritten inside the while loop).
I'm guessing what you were trying to do was something like:
hosts_file="/Users/bob/wsus.txt"
log_file="/Users/bob/disable.log"
# ...
while read -r host; do
# Do stuff with $host
done < $hosts_file >> $log_file 2>&1
3 - This looks suspicious to me:
if [[ $region == *sea1* ]]
Note: I haven't tested it yet, so this is my general feeling about this, might be wrong.
The $region isn't double-quoted, so make sure there could be no spaces / funny stuff happening there (but this should not be a problem inside a double-bracket test [[).
The *sea* looks like it would be expanded to match your current directory files matching this globbing. If you want to test this as a regular expression, you should use ~= operator or (my favorite for some reason) grep command:
if grep -q ".*sea.*" <<< "$region"; then
# Your code if match
else
# Your code if no match
fi
The -q keeps grep quiet
There is no need for test like [ or [[ because the return code of grep is already 0 if any match
The <<< simply redirects the right strings as the standard input of the left command (avoid useless piping like echo "$region" | grep -q ".*sea.*").
If this doesn't solve your problem, please provide a sample of your input file hosts_file as well as some output logs.
You could also try to see what's really going on under the hood by enclosing your script with set -x and set +x to activate debug/trace mode.

Piping curl followed by an echo some times truncates the output when using tail

Given the following 2 lines in a bash script,
LOCATION=$(curl -i -H "Windmill-Name: $APPLICATION_NAME" -H "Windmill-Identifier: $CFBundleIdentifier" -F "ipa=#$IPA" -F "plist=#$PLIST" $WINDMILL_BASE_URL/windmill/rest/windmill/$USER | grep ^Location | awk '{print $2}')
echo "[windmill] Use $LOCATION for accessing '$APPLICATION_NAME'"
In some cases, the echo string appears like below.
for accessing 'MultiPartIOSDemo'[a truncated $LOCATION]
The behaviour is not consistent but when it reproduces, the malformed output is consistent (i.e. a truncated $LOCATION at some range).
It looks like echo outputs the string to the buffer but the piping of curl isn't yet done and ends up writing its output on top.
Can't quite tell.
Update
Tried all of your suggestions, but the same problem occurs.
Have now dropped grep and the script looks like so
LOCATION=$(curl -i -H "Windmill-Name: $APPLICATION_NAME" -H "Windmill-Identifier: $CFBundleIdentifier" -F "ipa=#$IPA" -F "plist=#$PLIST" "$WINDMILL_BASE_URL/windmill/rest/windmill/$USER" | awk -W '/^Location/ {print $2}')
echo "[windmill] Use $LOCATION for accessing '$APPLICATION_NAME'"
Here are some more details.
The script that includes the above lines is wrapped in a
(
(
# bash script
) 2>&1 | tee $HOME/.windmill/$PROJECT_NAME.log
) 2>&1 | tee $HOME/.windmill/windmill.log
Hence the output of the echo is on both logs.
Just noticed that the above behaviour occurs while tailing, e.g.
tail -fn 20 ~/.windmill/windmill.log
However if I do a
more ~/.windmill/windmill.log
I can see that the echo message appears correctly. Notice the newline character "^M". Wondering if it has anything to do with the way tail parses the log.
[windmill] Use [correct $LOCATION] ^M for accessing 'MultiPartIOSDemo'
Clarified Question
Guess after all the above, there are really 2 questions.
Under what circumstances does the ^M appear in the log?
Why is tail parsing the log wrong, i.e. parsing ^M in such a way that it first outputs the "for accessing 'MultiPartIOSDemo'" then the "Use $LOCATION" on top.
I would (1) change
$WINDMILL_BASE_URL/windmill/rest/windmill/$USER
to
"$WINDMILL_BASE_URL/windmill/rest/windmill/$USER"
(2) use tee to debug
and (3) use awk to match "^location" instead of grep
LOCATION=$(curl -i -H "Windmill-Name: $APPLICATION_NAME" -H "Windmill-Identifier: $CFBundleIdentifier" -F "ipa=#$IPA" -F "plist=#$PLIST" "$WINDMILL_BASE_URL/windmill/rest/windmill/$USER" | tee /tmp/debug.$$ | awk ' /^Location/ {print $2}')
echo "[windmill] Use $LOCATION for accessing '$APPLICATION_NAME'"
Perhaps curl is returing an error. You can check the results and only parse it if no error occured
# Create a trace file
tfile=/tmp/trace.$$
# uncomment this line if you want to delete the trace file at the end of the script
#trap "/bin/rm $tfile" 0 1 15
curl -i -H "Windmill-Name: $APPLICATION_NAME" -H "Windmill-Identifier: $CFBundleIdentifier" -F "ipa=#$IPA" -F "plist=#$PLIST" "$WINDMILL_BASE_URL/windmill/rest/windmill/$USER" >$tfile
status=$?
if [ $status -eg 0 ]
then
echo curl was successful
LOCATION=$(awk '/^Location/ {print $2}' <$tfile)
echo "[windmill] Use $LOCATION for accessing '$APPLICATION_NAME'"
else
echo curl exited with status $status
fi

Lynx is stopping loop?

I'll just apologize beforehand; this is my first ever post, so I'm sorry if I'm not specific enough, if the question has already been answered and I just didn't look hard enough, and if I use incorrect formatting of some kind.
That said, here is my issue: In bash, I am trying to create a script that will read a file that lists several dozen URL's. Once it reads each line, I need it to run a set of actions on that, the first being to use lynx to navigate to the website. However, in practice, it will run once perfectly on the first line. Lynx goes, the download works, and then the subsequent renaming and organizing of that file go through as well. But then it skips all the other lines and acts like it has finished the whole file.
I have tested to see if it was lynx causing the issue by eliminating all the other parts of the code, and then by just eliminating lynx. It works without Lynx, but, of course, I need lynx for the rest of the output to be of any use to me. Let me just post the code:
!#/bin/bash
while read line; do
echo $line
lynx -accept_all_cookies $line
echo "lynx done"
od -N 2 -h *.zip | grep "4b50"
echo "od done, if 1 starting..."
if [[ $? -eq 0 ]]
then ls *.*>>logs/zips.log
else
od -N 2 -h *.exe | grep "5a4d"
echo "if 2 starting..."
if [[ $? -eq 0 ]]
then ls *.*>>logs/exes.log
else
od -N 2 -h *.exe | grep "5a4d, 4b50"
echo "if 3 starting..."
if [[ $? -eq 1 ]]
then
ls *.*>>logs/failed.log
fi
echo "if 3 done"
fi
echo "if 2 done"
fi
echo "if 1 done..."
FILE=`(ls -tr *.* | head -1)`
NOW=$(date +"%m_%d_%Y")
echo "vars set"
mv $FILE "criticalfreepri/${FILE%%.*}(ZCH,$NOW).${FILE#*.}" -u
echo "file moved"
rm *.zip *.exe
echo "file removed"
done < "lynx"
$SHELL
Just to be sure, I do have a file called "lynx" that contains the urls separated by a return each. Also, I used all those "echo"s to do my own sort of debugging, but I have tried it with and without the echo's. When I execute the script, the echo's all show up...
Any help is appreciated, and thank you all so much! Hope I didn't break any rules on this post!
PS: I'm on Linux Mint running things through the "terminal" program. I'm scripting with bash in Gedit, if any of that info is relevant. Thanks!
EDIT: Actually, the echo tests repeat for all three lines. So it would appear that lynx simply can't start again in the same loop?
Here is a simplified version of the script, as requested:
!#/bin/bash
while read -r line; do
echo $line
lynx $line
echo "lynx done"
done < "ref/url"
read "lynx"
$SHELL
Note that I have changed the sites the "url" file goes to:
`www.google.com
www.majorgeeks.com
http://www.sophos.com/en-us/products/free-tools/virus-removal-tool.aspx`
Lynx is not designed to use in scripts because it locks the terminal. Lynx is an interactive console browser.
If you want to access URLs in a script use wget, for example:
wget http://www.google.com/
For exit codes see: http://www.gnu.org/software/wget/manual/html_node/Exit-Status.html
to parse the html-content use:
VAR=`wget -qO- http://www.google.com/`
echo $VAR
I found a way which may fulfilled your requirement to run lynx command in loop with substitution of different url link.
Use
echo `lynx $line`
(Echo the lynx $line in single quote('))
instead of lynx $line. You may refer below:
your code
!#/bin/bash
while read -r line; do
echo $line
lynx $line
echo "lynx done"
done < "ref/url"
read "lynx"
$SHELL
try on below
!#/bin/bash
while read -r line; do
echo $line
echo `lynx $line`
echo "lynx done"
done < "ref/url"
I should have answered this question a long time ago. I got the program working, it's now on Github!
Anyway, I simply had to wrap the loop inside a function. Something like this:
progdownload () {
printlog "attmpting download from ${URL}"
if echo "${URL}" | grep -q "http://www.majorgeeks.com/" ; then
lynx -cmd_script="${WORKINGDIR}/support/mgcmd.txt" --accept-all-cookies ${URL}
else wget ${URL}
fi
}
URL="something.com"
progdownload

how to create the option for printing out statements vs executing them in a shell script

I'm looking for a way to create a switch for this bash script so that I have the option of either printing (echo) it to stdout or executing the command for debugging purposes. As you can see below, I am just doing this manually by commenting out one statement over the other to achieve this.
Code:
#!/usr/local/bin/bash
if [ $# != 2 ]; then
echo "Usage: testcurl.sh <localfile> <projectname>" >&2
echo "sample:testcurl.sh /share1/data/20110818.dat projectZ" >&2
exit 1
fi
echo /usr/bin/curl -c $PROXY --certkey $CERT --header "Test:'${AUTH}'" -T $localfile $fsProxyURL
#/usr/bin/curl -c $PROXY --certkey $CERT --header "Test:'${AUTH}'" -T $localfile $fsProxyURL
I'm simply looking for an elegant/better way to create like a switch from the command line. Print or execute.
One possible trick, though it will only work for simple commands (e.g., no pipes or redirection (a)) is to use a prefix variable like:
pax> cat qq.sh
${PAXPREFIX} ls /tmp
${PAXPREFIX} printf "%05d\n" 72
${PAXPREFIX} echo 3
What this will do is to insert you specific variable (PAXPREFIX in this case) before the commands. If the variable is empty, it will not affect the command, as follows:
pax> ./qq.sh
my_porn.gz copy_of_the_internet.gz
00072
3
However, if it's set to echo, it will prefix each line with that echo string.
pax> PAXPREFIX=echo ./qq.sh
ls /tmp
printf %05d\n 72
echo 3
(a) The reason why it will only work for simple commands can be seen if you have something like:
${PAXPREFIX} ls -1 | tr '[a-z]' '[A-Z]'
When PAXPREFIX is empty, it will simply give you the list of your filenames in uppercase. When it's set to echo, it will result in:
echo ls -1 | tr '[a-z]' '[A-Z]'
giving:
LS -1
(not quite what you'd expect).
In fact, you can see a problem with even the simple case above, where %05d\n is no longer surrounded by quotes.
If you want a more robust solution, I'd opt for:
if [[ ${PAXDEBUG:-0} -eq 1 ]] ; then
echo /usr/bin/curl -c $PROXY --certkey $CERT --header ...
else
/usr/bin/curl -c $PROXY --certkey $CERT --header ...
fi
and use PAXDEBUG=1 myscript.sh to run it in debug mode. This is similar to what you have now but with the advantage that you don't need to edit the file to switch between normal and debug modes.
For debugging output from the shell itself, you can run it with bash -x or put set -x in your script to turn it on at a specific point (and, of course, turn it off with set +x).
#!/usr/local/bin/bash
if [[ "$1" == "--dryrun" ]]; then
echoquoted() {
printf "%q " "$#"
echo
}
maybeecho=echoquoted
shift
else
maybeecho=""
fi
if [ $# != 2 ]; then
echo "Usage: testcurl.sh <localfile> <projectname>" >&2
echo "sample:testcurl.sh /share1/data/20110818.dat projectZ" >&2
exit 1
fi
$maybeecho /usr/bin/curl "$1" -o "$2"
Try something like this:
show=echo
$show /usr/bin/curl ...
Then set/unset $show accordingly.
This does not directly answer your specific question, but I guess you're trying to see what command gets executed for debugging. If you replace #!/usr/local/bin/bash with #!/usr/local/bin/bash -x bash will run and echo the commands in your script.
I do not know of a way for "print vs execute" but I know of a way for "print and execute", and it is using "bash -x". See this link for example.

Resources