I need 4 things from curl in a BASH script:
I need to capture brief humanly readable error message from curl into a bash variable.
I need to be able to check that the command completed
successfully or not.
I need the command to run
I don't want anything printed to console unless I echo it.
m=$(curl -u "$user":AP"$pass" -T "$pathA" "$url")
if [ $? -ne 0 ] ; then
echo "Error: ""$m"
fi
The problem is curl puts gibberish into $m. It just dumps the error to console instead of m. I don't want anything printed to console unless I echo it. And I only want to capture error descriptions. I tried many variations, nothing seemed to work for this use-case... at least nothing suggested here on Stack.
curl sends errors to STDERR and will not get captured by $m. The output of curl is sent to STDERR (that gibberish you mentioned).
One solution is to redirect STDERR to STDOUT by adding 2>&1 to your curl invocation in your script:
m=$(curl -u "$user":AP"$pass" -T "$pathA" "$url" 2>&1)
if [ $? -ne 0 ] ; then
echo "Error: ""$m"
fi
You could also use the --fail, --silent and --show-errors flags of curl like in this example if all you care about are the errors:
Making curl send errors to stderr and everything else to stdout
and some information about capturing STDERR to $variable in bash scripts:
Bash how do you capture stderr to a variable?
Similar to one suggested; I m just using the output stderror as $? is always 0.
OUTPUT=$(curl -X POST -H 'Content-type: application/json' --data "{\"text\":\"$MSG \"}" $SLACK_URL 2>1)
echo $OUTPUT
if [[ $OUTPUT == "ok" ]]; then
echo "$FILE_NAME Msg successfully sent"
elif [[ $OUTPUT == "invalid_token" ]]; then
echo "$FILE_NAME Slack url incorrect"
else
echo "$FILE_NAME Some issue Sending msg"
fi
Related
I'm creating a bash script to check the HTTP headers on remote hosts, I'm doing this via cURL and have noted that appending http://{host} will only work for services running on tcp\80, and not tcp\443. For example for HTTPS services, you require curl -I -k {host}, as opposed to HTTP services which only required curl -I {host}. This is my script:
for host in $(cat file.txt); do
echo " "
echo "Current host: "${host}
curl -I -k https://${host}
echo " "
echo "=============================================="
done
Now what I'm wanting is some condition operator to check that if the output is "Could not resolve host" then the script should run "curl -I http://{host}" on those hosts which the stdout contained the str value "Could not resolve host".
How can I achieve this in bash?
stdout will not contain Could not resolve host though, that's output to stderr. While you could capture stderr and then do string matching, there is a much, much simpler solution: exit code.
You can see here that curl will always exit with code 6 when it fails to resolve host. Thus, simply testing the exit code is sufficient:
curl -i -k http://nowaythisthingexists.test
if [[ $? -eq 6 ]]
then
echo "oopsie, couldn't resolve host!"
fi
Alternately, if you really want to do it by matching strings, make sure to redirect stderr to stdout (and possibly also kill stdout so it doesn't interfere):
output=$(curl -i -k http://nowaythisthingexists.test 2>&1 >/dev/null)
if [[ "$output" = *"Could not resolve host"* ]]
then
echo "oopsie, couldn't resolve host!"
fi
Obviously, you are not getting the output of your request this way, so you'd need to redirect it somewhere more useful than /dev/null — a file, or a Unix pipe. Now it's getting more complicated than it needs to be.
If another script or process, such as the sequential tasks in a Gitlab .gitlab-ci.yml file for instance, depend on a Bash script exiting without an error to continue, or fail to exit further execution; should the Bash script do any thing else than exit 0 for success or exit 1 for catch all errors?
Example code below tests if a site was successfully deployed.
#!/bin/bash
flag="false"
for i in {1..10}
do
response=$(curl -Is http://mysite/ | head -n 1 | tr -d '\r')
echo "$response"
if [ "$response" = "HTTP/1.1 200 OK" ]; then
echo "SITE UP"
flag="true"
exit 0
fi
sleep 1s
done
if [ "$flag" = "false" ]; then
echo "SITE DOWN"
exit 1
fi
Everything works 100% as expected, and the script is executed as ./test-up.sh > /dev/null to suppress the debugging echos used (as well as text returned by the curl command).
However, if the script fails nothing is printed on the command line with regard to errors. The command line stays blank. The same happens if exit 0 was encountered.
Is the above intended?
Echo to stderr
echo "SITE DOWN" >&2
I have written a bash script that does not show any errors. However I would like to add conditional block list if success then show email success else show error message in email as shown in the code below.
scp -i id_rsa -r testuser#1.1.1.:/data1/scp ~/data/scp/files
success >> ~/data/scp/files/log.txt 2>&1
if success
then
| mail -s "Download
Successfull" abc#test.com <<< "Files Successfully Downloaded"
else
| mail -s "Error: Download Failed" abc#test.com <<< "Error File download
Failed!"
fi
Here is the working script without If else block
#!/module/for/bash
scp -i id_rsa -r test#1.1.1.1:/data1/scp ~/data/scp/files
echo success! >> ~/data/scp/files/log.txt 2>&1 | mail -s "Download
Successfull" abc#test.com <<< "Files Successfully
Downloaded" | mail -s "Error: Download Failed" abc#test.com <<<
"Error:file download Failed!"
The scp man page states: The scp utility exits 0 on success, and >0 if an error occurs.
So you can do something like:
if scp -i id_rsa -r testuser#1.1.1.:/data1/scp ~/data/scp/files
then
mail -s "Download Successful" abc#test.com <<<"Files Downloaded"
else
mail -s "Download Error" abc#test.com <<<"Download error"
fi
or
scp -i id_rsa -r testuser#1.1.1.:/data1/scp ~/data/scp/files
if [[ $? -eq 0 ]]
then
mail -s "Download Successful" abc#test.com <<<"Files Downloaded"
else
mail -s "Download Error" abc#test.com <<<"Download error"
fi
finally you may also want to look at something like storing the scp output. Use -q to have scp not print out progress meters and what not:
MYOUT=$(scp -q -i id_rsa -r testuser#1.1.1.:/data1/scp ~/data/scp/files 2>&1)
if [[ $? -eq 0 ]]
then
mail -s "Download Successful" abc#test.com <<<"$MYOUT"
else
mail -s "Download Error" abc#test.com <<<"$MYOUT"
fi
This link should clear the air. Hope it helped!
#Korthrun has already posted several ways to accomplish what I think you're trying to do; I'll take a look at what's going wrong in your current script. You seem to be confused about a couple of basic elements of shell scripting: pipes (|) and testing for command success/failure.
Pipes are used to pass the output of one command into the input of another (and possibly then chain the output of the second command into the input of a third command, etc). But when you use a pipe string like this:
echo success! >> ~/data/scp/files/log.txt 2>&1 |
mail -s "Download Successfull" abc#test.com <<< "Files Successfully Downloaded" |
mail -s "Error: Download Failed" abc#test.com <<< "Error:file download Failed!"
the pipes aren't actually doing anything. The first pipe tries to take the output of echo and feed it to the input of mail, but the >> in the echo command sends its output to a file instead, so no actual data is sent to the mail command. Which is probably good, because the <<< on the mail command tells it to ignore the regular input (from the pipe) and feed a string as input instead! Similarly, the second pipe tries to feed the output from the first mail command (there isn't any) to the last mail command, but again it's ignored due to another <<< input string. The correct way to do this is simply to remove the pipes, and run each command separately:
echo success! >> ~/data/scp/files/log.txt 2>&1
mail -s "Download Successfull" abc#test.com <<< "Files Successfully Downloaded"
mail -s "Error: Download Failed" abc#test.com <<< "Error:file download Failed!"
This is also causing a problem in the other version of your script, where you use:
if success
then
| mail -s "Download Successfull" abc#test.com <<< "Files Successfully Downloaded"
Here, there's no command before the pipe, so it doesn't make any sense at all (and you get a shell syntax error). Just remove the pipe.
Now, about success/failure testing: you seem to be using success as a command, but it isn't one. You can either use the command you want to check the success of directly as the if conditional:
if scp ...; then
echo "It worked!"
else
echo "It failed!"
fi
or use the shell variable $? which returns the exit status of the last command (success=0, failure=anything else):
scp ...
if [ $? -eq 0 ]; then
...
There's a subtlety here that's easy to miss: the thing after if is a command, but in the second form it appears to be a logical expression (testing whether $? is equal to 0). The secret is that [ is actually a command that evaluates logical expressions and then exits with success or failure depending on whether the expression was true or false. Do not mistake [ ] for some sort of parentheses or other grouping operator, that's not what's going on here!
BTW, the [[ ]] form that Korthrun used is very similar to [ ], but isn't supported by more basic shells. It does avoid some nasty syntax oddities with [ ], though, so if you're using bash it's a good way to go.
Also, note that $? gives the status of the last command executed, so it gets reset by every single command that executes. For example, this won't work:
scp ...
echo "scp's exit status was $?"
if [ $? -eq 0 ]; then # Don't do this!!!!
...because the if is then looking at the exit status of the echo command, not scp! If you need to do something like this, store the status in a variable:
scp ...
scpstatus=$?
echo "scp's exit status was $scpstatus"
if [ $scpstatus -eq 0 ]; then
i am writing a simple bash script to "curl get" some values. Sometimes the code works and sometimes it fails, and says "empty reply from server".
How to set up a check for this in bash so that if the curl fails once it tries again until it gets the values?
while ! curl ... # add your specific curl statement here
do
{ echo "Exit status of curl: $?"
echo "Retrying ..."
} 1>&2
# you may add a "sleep 10" or similar here to retry only after ten seconds
done
In case you want the output of that curl in a variable, feel free to capture it:
output=$(
while ! curl ... # add your specific curl statement here
do
{ echo "Exit status of curl: $?"
echo "Retrying ..."
} 1>&2
# you may add a "sleep 10" or similar here to retry only after ten seconds
done
)
The messages about the retry are printed to stderr, so they won't mess up the curl output.
People are overcomplicating this:
until contents=$(curl "$url")
do
sleep 10
done
For me sometimes it happens when curl timed out and there is no information about that. Try curl with --connect-timeout 600 (in seconds) like:
curl --connect-timeout 600 "https://api.morph.io/some_stuff/data.json"
Maybe this helps you.
if you wanted to try the command until it succeeded, you could say:
command_to_execute; until (( $? == 0 )); do command_to_execute; done
Good day. I have a series of commands that I wanted to execute via a function so that I could get the exit code and perform console output accordingly. With that being said, I have two issues here:
1) I can't seem to direct stderr to /dev/null.
2) The first echo line is not displayed until the $1 is executed. It's not really noticeable until I run commands that take a while to process, such as searching the hard drive for a file. Additionally, it's obvious that this is the case, because the output looks like:
sh-3.2# ./runScript.sh
sh-3.2# com.apple.auditd: Already loaded
sh-3.2# Attempting... Enable Security Auditing ...Success
In other words, the stderr was displayed before "Attempting... $2"
Here is the function I am trying to use:
#!/bin/bash
function saveChange {
echo -ne "Attempting... $2"
exec $1
if [ "$?" -ne 0 ]; then
echo -ne " ...Failure\n\r"
else
echo -ne " ...Success\n\r"
fi
}
saveChange "$(launchctl load -w /System/Library/LaunchDaemons/com.apple.auditd.plist)" "Enable Security Auditing"
Any help or advice is appreciated.
this is how you redirect stderr to /dev/null
command 2> /dev/null
e.g.
ls -l 2> /dev/null
Your second part (i.e. ordering of echo) -- It may be because of this you have while invoking the script. $(launchctl load -w /System/Library/LaunchDaemons/com.apple.auditd.plist)
The first echo line is displayed later because it is being execute second. $(...) will execute the code. Try the following:
#!/bin/bash
function saveChange {
echo -ne "Attempting... $2"
err=$($1 2>&1)
if [ -z "$err" ]; then
echo -ne " ...Success\n\r"
else
echo -ne " ...Failured\n\r"
exit 1
fi
}
saveChange "launchctl load -w /System/Library/LaunchDaemons/com.apple.auditd.plist" "Enable Security Auditing"
EDIT: Noticed that launchctl does not actually set $? on failure so capturing the STDERR to detect the error instead.