How do I check the HTTP status code and also parse the payload - bash

Imagine I have the following code in a bash script:
curl -s https://cat-fact.herokuapp.com/facts/random?animal=cat | jq .
Notice that I wish to display the payload of the response by passing it to jq.
Now suppose sometimes those curls sometimes return a 404, in such cases my script currently still succeeds so what I need to do is check the return code and exit 1 as appropriate (e.g. for a 404 or 503). I've googled around and found https://superuser.com/a/442395/722402 which suggests --write-out "%{http_code}" might be useful however that simply prints the http_code after printing the payload:
curl -s --write-out "%{http_code}" https://cat-fact.herokuapp.com/facts/random?animal=cat | jq .
$ curl -s --write-out "%{http_code}" https://cat-fact.herokuapp.com/facts/random?animal=cat | jq .
{
"_id": "591f98783b90f7150a19c1ab",
"__v": 0,
"text": "Cats and kittens should be acquired in pairs whenever possible as cat families interact best in pairs.",
"updatedAt": "2018-12-05T05:56:30.384Z",
"createdAt": "2018-01-04T01:10:54.673Z",
"deleted": false,
"type": "cat",
"source": "api",
"used": false
}
200
What I actually want to is still output the payload, but still be able to check the http status code and fail accordingly. I'm a bash noob so am having trouble figuring this out. Help please?
I'm using a Mac by the way, not sure if that matters or not (I'm vaguely aware that some commands work differently on Mac)
Update, I've pieced this together which sorta works. I think. Its not very elegant though, I'm looking for something better.
func() {
echo "${#:1:$#-1}";
}
response=$(curl -s --write-out "%{http_code}" https://cat-fact.herokuapp.com/facts/random?animal=cat | jq .)
http_code=$(echo $response | awk '{print $NF}')
func $response | jq .
if [ $http_code == "503" ]; then
echo "Exiting with error due to 503"
exit 1
elif [ $http_code == "404" ]; then
echo "Exiting with error due to 404"
exit 1
fi

What about this. It uses a temporary file. Seems me a bit complicated but it separates your content.
# copy/paste doesn't work with the following
curl -s --write-out \
"%{http_code}" https://cat-fact.herokuapp.com/facts/random?animal=cat | \
tee test.txt | \ # split output to file and stdout
sed -e 's-.*\}--' | \ # remove everything before last '}'
grep 200 && \ # try to find string 200, only in success next step is done
echo && \ # a new-line to juice-up the output
cat test.txt | \ #
sed 's-}.*$-}-' | \ # removes the last line with status
jq # formmat json
Here a copy/paste version
curl -s --write-out "%{http_code}" https://cat-fact.herokuapp.com/facts/random?animal=cat | tee test.txt | sed -e 's-.*\}--' | grep 200 && echo && cat test.txt | sed 's-}.*$-}-' | jq

This is my attempt. Hope it works for you too.
#!/bin/bash
result=$( curl -i -s 'https://cat-fact.herokuapp.com/facts/random?animal=cat' )
status=$( echo "$result" | grep -E '^HTTPS?/[1-9][.][1-9] [1-9][0-9][0-9]' | grep -o ' [1-9][0-9][0-9] ')
payload=$( echo "$result" | sed -n '/^\s*$/,//{/^\s*$/ !p}' )
echo "STATUS : $status"
echo "PAYLOAD : $payload"
Output
STATUS : 200
PAYLOAD : {"_id":"591f98803b90f7150a19c23f","__v":0,"text":"Cats can't taste sweets.","updatedAt":"2018-12-05T05:56:30.384Z","createdAt":"2018-01-04T01:10:54.673Z","deleted":false,"type":"cat","source":"api","used":false}
AWK version
payload=$( echo "$result" | awk '{ if( $0 ~ /^\s*$/ ){ c_p = 1 ; next; } if (c_p) { print $0} }' )
Regards!
EDIT : I have simplified this even more by using the -i flag
EDIT II : Removed empty line from payload
EDIT III : Included an awk method to extract the payload in case sed is problematic

Borrowing from here you can do:
#!/bin/bash
result=$(curl -s --write-out "%{http_code}" https://cat-fact.herokuapp.com/facts/random?animal=cat)
http_code="${result: -3}"
response="${result:0:${#result}-3}"
echo "Response code: " $http_code
echo "Response: "
echo $response | jq
Where
${result: -3} is the 3rd index starting from the right of the string till the end. This ${result: -3:3} also would work: Index -3 with length 3
${#result} gives us the length of the string
${result:0:${#result}-3} from the beginning of result to the end minus 3 from the http_status code
The site cat-fact.herokuapp.com isn't working now so I had to test it with another site

Related

Commit list from GIT is always processed completely per loop instead of individual elements

I have a problem with my bash script. Namely, I am doing procedures against an API via CURL for each JSON file pushed to GIT. In principle the procedure works, however I have the problem that for processing in each loop the complete commit list is controlled against the API.
So if I push two JSON files to GIT, then each JSON file is executed twice against the API instead of each JSON file only once.
Example:
Git Push:
file1.json
file2.json
file3.json
file4.json
file5.json
Execution:
file1.json file1.json file1.json file1.json file1.json
file2.json file2.json file2.json file2.json file2.json
file3.json file3.json file3.json file3.json file3.json
file4.json file4.json file4.json file4.json file4.json
file5.json file5.json file5.json file5.json file5.json
Expectation would be each file only once.
I tried to solve the issue using arrays, but apparently it doesn't work as thought.
Here is the actual function from the code:
#!/bin/bash
# Create an empty array to store processed files
processed_files=()
#Login
endpoint=xxx
username=xxx
password=xxx
# Get list of files in commit
files=`git diff --name-only HEAD HEAD~1`
# Test each file that is a json
for file in $files
do
if [[ $file == *.json ]]
then
# Check if the file has already been processed
if [[ ! " ${processed_files[#]} " =~ " ${file} " ]]
then
# Add the file to the array
processed_files+=("$file")
echo "Jobexecution"
curl -k -s -H "Authorization: xxx" -X POST -F "definitionsFile=#../$file" -F "$endpoint/deploy"
submit=$(curl -k -s -H "Authorization: xxx" -X POST -F "jobDefinitionsFile=#../$file" -F "$endpoint/run")
runid=$(echo ${submit##*runId\" : \"} | cut -d '"' -f 1)
# Check job status
jobstatus=$(curl -k -s -H "Authorization: xxx" "$endpoint/run/status/$runid")
status=$(echo ${jobstatus##*status\" : \"} | cut -d '"' -f 1)
# Wait till jobs ended
echo "Wait till jobs ended"
until [[ $status == Ended* ]]; do
sleep 10
tmp=$(curl -k -s -H "Authorization: xxx" "$endpoint/run/status/$runid")
echo $tmp | grep 'Not OK' >/dev/null && exit 2
tmp2=$(echo ${tmp##*$'\"type\" : \"Folder\",\\n'})
status=$(echo ${tmp2##*\"status\" : \"} | cut -d '"' -f 1)
done
else
echo "Job was already executed. Ill skip this one."
fi
fi
done
# Logout
curl -k -s -H "Authorization: xxx" -X POST "$endpoint/session/logout"
# Exit
if [[ $status == *Not* ]]; then
echo 'Job failed!'
exit 1
else
echo 'Success!'
exit 0
fi
As already mentioned, I tried to solve the issue using arrays, but apparently it doesn't work as thought.
I solve the problem. The issue was, that the Jenkins Pipeline send always the whole commit list for each JSON-File.
To solve the problem, I execute the Bash-Script in the Jenkinspipeline with an argument, wich is the actually JSON-File in the loop.

Check return code in bash while capturing text

When running an ldapsearch we get a return code indicating success or failure. This way we can use an if statement to check success.
On failure when using debug it prints if the cert validation failed. How can I capture the output of the command while checking the sucess or failure of ldapsearch?
ldapIP=`nslookup corpadssl.glb.intel.com | awk '/^Address: / { print $2 }' | cut -d' ' -f2`
server=`nslookup $ldapIP | awk -F"= " '/name/{print $2}'`
ldap='ldapsearch -x -d8 -H "ldaps://$ldapIP" -b "dc=corp,dc=xxxxx,dc=com" -D "name#am.corp.com" -w "366676" (mailNickname=sdent)"'
while true; do
if [[ $ldap ]] <-- capture text output here ??
then
:
else
echo $server $ldapIP `date` >> fail.txt
fi
sleep 5
done
As #codeforester suggested, you can use $? to check the return code of the last command.
ldapIP=`nslookup corpadssl.glb.intel.com | awk '/^Address: / { print $2 }' | cut -d' ' -f2`
server=`nslookup $ldapIP | awk -F"= " '/name/{print $2}'`
while true; do
captured=$(ldapsearch -x -d8 -H "ldaps://$ldapIP" -b "dc=corp,dc=xxxxx,dc=com" -D "name#am.corp.com" -w "366676" "(mailNickname=sdent)")
if [ $? -eq 0 ]
then
echo "${captured}"
else
echo "$server $ldapIP `date`" >> fail.txt
fi
sleep 5
done
EDIT: at #rici suggestion (and because I forgot to do it)... ldap needs to be run before the if.
EDIT2: at #Charles Duffy suggestion (we will get there), we don't need to store the command in a variable.

curl in bash script vs curl one liner

This code ouputs a http status of 000 - which seems to indicate something didn't connect properly but when I do this curl outside of the bash script it works fine and produces a 200 so something with this code is off... any guidance?
#!/bin/bash
URLs=$(< test.txt | grep Url | awk -F\ ' { print $2 } ')
# printf "Preparing to check $URLs \n"
for line in $URLs
do curl -L -s -w "%{http_code} %{url_effective}\\n" $line
done
http://beerpla.net/2010/06/10/how-to-display-just-the-http-response-code-in-cli-curl/
your script works on my vt.
I added in a couple of debugging lines, this may help you to see where any metacharacters are getting in, as I would have to agree with the posted coments.
I've output lines in the for to a file which is then printed out with od.
I have amended the curl line to grab the last line, just to get the response code.
#!/bin/bash
echo -n > $HOME/Desktop/urltstfile # truncate urltstfile
URLs=$(cat testurl.txt | grep Url | awk -F\ ' { print $2 } ')
# printf "Preparing to check $URLs \n"
for line in $URLs
do echo $line >> $HOME/Desktop/urltstfile;
echo line:$line:
curl -IL -s -w "%{http_code}\n" $line | tail -1
done
od -c $HOME/Desktop/urltstfile
#do curl -L -s -w "%{http_code} %{url_effective}\\n" "$line\n"

Shell integer expression expected?

response=$(curl -sL -w \\n%{http_code} "http://<ip_addr>/api/1/app" -X DELETE)
echo response
if [ "$response" -eq 200 ]
then
echo "Got 200 OK"
else
echo "not getting the result"
fi
What i'm trying to do is to get the http response code.
I'm positive that the response should be 200 OK
When I run the script I'm getting
{
"result":true
}
200
tst.sh: line 302: [: {
200: integer expression expected
I even don't want to display
{
"result":true
}
I just want to print 200 and compare 200.
Just with curl command:
curl -sL -w '%{http_code}' "http://<ip_addr>/api/1/app" -X DELETE -o /dev/null
Get the last line of the output.
response=$(curl -sL -w \\n%{http_code} "http://<ip_addr>/api/1/app" -X DELETE | tail -1)
Besides tail -n you can also use:
awk 'END { print }'
sed -n '$p'
Another way if you're using bash is to remove everything before the last line:
shopt -s extglob
response=${response##*[[:space:]]}
In your above, you are missing a $ in your first echo, but no matter. You can strip the {"result":true} from the $response string
before the if statement:
response=${response#*\}\ }

BASH shell script echo to output on same line

I have a simple BASH shell script which checks the HTTP response code of a curl command.
The logic is fine, but I am stuck on "simply" printing out the "output".
I am using GNU bash, version 3.2.25(1)-release (x86_64-redhat-linux-gnu)
I would like to output the URL with a tab - then the 404|200|501|502 response. For example:
http://www.google.co.uk<tab>200
I am also getting a strange error where the "http" part of a URL is being overwritten with the 200|404|501|502. Is there a basic BASH shell scripting (feature) which I am not using?
thanks
Miles.
#!/bin/bash
NAMES=`cat $1`
for i in $NAMES
do
URL=$i
statuscode=`curl -s -I -L $i |grep 'HTTP' | awk '{print $2}'`
case $statuscode in
200)
echo -ne $URL\t$statuscode;;
301)
echo -ne "\t $statuscode";;
302)
echo -ne "\t $statuscode";;
404)
echo -ne "\t $statuscode";;
esac
done
From this answer you can use the code
response=$(curl --write-out %{http_code} --silent --output /dev/null servername)
Substituted into your loop this would be
#!/bin/bash
NAMES=`cat $1`
for i in $NAMES
do
URL=$i
statuscode=$(curl --write-out %{http_code} --silent --output /dev/null $i)
case $statuscode in
200)
echo -e "$URL\t$statuscode" ;;
301)
echo -e "$URL\t$statuscode" ;;
302)
echo -e "$URL\t$statuscode" ;;
404)
echo -e "$URL\t$statuscode" ;;
* )
;;
esac
done
I've cleaned up the echo statements too so for each URL there is a new line.
try
200)
echo -ne "$URL\t$statuscode" ;;
I'm taking a stab here, but I think what's confusing you is the fact that curl is sometimes returning more than one header info (hence more than one status code) when the initial request gets redirected.
For example:
[me#hoe]$ curl -sIL www.google.com | awk '/HTTP/{print $2}'
302
200
When you're printing that in a loop, it would appear that the second status code has become part of the next URL.
If this is indeed your problem, then there are several ways to solve this depending on what you're trying to achieve.
If you don't want to follow redirections, simple leave out the -L option in curl
statuscode=$(curl -sI $i | awk '/HTTP/{print $2}')
To take only the last status code, simply pipe the whole command to tail -n1 to take only the last one.
statuscode=$(curl -sI $i | awk '/HTTP/{print $2}' | tail -n1)
To show all codes in the order, replace all linebreaks with spaces
statuscode=$(curl -sI $i | awk '/HTTP/{print $2}' | tr "\n" " ")
For example, using the 3rd scenario:
[me#home]$ cat script.sh
#!/bin/bash
for URL in www.stackoverflow.com stackoverflow.com stackoverflow.com/xxx
do
statuscode=$(curl -siL $i | awk '/^HTTP/{print $2}' | tr '\n' ' ')
echo -e "${URL}\t${statuscode}"
done
[me#home]$ ./script.sh
www.stackoverflow.com 301 200
stackoverflow.com 200
stackoverflow.com/xxx 404

Resources