How can I get a specific field from the json response?
#!/bin/bash -
status=`curl -sk -H "api-token: $TOKEN" -H "Content-Type: application/json" https://path_to/values`
The response is
{
"cancelled": false,
"percentage": 0.5,
"state": "running"
}
I want to poll the 'status' that the response percentage is 100 and the cancelled field is always true. Can this be done without another tool like jq?
EDIT
I`m trying to figure out if I can install jq on the system. Is my approach correct using jq?
while
update_status=`curl -sk -H "api-token: $TOKEN" -H "Content-Type: application/json" https://path_to/values`
cancelled=$(jq -r '.cancelled' <<< "$update_status")
percentage_complete=$(jq -r '.percentage_complete' <<< "$update_status")
state=$(jq -r '.state' <<< "$update_status")
[[ $cancelled -eq 1 || $state == 'running' ]]
do true; done
"cancelled" is a boolean and "state" is a string with the values "running" or "not_running".
How can I add a log message which shows if the update fails or not? I`m not pretty sure with the do while loop...
echo "[INFO] Update done" ## depending on the failed var?
Using jq and reading the results into an array:
readarray -t dat <<< "$(curl -sk -H "api-token: $TOKEN" -H "Content-Type: application/json" https://path_to/values | jq -r '.cancelled,.percentage,.state')"
The array can then be used in an if statement:
if [[ "${dat[0]" == "true" && "${dat[1]" == "100" ]]
then
echo "There are no issues"
else
echo "There are issues"
fi
If jq is really not an option and if the json returned is as posted, you can use awk and return back an error code:
if (curl -sk -H "api-token: $TOKEN" -H "Content-Type: application/json" https://path_to/values | awk '/(cancelled)|(percentage)|(state)/ { gsub("[\",:]","",$0);gsub(" ","",$1);map[$1]=$2 } END { if ( map["cancelled"]=="false" && map["percentage"] == 100 ) { exit 0 } else { exit 1 } }');
then
echo "There are no issues"
else
echo "There are issues"
fi
Pipe the output of the curl command into awk and where there is "cancelled", "percentage" or "state" in the line, process. Remove any "," or double quotes or ":" from the line and then remove any spaces from the first space delimited field with gsub and then add to an array called map and use the first field as the index and the second field as the value. At the end, check the indexes of the map array and exit with 0 if all are as expected, otherwise, exit with 0.
Related
So I have the following bash file and right now its looping a curl request based on the for loop. However, i am tying to find out how to continue looping until the response is empty.
Unfortunately the API that I am calling is based on pages with a maximum responses of 500 results per page. I am trying to pull the data since 2017 so its a lot of data.
I want to continue countering until the response is empty.
#!/bin/bash
# Basic while loop
counter=1
for ((i=1;i<=2;i++));
do
curl -o gettext.txt --request GET \
--url "https://api.io/v1/candidates?page=${counter}&per_page=500" \
--header 'Authorization: Basic aklsjdl;fakj;l;kasdflkaj'
((counter++))
done
echo $counter
echo All done
Anyone have an idea?
As stated in author's comment on his/her own post, the returned data is in json format. The author didn't ask how to append two json files, but it is a necessary step for him/her to accomplish his/her job. In order to append two json's, json1 and json2, maybe skipping json1 last byte } and json2 first byte {, and appending , between them would be enough. Here I am using jq to join two jsons as a more generic approach.
In the examples shown below, the nextjsonchunk file is the json file got at each request. If it has contents, it is appended to the mainjsonfile with jq. If it seems to be empty (inferred by its size) the loop breaks and the result is moved to the present folder and cleanup is made.
Using curl:
#!/usr/bin/env bash
tempfolder=/dev/shm # temporary memory parition, avaiable in ubuntu
emptyjsonize=10 # the minimum json file length, to be used as a threshold
for ((counter=1; 1; counter++))
do
curl "https://api.io/v1/candidates?page=${counter}&per_page=500" \
--header "Authorization: Basic aklsjdl;fakj;l;kasdflkaj" \
--ouput $tempfolder/nextjsonchunk
if [ $(wc -c <$tempfolder/nextjsonchunk) -le $emptyjsonize ]; then break; fi
jq -s '.[0]*.[1]' $tempfolder/mainjsonfile $tempfolder/nextjsonchunk > $folder/mainjsonfile
done
rm $tempfolder/nextjsonchunk # cleaning up
mv $tempfolder/mainjsonfile ./jsonresultfile # end result
Alternativelly, using wget:
#!/usr/bin/env bash
tempfolder=/dev/shm # temporary memory parition, avaiable in ubuntu
emptyjsonize=10 # the minimum json file length, to be used as a threshold
for ((counter=1; 1; counter++))
do
wget "https://api.io/v1/candidates?page=${counter}&per_page=500" \
--header="Authorization: Basic aklsjdl;fakj;l;kasdflkaj" \
--ouput-document $tempfolder/nextjsonchunk
if [ $(wc -c <$tempfolder/nextjsonchunk) -le $emptyjsonize ]; then break; fi
jq -s '.[0]*.[1]' $tempfolder/mainjsonfile $tempfolder/nextjsonchunk > $folder/mainjsonfile
done
rm $tempfolder/nextjsonchunk # cleaning up
mv $tempfolder/mainjsonfile ./jsonresultfile # end result
It is a good idea to take two sample json and test the merging between them, to check if it is being done properly.
It is also good to assure if the empty json file check is ok. The 10 byte was just a guess.
A tmpfs (in memory) partition, /dev/shm was used in the examples, to avoid a many writes, but its use is optional.
You can use break to end the loop at any point:
#!/bin/bash
for ((counter=1; 1; counter++)); do
curl -o gettext.txt --request GET \
--url "https://api.io/v1/candidates?page=${counter}&per_page=500" \
--header 'Authorization: Basic aklsjdl;fakj;l;kasdflkaj'
if [ ! -s gettext.txt ]; then
break;
fi
# do something with gettext.txt
# as in your question, it will be overwritten in the next iteration
done
echo "$counter"
echo "All done"
Like this?
#!/bin/bash
# Basic while loop
counter=1
while true; do
data=$(curl --request GET \
--url "https://api.io/v1/candidates?page=${counter}&per_page=500" \
--header 'Authorization: Basic aklsjdl;fakj;l;kasdflkaj')
[[ $data ]] || break
echo "$data" >> gettext.txt
((counter++))
done
echo $counter
echo All done
I am using this bash script to post a new message to my rocket.chat instance.
#!/usr/bin/env bash
function usage {
programName=$0
echo "description: use this program to post messages to Rocket.chat channel"
echo "usage: $programName [-b \"message body\"] [-u \"rocket.chat url\"]"
echo " -b The message body"
echo " -u The rocket.chat hook url to post to"
exit 1
}
while getopts ":b:u:h" opt; do
case ${opt} in
u) rocketUrl="$OPTARG"
;;
b) msgBody="$OPTARG"
;;
h) usage
;;
\?) echo "Invalid option -$OPTARG" >&2
;;
esac
done
if [[ ! "${rocketUrl}" || ! "${msgBody}" ]]; then
echo "all arguments are required"
usage
fi
read -d '' payLoad << EOF
{"text": "${msgBody}"}
EOF
echo $payLoad
statusCode=$(curl \
--write-out %{http_code} \
--silent \
--output /dev/null \
-X POST \
-H 'Content-type: application/json' \
--data "${payLoad}" ${rocketUrl})
echo ${statusCode}
Everthings works fine, so i can send a new message like this
./postToRocket.sh -b "Hello from here" -u $RocketURL
But when i try to add a message with multiple lines like this
./postToRocket.sh -b "Hello from here\nThis is a new line" -u $RocketURL
it doesn't work. I get the following output:
{"text": "Hello from heren New Line"}
200
So what do i need to change, to use break line with these bash script. Any ideas?
First, the thing making the backslash in your \n disappear was the lack of the -r argument to read. Making it read -r -d '' payLoad will fix that. However, that's not a good solution: It requires your callers to pass strings already escaped for inclusion in JSON, instead of letting them pass any/every possible string.
To make valid JSON with an arbitrary string -- including one that can contain newline literals, quotes, backslashes, or other content that has to be escaped -- use jq:
payLoad=$(jq -n --arg msgBody "$msgBody" '{"text": $msgBody}')
...and then, after doing that, amend your calling convention:
./postToRocket.sh -b $'Hello from here\nThis is a new line' -u "$RocketURL"
I believe this has already been answered in SO here
Should work by adding the $ sign and using single quotes:
./postToRocket.sh -b $'Hello from here\nThis is a new line' -u $RocketURL
i need help in understanding what is wrong with the below shell script,all its doing is executing a curl script using shell scripting and searching for a string domain_id in the output of curl command ,once it finds the search all it does it simply displays valid url
Here is the full code for shell script below
#!/bin/sh
# Black Box Tester!
url=”https://api.platform.abc.com/auth/oauth/token“
content=”$(curl --location --request POST “$url” --header 'Content-Type:
application/x-www-form-urlencoded' --header 'Authorization: Basic
V0pSWURISA==' --data-raw 'grant_type=password&username=event-
player1#abc.com&password=********' | grep domain_id”
if [ ! -z $content ] && [ $content -eq domain_id ]
then
echo “valid url”
else
echo “invalid url”
fi
Below is the error i get in console after i try executing the script
WS-126691A:loginimagedocker carolyn$ ./login.sh
./login.sh: line 4: unexpected EOF while looking for matching `)'
./login.sh: line 11: syntax error: unexpected end of file
You are trying to pass parameter in multiple line without mentioning end of line.
use \ to pass multi line argument
#!/bin/sh
# Black Box Tester!
url=”http://api.platform.abc.com/auth/oauth/token“
content=”$(curl --location --request POST “$url” --header 'Content-Type: \
application/x-www-form-urlencoded' --header 'Authorization: Basic \
V0pSWURISA==' --data-raw 'grant_type=password&username=event- \
player1#abc.com&password=********' | grep domain_id” )
if [ ! -z $content ] && [ $content -eq domain_id ]
then
echo “valid url”
else
echo “invalid url”
fi
I am trying below code to get response body and status:
read -ra result <<< $(curl -i --insecure \
-H "Accept: application/json" \
-H "Content-Type:application/json" \
-X POST --data "$configData" $openingNode"/voice/v1/updateWithPh")
status=${result[1]}
response=${result[#]}
echo $status
Problem here is -
I get both status code and response Body correctly.
But when I create a bash function and send it as an argument, the response body changes to "HTTP/1.1" in the function as shown below.
echo $(validateUpdate $configData $response)
Code for the function -
function validateUpdate(){
echo $1
echo $2
}
$2 prints as "HTTP/1.1"
What is the reason? How to rectify this issue?
You need to enclose your variables in double quotes to prevent bash splitting it into separate tokens.
Try
echo $(validateUpdate "$configData" "$response")
or even better (echo is useless as #tripleee points out; furthermore curly braces improves readability):
validateUpdate "${configData}" "${response}"
use same thing inside of your function
echo "$2"
I have a script that runs curl. I want to be able to optionally add a -H parameter, if a string isn't empty. What's complex is the levels of quoting and spaces.
caption="Test Caption"
if [ "${caption}" != "" ]; then
CAPT=-H "X-Caption: ${caption}"
fi
curl -A "$UA" -H "Content-MD5: $MD5" -H "X-SessionID: $SID" -H "X-Version: 1" $CAPT http://upload.example.com/$FN
The idea is that the CAPT variable is either empty, or contains the desired -H header in the same form as the others, e.g., -H "X-Caption: Test Caption"
The problem is when run, it interprets the assignment as a command to be executed:
$bash -x -v test.sh
+ '[' 'Test caption' '!=' '' ']'
+ CAPT=-H
+ 'X-Caption: Test caption'
./test.sh: line 273: X-Caption: Test caption: command not found
I've tried resetting IFS before the code, but it didn't make a difference.
The key to making this work is to use an array.
caption="Test Caption"
if [[ $caption ]]; then
CAPT=(-H "X-Caption: $caption")
fi
curl -A "$UA" -H "Content-MD5: $MD5" -H "X-SessionID: $SID" -H "X-Version: 1" "${CAPT[#]}" "http://upload.example.com/$FN"
If you only need to know whether or not the caption is there, you can interpolate it when it needs to be there.
caption="Test Caption"
NOCAPT="yeah, sort of, that would be nice"
if [ "${caption}" != "" ]; then
unset NOCAPT
fi
curl ${NOCAPT--H "X-Caption: ${caption}"} -A "$UA" ...
To recap, the syntax ${var-value} produces value if var is unset.
I finally did get it to work. Part of the problem is specific to curl, in that when using the -H option to set custom headers, it seems to work best when everything after the -H (that is, both the custom header name and value) are protected by single quotes. Then, I needed to pass the constructed string through eval to get it to work.
To make this easier to read, I store a single quote in a variable named TICK.
Example:
TICK=\'
#
HDRS=""
HDRS+=" -H ${TICK}Content-MD5: ${MD5}${TICK}"
HDRS+=" -H ${TICK}X-SessionID: ${SID}${TICK}"
HDRS+=" -H ${TICK}X-Version: 1.1.1${TICK}"
HDRS+=" -H ${TICK}X-ResponseType: REST${TICK}"
HDRS+=" -H ${TICK}X-ID: ${ID}${TICK}"
if [ "${IPTC[1]}" != "" ]; then
HDRS+=" -H ${TICK}X-Caption: ${IPTC[1]}${TICK}"
fi
if [ "${IPTC[2]}" != "" ]; then
HDRS+=" -H ${TICK}X-Keywords: ${IPTC[2]}${TICK}"
fi
#
# Set curl flags
#
CURLFLAGS=""
CURLFLAGS+=" --cookie $COOKIES --cookie-jar $COOKIES"
CURLFLAGS+=" -A \"$UA\" -T ${TICK}${the_file}${TICK} "
eval curl $CURLFLAGS $HDRS -o $OUT http://upload.example.com/$FN