How to hande a curl response fail in Bash script - bash

Question

How can I handle request fails in this example of bash curl requests. I.e. if all servers are responde with JSON all is okay and I have JSON file at end of a cycle. But if one of this servers not responde with JSON or not responde at all I do have nothing in "/data.json" file, even all other servers are working perfectly. How can I catch a server fail and skip it?
#!/bin/bash
CONFIG=config.json
jsondata=data.json
i=1
echo "{ \"success\": \"OK\", \"servers\": {" > $jsondata
jq -r '.servers|keys[]' $CONFIG | while read key ; do
if [ "$i" -ne "1" ]; then
echo "," >> $jsondata
fi
echo "\"server$i\": {" >> $jsondata
RESPONSE=$(curl -s $HTTP://$IP:$PORT/api)
DATA1=$(echo $RESPONSE | jq '.data1')
DATA2=$(echo $RESPONSE | jq '.data2')
echo " \"data1\": $DATA1", >> $jsondata
echo " \"data2\": $DATA2", >> $jsondata
echo "}" >> $jsondata
((i++))
done
echo "}," >> $jsondata

Answer 1

First, let's assume we have a function to make a single API call.
do_api () {
key=$1
curl $HTTP://$IP:$PORT/api # Presumably, key is needed here somewhere
}
I'll also assume that the intended output is a JSON object that has at least two fields, data1 and data2. Now, we can write a simple pipeline with three stages:
Read keys from $CONFIG
Make an API call for each key
Generate the desired output from the combined output of all the API calls.
It's not too complicated:
jq -r '.servers | keys[]' | # stage 1
while read key; do do_api "$key"; done | # stage 2
jq -s 'to_entries |
map({key: "server(\.key+1)",
value: {data1: .value.data1,
data2: .value.data2}
}) |
{success: "OK", servers: from_entries}' # stage 3
do_api should output nothing for a particular key of curl fails, but you can modify it to produce some sort of default data if you wish:
do_api () {
key=$1
curl --fail ... || jq -n '{data1: null, data2: null}'
}
jq -s 'to_entries' takes an input like
{ "data1": ..., "data2": ... }
{ "data1": ..., "data2": ... }
(which is what we expect from curl), and produces as output
[ { "key": 0, "value": { "data1": ..., "data2": ... } },
{ "key": 1, "value": { "data1": ..., "data2": ... } }
]
The map(...) filter takes the preceding array and produces the keys and values we want to add to the servers object in the final result, which is created by a call to from_entries.
Here is a full example. tmp.json contains the simulated output of do_api, complete with an extra field data3 that will be filtered from the final output.
$ cat tmp.json
{
"data1": "foo",
"data2": "bar",
"data3": "baz"
}
{
"data1": "hello",
"data2": "world",
"data3": "bye"
}
$ jq -s 'to_entries | map({key: "server\(.key+1)", value: {data1: .value.data1, data2: .value.data2}}) | {success: "OK", servers: from_entries}' tmp.json
{
"success": "OK",
"servers": {
"server1": {
"data1": "foo",
"data2": "bar"
},
"server2": {
"data1": "hello",
"data2": "world"
}
}
}