How to fetch next URL from JSON using Bash - bash

Question

I have created a simple script to store response from a third party API
The request is like this..
https://externalservice.io/orders?key=password&records=50&offset=0
The response is as follows:
{
"results": [
{
"engagement": {
"id": 29090716,
"portalId": 62515,
"active": true,
"createdAt": 1444223400781,
"lastUpdated": 1444223400781,
"createdBy": 215482,
"modifiedBy": 215482,
"ownerId": 70,
"type": "NOTE",
"timestamp": 1444223400781
},
},
],
"hasMore": true,
"offset": 4623406
}
If there is a hasMore attribute, I need to read the offset value to get the next set of records.
Right now I've created a script that simply loops over the estimated number of records (I believed there is) and thought incrementing the offset would work but this is not the case as the offset is not incremental.
#!/bin/bash
for i in {1..100}; do
curl -s "https://externalservice.io/orders?key=password&records=50&offset=$i" >>outfile.txt 2>&1
done
Can someone explain how I can read continue the script reading the offset value until hasMore=false?

Answer 1

You can read a value from a json using the jq utility:
$ jq -r ".hasMore" outfile
true
Here is what you could use:
more="true"
offset=0
while [ $more = "true" ]; do
echo $offset
response=$(curl -s "https://example.com/orders?offset=$offset")
more=$(echo $response | tr '\r\n' ' ' | jq -r ".hasMore")
offset=$(echo $response | tr '\r\n' ' ' | jq -r ".offset")
done

Answer 2

You can use jq utility to extract specific attribute from you json response:
$ jq -r ".hasMore" outfile
true
jq expects perfectly valid json input, otherwise it will print error.
Most common mistake is to echo json stored in variable. A mistake, because echo will interpret all escaped newlines in your json and will send unescaped newlines within values causing jq to throw parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line message.
To avoid modification to json (and avoiding an error) we need to use printf instead of echo.
Here is the solution without breaking json content:
more="true"
offset=0
while [ $more = "true" ]; do
echo $offset
response=$(curl -s "https://example.com/orders?offset=$offset")
more=$(printf "%s\n" "$response" | jq -r ".hasMore")
offset=$(printf "%s\n" "$response" | jq -r ".offset")
done