Corrupted zipfile after curl request - bash

Question

I am using the Qualtrics API to retrieve survey data at scheduled intervals. Below is my shell script (bash) get_responses.sh which posts the export, measures the download completion rate, gets the export, and stores/unzips the file.
STARTDATE=$(date -v-7d "+%Y-%m-%d")
STARTDATESTRING=$STARTDATE"T00:00:00-07:00:00"
ENDDATE=$(date "+%Y-%m-%d")
ENDDATESTRING=$ENDDATE"T00:00:00-07:00:00"
result=$(curl -X POST -H 'X-API-TOKEN: MYAPITOKEN' -H 'Content-Type: application/json' -d '{
"surveyId": "SV_000000000000",
"startDate": "startDate": "'"$STARTDATESTRING"'",
"endDate": "endDate": "'"$ENDDATESTRING"'",
"format": "csv",
"useLocalTime": true,
"useLabels": true
}' "https://myorg.qualtrics.com/API/v3/responseexports")
es_id=$(echo "$result" | /usr/local/Cellar/jq/1.5_2/bin/jq --raw-output '.result.id')
curl -H "X-API-TOKEN: MYAPITOKEN" "https://myorg.qualtrics.com/API/v3/responseexports/${es_id}"
curl -X GET -H "Content-Type: application/json" -H "X-API-TOKEN: MYAPITOKEN" "https://myorg.qualtrics.com/API/v3/responseexports/${es_id}/file" -o "/Users/myname/Desktop/WEA/response.zip"
unzip "/Users/myname/Desktop/WEA/response.zip" -d "/Users/myname/Desktop/WEA/"
I intermittently get the following error-
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 339 100 133 100 206 165 256 --:--:-- --:--:-- --:--:-- 256
{"result":{"percentComplete":0.0,"file":null,"status":"in progress"},"meta":{"httpStatus":"200 - OK","requestId":"2c55524c-03aa-495c-8de7-b54d5b441b34"}} % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 129 100 129 0 0 405 0 --:--:-- --:--:-- --:--:-- 405
End-of-central-directory signature not found. Either this file is not
a zipfile, or it constitutes one disk of a multi-part archive. In the
latter case the central directory and zipfile comment will be found on
the last disk(s) of this archive.
unzip: cannot find zipfile
directory in one of /Users/myname/Desktop/response.zip or
/Users/myname/Desktop/WEA/response.zip.zip, and cannot find /Users/myname/Desktop/WEA/response.zip.ZIP, period.
The odd part is that this error usually only occurs when I first run the script. If I rerun it, without making any changes, it will go through with no errors. To my understanding this error message means a corrupted zip file. Am I corrupting the POST request by saving it into a variable? I need to capture the post request output somehow as it provides the es_id that I need for the subsequent GET request. I would hard code it but the es_id refreshes weekly.

Answer 1

Using #T.Gibbons advice I was able to eliminate the error on the initial run by incorporating a while loop into the script.
STARTDATE=$(date -v-7d "+%Y-%m-%d")
STARTDATESTRING=$STARTDATE"T00:00:00-07:00:00"
ENDDATE=$(date "+%Y-%m-%d")
ENDDATESTRING=$ENDDATE"T00:00:00-07:00:00"
post_response=$(curl -X POST -H 'X-API-TOKEN: MYAPITOKEN' -H 'Content-Type: application/json' -d '{
"surveyId": "SV_00000000000000",
"startDate": "'"$STARTDATESTRING"'",
"endDate": "'"$ENDDATESTRING"'",
"format": "csv",
"useLocalTime": true,
"useLabels": true
}' "https://myorg.qualtrics.com/API/v3/responseexports")
es_id=$(echo "$post_response" | /usr/local/Cellar/jq/1.5_2/bin/jq --raw-output '.result.id')
percent_complete=0
while [ $percent_complete -ne 100 ]
do
response=$(curl -H "X-API-TOKEN: MYAPITOKEN" "https://myorg.qualtrics.com/API/v3/responseexports/${es_id}")
percent_complete=$(echo "$response" | /usr/local/Cellar/jq/1.5_2/bin/jq --raw-output '.result.percentComplete')
done
curl -X GET -H "Content-Type: application/json" -H "X-API-TOKEN: MYAPITOKEN" "https://myorg.qualtrics.com/API/v3/responseexports/${es_id}/file" -o "/Users/myname/Desktop/WEA/response.zip"
unzip "/Users/myname/Desktop/WEA/response.zip" -d "/Users/myname/Desktop/WEA"
rm "/Users/myname/Desktop/WEA/response.zip"