cURL using bash script with while read text file - bash

Hi there anyone there having the same trouble like mine?
whenever I cURL the $list from the list.txt it just displaying {} which is a blank response from the API does my code should be really working properly or it is just a bug?
I know the $list is working because I can update the database status
Please this is a bit urgennnnttt :(
#! /bin/bash
filename=/var/lib/postgresql/Script/list.txt
database='dbname'
refLink='URL'
authorization='Authorization: Basic zxc'
expireDate=$(date -d "+3 days")
body="Message."
while IFS=' ' read -r list
do
wow=$(curl --location --request POST $refLink \
--header 'Authorization: Basic $authorization' \
--header 'Content-Type: application/json' \
--data-raw '{
"title":"Expiration Notice",
"body":"$body",
"min" :[{"mobileNumber" : "$list"}],
"type" : "Notification",
"action_type" : "NotificationActivity"}')
echo "result: '$result'"
RESP=$(echo "$result" | grep -oP "^[^a-zA-Z0-9]")
echo "RESP:'$RESP'"
echo $body
#echo $wow >> logs.txt
psql -d $database -c "UPDATE tblname SET status='hehe' WHERE mobile='$list'"
done < $filename

Your "$list" JSON entry is not populated with the content of the $list variable because it is within single quotes of the --data-raw curl parameter.
What you need is compose your JSON data for the query before-hand, preferably with the help of jq or some other JSON processor, before sending it as argument to the curl's POST request.
Multiple faults in your scripts (not exhaustive):
Shebang is wrong with a space #! /bin/bash
expireDate=$(date -d "+3 days") return date in locale format and this may not be what you need for your request.
The request and the response data are not processed with JSON grammar aware tools. grep is not appropriate for JSON data.
Some clues but cannot fix your script more without knowing more about the API answers and functions you use.
Anyway here is how you can at least compose a proper JSON request:
#!/usr/bin/env bash
filename='/var/lib/postgresql/Script/list.txt'
database='dbname'
refLink='URL'
authorization='zxc'
expireDate=$(date -R -d "+3 days")
body="Message."
while IFS=' ' read -r list; do
raw_json="$(
jq -n --arg bdy "$body" --arg mobN "$list" \
'.action_type="NotificationActivity"|.title="Expiration Notice"|.type="Notification"|.body=$bdy|.min[0].mobileNumber=$mobN|.'
)"
json_reply="$(curl --location --request POST "$refLink" \
--header "Authorization: Basic $authorization" \
--header 'Content-Type: application/json' \
--data-raw "$raw_json")"
echo "json_reply: '$json_reply'"
echo "$body"
# psql -d "$database" -c "UPDATE tblname SET status='hehe' WHERE mobile='$list'"
done <"$filename"

Related

how to put 2 elements from jq variables in one request?

There is a part of the script where each request receives a response and is written to a variable. How to do it in one request with writing to variables?
boolStatus=$(curl -X 'GET' \
"https://tsit-app1/api/v2/workItems/$case?versionNumber=0" \
-H 'accept: application/json' \
-H "Authorization: $apiKey" | jq '.isAutomated')
echo $boolStatus
name=$(curl -X 'GET' \
"https://tsit-app1/api/v2/workItems/$case?versionNumber=0" \
-H 'accept: application/json' \
-H "Authorization: $apiKey" | jq '.name')
echo $name
I tried
curl -X 'GET' \
"https://tsit-app1/api/v2/workItems/$case?versionNumber=0" \
-H 'accept: application/json' \
-H "Authorization: $apiKey" | jq '"boolStatus=\(.isAutomated)", "name=\(.name)"'
but in echo i get
"boolStatus=true",
"name=bla bla"
need to
echo $boolStatus
true
echo $name
bla bla
One way would be to use the #sh string interpolation and then use your shell's eval.
Using the string interpolation would output something like:
boolStatus=true
name='abc'
which can then be fed to eval:
vars="$(curl ... | jq -r '#sh "boolStatus=\(.isAutomated)", #sh "name=\(.name)"')"
eval "$vars"
or explicitly output the line break:
jq -r '#sh "boolStatus=\(.isAutomated)\nname=\(.name)"'
Disclaimer: Note that this will evaluate any shell code and might open your system to malicious code (#sh escapes the values, but it's always a good idea to be aware of this).
Use process substitution to allow two uses of read to read from the output of jq.
{ read boolStatus; read name; } < <(curl ... | jq -r '.isAutomated, .name')
(assuming the name does not contain any newlines).
At the very least, you can save the output to process with jq twice.
response=$(curl ...)
name=$(echo "$response" | jq -r .name)
boolStatus=$(echo "$response" | jq -r .isAutomated)

curl: (3) URL using bad/illegal format or missing URL AND HTTP/1.1 403 Forbidden

I have a problem with this code. It states that there is a URL issue but I don't see that as right. I am not responsible for the server though. I just wanted to test some IoT properties with basic linux commands.
#!/bin/bash
a=Client secret
c=Custom domain
b=Client id:
#!/bin/bash
response=$(curl -X POST --user $b:$a "https://${c}.auth.eu-central-1.amazoncognito.com/oauth2/token?grant_type=client_credentials" -H 'Content-Type: application/x-www-form-urlencoded')
token=$(jq -r '.access_token' <<< "$response")
secondsValid=$(jq -r '.expires_in' <<< "$response")
refreshToken=$(jq -r '.refresh_token' <<< "$response")
for i in {1..4}
do
declare -i TIME
TIME=$(date +%s)
TEMP=$(sensors -j | jq '."cpu_thermal-virtual-0"."temp1"."temp1_input"')
curl -i -k -X POST -H "Authorization: $token"-H "Content-Type: application/json" --data '{ "id":"2","timestamp":"'${TIME}'","data":"'${TEMP}'"}' https://dv7knsjzph.execute-api.eu-central-1.amazonaws.com/prod/boxtronic-devices/2/data/
sleep 1
echo "$refreshToken"
echo "$secondsValid"
echo "$TIME"
done
Anyone knows why I am getting the error ?

passing values with spaces in curl command using POST

I am trying to pass values with spaces in curl POST method. I am directing the values through a txt file. POST command does not allow me to pass values with spaces using the for while loop, But when i pass it without while loop it accepts the value without any error.
Below are the commands
This works perfectly fine
curl -d '{"name": "equity calculation support", "email": "email#test.com"}' -H "Authorization: Basic YWRtaW46YWRtaW4=" -H "Content-Type: application/json" -H "Accept: application/json" -X POST http://localhost:3000/api/teams
{"message":"Team created","teamId":103}
when using while loop and IFS it doesn't take the values with spaces:
while IFS= read -r line ; do curl -d '{"name": "'$line'"}' -H "Authorization: Basic YWRtaW46YWRtaW4=" -H "Content-Type: application/json" -H "Accept: application/json" -X POST 'http://localhost:3000/api/teams'; done < /tmp/group.txt
group.txt file contains the values .
You aren't quoting the expansion of $line:
while IFS= read -r line ; do
curl -d '{"name": "'"$line"'"}' \
-H "Authorization: Basic YWRtaW46YWRtaW4=" \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-X POST 'http://localhost:3000/api/teams'
done < /tmp/group.txt
However, it's a better idea to let a tool like jq produce the JSON, to ensure that any characters in $line that need to be escaped to produce proper JSON do, indeed, get escaped.
while IFS= read -r line; do
d=$(jq -n --argjson x "$line" '{name: $x}')
curl -d "$d" ...
done < /tmp/group.txt
It looks like the JSON you want to create would fit on a single line, so you could also process all of /tmp/group.txt with a single call to jq, and pipe its output to your loop.
jq -c -R '{name: .}' | while IFS= read -r line; do
curl -d "$line" ...
done

Passing arguments values to shell script

I'm using the following script to purge cache from cdn,
#!/bin/bash
## API keys ##
zone_id=""
api_key=""
login_id=""
akamai_crd=""
## URL ##
urls="$1"
[ "$urls" == "" ] && { echo "Usage: $0 url"; exit 1; }
echo "Purging $urls..."
curl -X DELETE "https://api.cloudflare.com/client/v4/zones/${zone_id}/purge_cache" \
-H "X-Auth-Email: ${login_id}" \
-H "X-Auth-Key: ${api_key}" \
-H "Content-Type: application/json" \
--data "{\"files\":[\"${urls}\"]}"
#echo "CF is done now purging from Akamai ..."
echo "..."
curl -v -s https://api.ccu.akamai.com/ccu/v2/queues/default -H "Content-Type:application/json" -d '{"objects":["$urls"]}' -u $akamai_crd
The 1st part for cloudflare is working fine the 2nd part when I pass it to Akamai
["$urls"]
I keep getting an error and it's passing the url as an argument it returns the variable itself ($urls) not the arg value.
I ran the script as following:
sh +x script.sh url
Any advise here?
First i would change this:
SCRIPTNAME=$(basename "$0")
...
if [ $# != 1 ] then
echo "Usage: $SCRIPTNAME url"
exit
fi
$urls="$1"
Change your second curl command as the following (you need to escape the quotes):
--data "{\"files\":[\"${urls}\"]}"
Avoid creating JSON by hand like this; you can't guarantee that the resulting JSON is properly escaped. Use a tool like jq instead.
#!/bin/bash
## API keys ##
zone_id=""
api_key=""
login_id=""
akamai_crd=""
## URL ##
url=${1:?Usage: $0 url}
headers=(
-H "X-Auth-Email: $login_id"
-H "X-Auth-Key: $api_key"
-H "Content-Type: application/json"
)
purge_endpoint="https://api.cloudflare.com/client/v4/zones/${zone_id}/purge_cache"
echo "Purging $url..."
jq -n --arg url "$url" '{files: [$url]}' |
curl -X DELETE "$purge_endpoing" "${headers[#]}" --data #-
#echo "CF is done now purging from Akamai ..."
echo "..."
jq -n --arg url "$url" '{objects: [$url]}' |
curl -v -s -H "Content-Type:application/json" -d #- -u "$akamai_crd"

Unable to send large files to elasticsearch using curl: argument too long

This is the script i used to export some documents to elasticsearch but no luck
#!/bin/ksh
set -v
trap read debug
date=$(date +%Y-%m-%d);
echo $date;
config_file="/home/p.sshanm/reports_elastic.cfg";
echo $config_file;
URL="http://p-acqpes-app01.wirecard.sys:9200/reports-"$date"";
echo $URL;
find /transfers/documents/*/done/ -type f -name "ABC-Record*_${date}*.csv"|
while IFS='' read -r -d '' filename
do
echo "filename : ${filename}"
var=$(base64 "$filename"| perl -pe 's/\n//g');
#if i use below it will fail as argument too long , so i used with curl option #
# var1= $(curl -XPUT 'http://localhost:9200/reports-'$date'/document/reports?pipeline=attachment&pretty' -d' { "data" : "'$var'" }')
var1=$(curl -X PUT -H "Content-Type: application/json" -d #- "$URL" >>CURLDATA
{ "data": "$var" }
CURL_DATA)
done;
If i use below it as
var1= $(curl -XPUT 'http://localhost:9200/reports-'$date'/document/reports?pipeline=attachment&pretty' -d' { "data" : "'$var'" }')
will fail as below, so i used with curl option #
argument too long
Your syntax to read from stdin is wrong, the here-doc string should have been (<<) and the de-limiters are mis-matching use CURL_DATA at both places.
curl -X PUT -H "Content-Type: application/json" -d #- "$URL" <<CURL_DATA
{ "data": "$var" }
CURL_DATA

Resources