Scripted concatenation of strings in curl command payload - bash

I use curl to test an user account creation API as follows:
curl -s -X POST "https://$APISERVER/users" \
-H 'Content-Type: application/json' \
-d '{ \
"username": "'$NEWUSERNAME'", \
"firstName": "'$NEWUSERFIRSTNAME'", \
"lastName": "'$NEWUSERLASTNAME'", \
"displayName": "'$NEWUSERDISPLAYNAME'", \
"password": "'$NEWUSERPASSWORD'" \
}'
and the variables are supplied via command line arguments:
APISERVER=http://localhost:8080
NEWUSERNAME=$1
NEWUSERPASSWORD=$2
NEWUSERFIRSTNAME=$3
NEWUSERLASTNAME=$4
# Calculated variable
NEWUSERDISPLAYNAME="${NEWUSERFIRSTNAME} ${NEWUSERLASTNAME}"
An example invocation of the script is as follows: ./test-new-user.sh jdoe Hello123 John Doe, resulting in the following variable values:
NEWUSERNAME=jdoe
NEWUSERPASSWORD=Hello123
NEWUSERFIRSTNAME=John
NEWUSERLASTNAME=Doe
(I intended NEWUSERDISPLAYNAME to be set to "John Doe")
But I get back an exception from the server, because the payload in the curl command appears to be cut-off, incomplete or malformed.
JSON parse error: Unexpected end-of-input in VALUE_STRING\n at [Source:
java.io.PushbackInputStream#2eda6052; line: 1, column: 293]; nested
exception is com.fasterxml.jackson.databind.JsonMappingException:
Unexpected end-of-input in VALUE_STRING\n at [Source:
java.io.PushbackInputStream#2eda6052; line: 1, column: 293]\n at
[Source: java.io.PushbackInputStream#2eda6052; line: 1, column: 142]
(through reference chain:
com.mycompany.api.pojos.NewUser[\"displayName\"])"
If I hard code value for displayName in the above curl command (as below), the user creation request goes through and works perfectly.
"displayName": "John Doe", \
I suspect it has to do with the space in displayName and how I insert the value for displayName using "'$NEWUSERDISPLAYNAME'". Is there a safe way to perform variable substitution in the curl command's POST request payload?

You need to quote shell variables:
curl -s -X POST "https://$APISERVER/users" \
-H 'Content-Type: application/json' \
-d '{ \
"username": "'"$NEWUSERNAME"'", \
"firstName": "'"$NEWUSERFIRSTNAME"'", \
"lastName": "'"$NEWUSERLASTNAME"'", \
"displayName": "'"$NEWUSERDISPLAYNAME"'", \
"password": "'"$NEWUSERPASSWORD"'" \
}'
In order to avoid excessive quoting, try this printf:
printf -v json -- '{ "username": "%s", "firstName": "%s", "lastName": "%s", "displayName": "%s", "password": "%s" }' \
"$NEWUSERNAME" "$NEWUSERFIRSTNAME" "$NEWUSERLASTNAME" "$NEWUSERDISPLAYNAME" "$NEWUSERPASSWORD"
curl -s -X POST "https://$APISERVER/users" \
-H 'Content-Type: application/json' \
-d "$json"

just use
$(echo $varname)
works for me

Related

Mutiple varaibles in while loop of bash scripting

Suppose i have the below curl where i will be reading two of the varaibles from a file.how can we accomodate both the varaibles in a single while loop
while read p; do
curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' --header 'HTTP_X_MERCHANT_CODE: STA' --header 'AK-Client-IP: 135.71.173.56' --header 'Authorization: Basic qwrewrereererweer' -d '{
"request_details": [
{
"id": "$p", #first dynamic varaible which will be fetched from the file file.txt
"id_id": "$q", #second dynamic varaible to be fetched from the file file.txt
"reason": "Pickup reattempts exhausted"
}
]
}' api.stack.com/ask
done<file.txt
file.tx will have two columns from which the dynamic variables whill be fetched to the above curl. Pls let me know how can we accomodate both the variable in the above while loop
i will need bit of a help regarding the same
Since you'll want to use a tool like jq to construct the payload anyway, you should let jq parse the file instead of using the shell.
filter='split(" ") |
{ request_details: [
{
id: .[0],
id_id: .[1],
reason: "Pickup reattempts exhausted"
}
]
}'
jq -cR "$filter" file.txt |
while IFS= read -r payload; do
curl -X POST --header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'HTTP_X_MERCHANT_CODE: STA' \
--header 'AK-Client-IP: 135.71.173.56' \
--header 'Authorization: Basic qwrewrereererweer' \
-d "$payload"
done
The -c option to jq ensures the entire output appears on one line: curl doesn't need a pretty-printed JSON value.
read accepts multiple target variable names. The last one receives all the content not yet read from the line. So read p reads the whole line, read p q would read the first token (separated by whitespace) into p and the rest into q, and read p q r would read the first two tokens into p and q and any remaining junk into r (for example if you want to support comments or extra tokens in file.txt).

how to get multiple values from a json array/object using jq

how to get multiple values from a JSON array/object using jq in a single line?
here curl request and its response
curl -s -k --location --request GET "https://${HOSTNAME}/api/v1/projects/myProjects?page=1&pageSize=2" --header "Authorization: Bearer "$token"" | jq -r '.["data"]'
# Response
[
{
"id": "8a70803f8045722601804f62d54c5d9d",
"createdBy": "8a70802980325cdc0180326b5fe60006",
"createdDate": "2022-04-22T03:48:38.860+0000",
"modifiedBy": "8a70802980325cdc0180326b5fe60006",
"modifiedDate": "2022-04-22T03:48:38.860+0000",
"version": null,
"inactive": false,
"name": "Netbank734113",
},
{
"id": "8a70801c804568ae01804f625a923f8d",
"createdBy": "8a70802980325cdc0180326b5fe60006",
"createdDate": "2022-04-22T03:48:07.442+0000",
"modifiedBy": "8a70802980325cdc0180326b5fe60006",
"modifiedDate": "2022-04-22T03:48:07.442+0000",
"version": null,
"inactive": false,
"name": "Netbank734112",
}
]
Now try the below command to get the id and name in one line but duplicate results are coming.
result=$(curl -s -k --location --request GET "https://${HOSTNAME}/api/v1/projects/myProjects?page=1&pageSize=2" --header "Authorization: Bearer "$token"" | jq -r '.["data"]|.[].name +" "+ .[].id')
echo "$result"
# response
Netbank734113 8a70803f8045722601804f62d54c5d9d
Netbank734112 8a70803f8045722601804f62d54c5d9d
Netbank734113 8a70801c804568ae01804f625a923f8d
Netbank734112 8a70801c804568ae01804f625a923f8d
how do I get rid of those duplicated values and basically get the below response?
Netbank734113 8a70803f8045722601804f62d54c5d9d
Netbank734112 8a70801c804568ae01804f625a923f8d
Tried this command with your inputs getting the below errors, how do I modify this curl command to get desired results.
result=$(curl -s -k --location --request GET "https://${HOSTNAME}/api/v1/projects/myProjects?page=1&pageSize=2" --header "Authorization: Bearer "$token"" | jq -r '["data"]|.[].unique_by(.name, .id) | map([.name, .id])[] | #tsv')
# error response
jq: error: syntax error, unexpected '(', expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
["data"]|.[].unique_by(.name, .id) | map([.name, .id])[] | #tsv
jq: 1 compile error
# This one worked
result=$(curl -s -k --location --request GET "https://${HOSTNAME}/api/v1/projects/myProjects?page=1&pageSize=2" --header "Authorization: Bearer "$token"" | jq -r '.["data"]|.[]|.| .name + " " + .id')
With .[].name + " " + .[].id' you iterate twice over the array. Iterate once and extract your data in one go:
curl … | jq -r '.data[] | .name + " " + .id'
Netbank734113 8a70803f8045722601804f62d54c5d9d
Netbank734112 8a70801c804568ae01804f625a923f8d
Demo
You might also be interested in using string interpolation:
curl … | jq -r '.data[] | "\(.name) \(.id)"'
Demo

argument list too long curl

Trying to solve "argument list too long"
I have been searching for a solution and found the closest one to my issue
curl: argument list too long
however the response is not clear as I am still having the issue "argument list too long"
curl -X POST -d #data.txt \
https://Path/to/attachments \
-H 'content-type: application/vnd.api+json' \
-H 'x-api-key: KEY' \
-d '{
"data": {
"type": "attachments",
"attributes": {
"attachment": {
"content": "'$(cat data.txt | base64 --wrap=0)'",
"file_name": "'"$FileName"'"
}
}
}
}'
thank you
Use jq to format your base64 encoded data string into a proper JSON string, and then pass the JSON data as standard input to the curl command.
#!/usr/bin/env sh
attached_file='img.png'
# Pipe the base64 encoded content of attached_file
base64 --wrap=0 "$attached_file" |
# into jq to make it a proper JSON string within the
# JSON data structure
jq --slurp --raw-input --arg FileName "$attached_file" \
'{
"type": "attachments",
"attributes": {
"attachment": {
"content": .,
"file_name": $FileName
}
}
}
' |
# Get the resultant JSON piped into curl
# that will read the data from the standard input
# using -d #-
curl -X POST -d #- \
'https://Path/to/attachments' \
-H 'content-type: application/vnd.api+json' \
-H 'x-api-key: KEY'
Per the linked answer
you are trying to pass the entirety of the base64'd content on the command line
This is a limitation of the shell, not curl. That is, the shell is responding with error argument list too long. The program curl is never even started.
The recommendation is
curl has the ability to load in data to POST from a file
Write the json data to some file /tmp/data.json using piping.(the commands will use piping | and file redirection > >> which can handle arbitrarily large amounts of data. Whereas, you cannot place arbitrarily large amounts of data into a single command, there is a limit).
echo -n '
{
"data": {
"type": "attachments",
"attributes": {
"attachment": {
"content": "' > /tmp/data.json
cat data.txt | base64 --wrap=0 >> /tmp/data.json
echo -n '",
"file_name": "'"$FileName"'"
}
}
}
}' >> /tmp/data.json
Pass that file path /tmp/data.json to the curl command using # so curl knows it's a file path.
curl -X POST -d #/tmp/data.json \
"https://Path/to/attachments" \
-H 'content-type: application/vnd.api+json' \
-H 'x-api-key: KEY'

for loop over sequence, then make string and post via json

Consider this simple bash script : (I am writing myurl as a placeholder for the correct URL; and myhost as the placeholder for the correct hostname for privacy here.)
for a in `seq 400 450`; do
curl --request POST \
--url myurl \
--header 'Content-Type: application/json' \
--header 'Host: myhost' \
--data '{
"devID": "21010010",
"count": 0,
"fileEnd": 1,
"line": "",
"file": "$a"
}'
done
This is rejected as bad request. I want, the output of seq, (e.g. the number 410) to be a quote delimated string. Notice the line "file": "$a". If I write "file": "410", this works perfectly, and also for any other file between 400 and 450 - as long as it is surrounded by "" quotes.
Question :
I want, that in each iteration of the loop, the curl request should include the number coming from the seq command surrounded by "". So the first call should be :
curl --request POST \
--url myurl \
--header 'Content-Type: application/json' \
--header 'Host: myhost' \
--data '{
"devID": "21010010",
"count": 0,
"fileEnd": 1,
"line": "",
"file": "400"
}'
The next one to be :
curl --request POST \
--url myurl \
--header 'Content-Type: application/json' \
--header 'Host: myhost' \
--data '{
"devID": "21010010",
"count": 0,
"fileEnd": 1,
"line": "",
"file": "401"
}'
and so on.
How can I do this? Thank you.
The --data portion is wrapped in single quotes ...
'{"devID": ... "file": "$a"}'
... which disables the expansion of variable references inside said single quotes.
Consider:
$ a=410
$ echo '"devID": "21010010", "file": "${a}"'
"devID": "21010010", "file": "${a}" # ${a} not expanded
You could wrap the --data section in double quotes, but then you need to also escape all of the nested double quotes.
Another idea is to break the --data section into 2x sections wrapped in single quotes and place the ${a} reference between the 2x sections, eg:
$ echo '"devID": "21010010", "file": "'${a}'"' # change "${a}" to "'${a}'"
"devID": "21010010", "file": "410" # ${a} successfully expanded
NOTE: There are several posts regarding the difference between single and double quotes, eg, this link

cURL command doesn't work from terminal prompt

Following the documentation here under /campaigns I try the example with my api-key.
I am using the terminal window on a Mac (bash shell).
curl -X POST https://api.mailerlite.com/api/v2/campaigns \
-d '{
"subject": "Regular campaign subject",
"segments": 1291259,
"type": "regular"
}' \
-H "Content-Type: application/json" \
-H "X-MailerLite-ApiKey: <api-key>" \
I paste this whole thing into the prompt even though that doesn't seem the correct way to do this.
Part of the command is understood and then part way through, the shell asks this Display all 114 possibilities question:
~/: curl -X POST https://api.mailerlite.com/api/v2/campaigns \
> -d '{
>
Display all 114 possibilities? (y or n)
Followed by a directory listing, Followed by:
> campaign subject",
> "segments": 1291259,
> "type": "regular"
> }' \
> -H "Content-Type: application/json" \
> -H "X-MailerLite-ApiKey: <api-key>" \
>
{"error":{"code":400,"message":"Campaign type is missing"}}~/:
What is the proper way to issue a multi-line cURL command from a bash shell?
Replace tabs with spaces, and drop the last backslash, then it should work just fine.
curl -X POST https://api.mailerlite.com/api/v2/campaigns \
-d '{
"subject": "Regular campaign subject",
"segments": 1291259,
"type": "regular"
}' \
-H "Content-Type: application/json" \
-H "X-MailerLite-ApiKey: <api-key>"
It gives the following output to me:
$ curl -X POST https://api.mailerlite.com/api/v2/campaigns \
> -d '{
> "subject": "Regular campaign subject",
> "segments": 1291259,
> "type": "regular"
> }' \
> -H "Content-Type: application/json" \
> -H "X-MailerLite-ApiKey: <api-key>"
{"campaign_type":"regular","date":"2020-10-06 19:21:47","account_id":<id>,"id":<id>,"mail_id":<id>,"options":{"current_step":"step3","send_type":"regular","campaign_type":"regular","date":"2020-10-06 19:21:47"}}

Resources