Curl: how to pass value of specific key from file - bash

wasn't able to find and I'm not sure if it possible at all. I know that I can pass the whole payload via #file.json, something like
$ curl -X POST -H "Content-Type: application/json" -d #test.json \
http://localhost/api/v3/endpoint
$ cat test.json
{
"key1": "value1",
"key2": "value2"
}
But now I need to upload gpg public key and I can't store the key inside test.json. What I need is something like
$ curl -X POST -H "Content-Type: application/json" \
http://localhost/api/v3/endpoint \
-d '{"key1": "value1", "key2": "#gpg-public.key"}'
Thanks in advance
As an option in bash I can use the following
$ curl -X POST -H "Content-Type: application/json" \
http://localhost/api/v3/endpoint \
-d "{\"key1\": \"value1\", \"key2\": \"$(cat gpg-public.key)\"}"
but I'm looking for more elegant way

I can't say this is more elegant, but at least it's different.
jq -Rn '{key2: input}' <pgp-public.key |
jq -s '.[0] * .[1]' test.json - |
curl -d #- ...
Credit for the jq hacks: Simo Kinnunen's answer here and Charles Duffy's answer here.

The first rule of JSON is to use a tool that knows JSON to generate dynamic values. A good tool is jq. Now, instead of storing a JSON file on disk, store a jq filter that can act as a template for your JSON file.
# test.jq
{
key1: "value1",
key2: $gpg
}
Now, you can run jq using this template as a filter, and supply a value for it to replace $gpg:
$ jq -n -f test.jq --arg gpg hi
{
"key1": "value1",
"key2": "hi"
}
Once you have this command, you can use it to pass the generated JSON directly to curl via standard input; the -d option can take #- as an argument to read from standard input.
jq -n -f test.q --arg gpg hi | curl ... -d #-
For simple jobs, you can specify the filter on the command line instead of reading it from a file.
jq -n '{key1: "value1", key2: $gpg}' --arg gpg hi | curl -d #-

use multipart/form-data with the -F parameter, it supports uploading multiple files at once.
curl -F json=#test.json -F pgp=#pgp-public.key http://localhost/api/v3/endpoint
ofc, the API endpoint will also need to support multipart/form-data for this to work.
edit: going by your comments, if you have to send a single json file, then edit the json file as needed and send the edited version, unfortunately bash is not a suitable language for editing json (although i'm sure it's possible), i'd recommend using some other scripting language for that. here's an example with php-cli to edit the json, then sending the edited json to curl via stdin:
php -r '$p=json_decode(file_get_contents("test.json"),true);$p["key2"]=file_get_contents("pgp-public.key");echo json_encode($p);' |
curl -d #- -X POST -H "Content-Type: application/json" http://localhost/api/v3/endpoint
(but any language with json support should suffice, like python/perl/php)

Related

extract a value from the output of a script and store in a variable winodws?

According to the document Get Azure AD tokens for service principals:
curl -X POST -H 'Content-Type: application/x-www-form-urlencoded' \
https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/token \
-d 'client_id=<client-id>' \
-d 'grant_type=client_credentials' \
-d 'scope=2ff814a6-3304-4ab8-85cb-cd0e6f879c1d%2F.default' \
-d 'client_secret=<client-secret>'
Now, I could get the correct output,like:
The Azure AD access token is in the access_token value within the output of the call.
What I want is that I need the get the value of the access_token and set it to the variable, so that I could use it in next REST API scripts.
But I'm not very familiar with Bash and curl, can anyone offer advice?
use jq to extract access_token from the json, and VAR=$(...) to store it in a variable,
ACCESS_TOKEN=$(curl -X POST -H 'Content-Type: application/x-www-form-urlencoded' \
https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/token \
-d 'client_id=<client-id>' \
-d 'grant_type=client_credentials' \
-d 'scope=2ff814a6-3304-4ab8-85cb-cd0e6f879c1d%2F.default' \
-d 'client_secret=<client-secret>' \
| jq -r .access_token )
then you can use ACCESS_TOKEN like
curl -d access_token="$ACCESS_TOKEN"
but be wary, bash is a shitty scripting language, you should not attempt to use bash for complex logic, you should probably switch to a better scripting language like Python, Perl, or PHP, rather than implementing complex logic in Bash. (same goes for Windows's cmd and PowerShell. all 3 are languages unsuitable, but not incapable, of complex logic)

Passing Variables in CURL GET

I'm trying to write a sh script that reads data from a JSON, and by parsing, pass the data as variables for a curl request (GET)
The parsing is done correctly but the CURL request continues to give problems.
I parse the variables, after opening the JSON, with the help of JQ:
ruleId=$($JSON | jq '.[].ruleId')
alert=$($JSON | jq '.[].alertName')
...
I'm passing the variables in this way:
curl --data-urlencode "ruleId=$ruleId" --data-urlencode "newLevel=$newLevel" --data-urlencode "url=$url" --data-urlencode "urlIsRegex=$urlIsRegex" --data-urlencode "enabled=$enabled" --data-urlencode "parameter=$parameter" --data-urlencode "evidence=$evidence" "http://localhost:8090/JSON/alertFilter/action/addGlobalAlertFilter"
The error that i get back is:
curl: (3) URL using bad/illegal format or missing URL
Can you help me?
Here is my test script:
!/bin/sh
#set -e
# Info to test this script
# 1. Start docker
# 2. Run this command from terminal: docker run -u zap -p 8080:8090 -i owasp/zap2docker-stable zap.sh -daemon -host 0.0.0.0 -port 8080 -config api.disablekey=true -config api.addrs.addr.name=.* -config api.addrs.addr.regex=true
# 2. Get the JSON from api (test for now)
# 3. bash filter.sh
# 4. Check for filter to be set with browser http://localhost:8080/UI/alertFilter/ => view global filter
#open example JSON file (API) and check for alert filter list
curl -s 'https://api.npoint.io/c29e3a68be632f73fc22' > whitelist_tmp.json
whitelist=whitelist_tmp.json
#Parser WITOUT loop
ruleId=$($whitelist | jq '.[].ruleId')
alert=$($whitelist | jq '.[].alertName')
newLevel=$($whitelist | jq '.[].newLevel')
url=$($whitelist | jq '.[].url')
urlIsRegex=$($whitelist | jq '.[].urlIsRegex')
enabled=$($whitelist | jq '.[].enabled')
parameter=$($whitelist | jq '.[].parameter')
evidence=$($whitelist | jq '.[].evidence')
#Fist test-query WITHOUT url-encoding => not working
#curl -X -4 --retry 2 --retry-connrefused --retry-delay 3 "http://localhost:8080/JSON/alertFilter/action/addGlobalAlertFilter/?ruleId=${ruleId}\&newLevel=${newLevel}\&url=${url}\&urlIsRegex=${urlIsRegex}\&parameter=${parameter}\&enabled=${enabled}\&evidence=${evidence}"
#Second test-query WITH url-encoding
curl --data-urlencode "ruleId=$ruleId" --data-urlencode "newLevel=$newLevel" --data-urlencode "url=$url" --data-urlencode "urlIsRegex=$urlIsRegex" --data-urlencode "enabled=$enabled" --data-urlencode "parameter=$parameter" --data-urlencode "evidence=$evidence" "http://localhost:8080/JSON/alertFilter/action/addGlobalAlertFilter"
jq accepts a filename argument, and it would be better to use that than change whitelist to include a call to cat:
jq -r 'MYFILTER' "$whitelist"
Using filters such as .[].evidence' makes your script very fragile, as it assumes the top-level array will always have just one entry. If you want just the first entry in the array, you could specify that, though maybe using first would be even more robust:
first(.[].evidence)
Although jq is fairly light-weight, it is worth noting that all those calls to jq could be reduced to one, though of course your shell script would then have to be modified as well. See e.g.
Attempting to assign multiple variables using jq
Try changing
whitelist=whitelist_tmp.json
to
whitelist="cat whitelist_tmp.json"
I think the error was occurring because shell was trying & failing to execute "whitelist_tmp.json" as its own command/function instead of piping the contents of the file to jq.
You also want to add the -r option to jq in the variables you are going to pass in curl, like so:
ruleId=$($whitelist | jq -r '.[].ruleId')
alert=$($whitelist | jq -r '.[].alertName')
newLevel=$($whitelist | jq -r '.[].newLevel')
url=$($whitelist | jq -r '.[].url')
urlIsRegex=$($whitelist | jq -r '.[].urlIsRegex')
enabled=$($whitelist | jq -r '.[].enabled')
parameter=$($whitelist | jq -r '.[].parameter')
evidence=$($whitelist | jq -r '.[].evidence')
Without the -r option, any strings that jq outputs will be quoted, which could be escaping your script's quotation in the curl command. The -r option tells jq to output the raw string data without any quotation.

bash script to convert string value to json and then return json

I am very new to bash scripting, please can someone point me in the right direction on how to accomplish my task?
I have a curl call which returns a string, which I want to convert to json.
My curl statement -
curl --insecure -X POST 'https://url/api/IPAM/GetIP' --header 'Content-Type: application/json' --header -d '{"key1": "value1"}'
This curl statement returns a string ,for example: 10.100.100.100
I want to fetch this string and return the output in json format:
{"IP":"10.100.100.100"}
I don't want to use jquery or python to do this because this entire script will be run by a wrapper that only understands bash.
You can use jq to process your IP string into a JSON string and package it into a JSON object of your choice.
ip="10.100.100.100"
jq --arg ip "$ip" -cn '{"IP":$ip}'
Result:
{"IP":"10.100.100.100"}
Now if working with the result of your example curl POST request:
rawip_string=$(curl --insecure -X POST 'https://url/api/IPAM/GetIP' --header 'Content-Type: application/json' --header -d '{"key1": "value1"}')
jq --arg ip "$rawip_string" -cn '{"IP":$ip}'
One way to not rely on external tools like jq, you can get the output ip attribute that to a variable and concatenate it.
$ return=$(echo 10.100.100.100)
$ echo "{\"IP\":\"${return}\"}"
{"IP":"10.100.100.100"}
Like this
printf '{"IP":"%s"}' "$(curl --insecure -X POST 'https://url/api/IPAM/GetIP' --header 'Content-Type: application/json' --header -d '{"key1": "value1"}')"

"Unexpected end of JSON input" error when trying to do curl POST command [duplicate]

This question already has answers here:
When to wrap quotes around a shell variable?
(5 answers)
Closed 2 years ago.
I am having issues with sending the proper json data in a curl -X POST command. I have successfully run the POST command locally on my mac by copy and pasting the hardcoded values in but I am now trying to create a .sh script to automate this process. Upon running the code below I get this error:
{"message":"Unexpected end of JSON input"}
Here is the output json from JSON_STRING with names made generic and nothing else changed:
{ "url": "api_url", "tileset": "username.filename" }
Once I can figure out how to properly format the json in the POST command I know it will work, but I can't seem to get the syntax right. Hoping a set of fresh/experience bash eyes will be able to catch my mistake:). Also, all variables that I have are correct and already been confirmed by running variable values in mac terminal. Thanks in advance for the help!
curl_url="http://${bucket}.s3.amazonaws.com/${key}"
echo $curl_url
tileset_id="username.filename"
JSON_STRING=$(jq -n \
--arg bn "$curl_url" \
--arg on "$tileset_id" \
'{url: $bn, tileset: $on}')
echo $JSON_STRING
curl -X POST -H "Content-Type: application/json" -H "Cache-Control: no-cache" -d $JSON_STRING 'apiurl'
Absolutely required to quote the shell variable:
curl ... -d "$JSON_STRING" http://example.com/end/point
Otherwise, the shell will do word splitting, and the argument to -d becomes just {"url":
as a side note, bash arrays can help with the readability:
curl_url="http://${bucket}.s3.amazonaws.com/${key}"
tileset_id="username.filename"
JSON_STRING=$(
jq -n \
--arg bn "$curl_url" \
--arg on "$tileset_id" \
'{url: $bn, tileset: $on}'
)
curl_opts=(
-X POST
-H "Content-Type: application/json"
-H "Cache-Control: no-cache"
-d "$JSON_STRING"
)
curl "${curl_opts[#]}" 'apiurl'

POST multiple files with -d in curl

I'm using curl to create several classifications. I have written the json for the many classifications and they are in one folder. I would like to create all the classifications in one go. But using curl I can only create them one at a time. How could I make them in one request?
curl -u admin:admin -H "Content-Type: application/json" -X POST -d #pii.json http://127.0.0.1:21000/api/atlas/v2/types/typedefs
The curl manual for -d says 'Multiple files can also be specified'. How can I do this? All my attempts have failed.
Do I need a bash script instead? If so, could you help me - I'm not a coder and I'm struggling without an example!
Thanks in advance.
You probably don't want to use multiple -d with JSON data since curl concatenates multiple ones with a & in between. As described in the man page for -d/--data:
If any of these options is used more than once on the same command
line, the data pieces specified will be merged together with a
separating &-symbol. Thus, using '-d name=daniel -d skill=lousy' would
generate a post chunk that looks like 'name=daniel&skill=lousy'.
You can however easily and conveniently pass several files on stdin to let curl use them all in one go:
cat a.json b.json c.json | curl -d#- -u admin:admin -H "Content-Type: application/json" http://127.0.0.1:21000/api/atlas/v2/types/typedefs
(please note that -X POST has no place on a command line that uses -d)
I found the following to work in the end:
<fileToUpload.dat xargs -I % curl -X POST -T "{%}" -u admin:admin -H "Content-Type: application/json" http://127.0.0.1:21000/api/atlas/v2/types/typedefs
Where fileToUpload.dat contained a list of the .json files.
This seemed to work over Daniel's answer, probably due to the contents of the files. Hopefully this is useful to others if Daniel's solution doesn't work for them.
I needed to upload all the *.json files from a folder via curl and I made this little script.
nfiles=*.json
echo "Enter user:"
read user
echo "Enter password:"
read -s password
for file in $nfiles
do
echo -e "\n----$file----"
curl --user $user:$password -i -X POST "https://foo.bar/foo/bar" -H "Content-Type: application/json" -d "#$file"
done
Maybe fits your needs.

Resources