Variable expansion is not working as intended with curl - shell

I am having some trouble figuring out the expansion of a variable in my shell script. If I replace the variable with the desired string it works.
#!/bin/zsh
KEY="$(curl -Ivs -X GET "http://admin:admin#192.168.1.1" &> >(awk '/^> Authorization/{ print $3 " " $4 }'))"
# The string returned by the curl and awk command is Basic "YWRtaW46YWRtaW4=" without double quotes.
curl -H "Authorization: $KEY" "http://192.168.1.1/userRpm/WlanMacFilterRpm.htm?Page=1&exclusive=1"
# This doesn't work
curl -H "Authorization: Basic YWRtaW46YWRtaW4=" "http://192.168.1.1/userRpm/WlanMacFilterRpm.htm?Page=1&exclusive=1"
# This works
The only thing thats different in the above two lines is.
-H "Authorization: $KEY"
-H "Authorization: Basic YWRtaW46YWRtaW4="

HTTP generally uses DOS style CR+LF line terminators. Whenever you parse data out of curl, you have to account for this.
To check if this is the problem, run your script with bash -x yourscript or zsh -x yourscript to see trace output that shows otherwise invisible carriage returns:
var=$'value\C-M' # zsh
var=$'value\r' # bash
(Dash and ash/busybox unfortunately doesn't highlight this problem, so try with one of the above shells)
To strip them, pipe your data through tr -d '\r'.

Related

curl set dynamic header token throws invalid key

Im trying to hit a rest api with token in header.
apikeyName="$(date '+%s')"
key=$(curl -k -X POST -H "Content-Type: application/json" \
-d '{"name":"'$apikeyName'", "role": "Admin"}' \
http://admin:admin#localhost:3000/api/auth/keys | jq '.key')
echo $key
# # Alerting API
curl -k -X GET 'http://localhost:3000/api/alert-notifications' -H 'Authorization: Bearer '$key'';
Terminal output
"eyJrIjoiaWJPaDFFZXZMeW1RYU90NUR4d014T3hYUmR6NDVUckoiLCJuIjoiMTY3NTM1OTc4OCIsImlkIjoxfQ=="
{"message":"invalid API key","traceID":""}
First 1 is the key printing and last one from api response. I tried to hardcode the key and it works.
Short answer: Use jq -r '.key' to extract the key from the json response without adding quotes to it.
Long answer: There is a difference between quotes on the command line and quotes embedded in a variable. Consider:
key='"abcd"'
printf '%s\n' $key "abcd"
# prints:
# "abcd"
# abcd
Quotes on the command line are bash syntax. Bash notes what is being quoted and then removes the quotes from the command line when it's done, thus printf only prints abcd in the second case above.
Quotes inside a variable are plain old data. Bash doesn't do anything with them, so they get passed through to the command like any other data and printf prints "abcd" in the first case.
In your curl case the receiver doesn't expect the key to have quotes embedded in the data. So, curl -blah "keydata" works fine because bash takes the quotes out, but curl -blah $key fails because bash does NOT remove the embedded quotes.
See also: BashParser

How to execute curl command stored in heredoc in bash script?

When write bash scripts, I want to store my whole curl command in heredoc to get a better layout. The following works fine:
#/bin/bash
read -r -d '' command1 <<- MULTI_STRING_SCOPE
curl -v www.stackoverflow.com
MULTI_STRING_SCOPE
But when add some json data with the -d option, the command is executed weirdly. For example:
#/bin/bash
read -r -d '' command2 <<- MULTI_STRING_SCOPE
curl -v www.stackoverflow.com
-d '{
"hello":"world"
}'
MULTI_STRING_SCOPE
response2=$(${command2})
Wrong logs from terminal:
curl: (3) URL using bad/illegal format or missing URL
curl: (3) unmatched close brace/bracket in URL position 1:
}'
And it seems that the curl take line }' as a seperated url, and thus the json data not sent as a unit.
How to solve the problem? Any suggestions will be highly appreciated.
I learned from this post to make heredoc work with curl command.
As the comment made by #Gordon Davisson in current post, we should not mix command with data. Since the json data set to -d option is only data and other parts is command, so I decide to use heredoc to store only the json data and remain other parts to be command itself, rather than store them in string by heredoc.
The result bash script should be something like this:
#/bin/bash
response3=$(curl -v www.stackoverflow.com \
-d #- <<- MULTI_STRING_SCOPE
{
"hello":"world"
}
MULTI_STRING_SCOPE
)
Notice: heredoc indent only works with tab, not with blanks. Be careful, especially when you are working with editors like Visual Studio Code, which may have already set indent as blanks for you.

Extract a specific value from an output stream in bash

I have this command:
`curl -X POST https://server.domain.com/v2/jobs/$job_ID/runs?project_id=$project_ID -H "Content-Type: application/json" -H "Authorization: Bearer ${token}" -d "{ \"job_run\": {} }"
and it gives this output
quote
Response from job execution is {"metadata":{"rov":{"mode":0,"collaborator_ids":{}},"project_id":"aff59748-260a-476e-9578-b4f4a93e7a92","sandbox_id":"aff59748-260a-476e-9578-b4f4a93e7a92","usage":{"last_updated_at":"2021-02-12T00:31:19Z","last_updater_id":"1000331040","last_update_time":1613089879840,"last_accessed_at":"2021-02-12T00:31:19Z","last_access_time":1613089879840,"last_accessor_id":"1000331040","access_count":0},"name":"Notebook Job","description":"","tags":[],"asset_type":"job_run","origin_country":"us","rating":0,"total_ratings":0,"catalog_id":"d47cd45c-0161-4b89-ba4f-c0e48272f08e","created":1613089879840,"created_at":"2021-02-12T00:31:19Z","owner_id":"1000331040","size":0,"version":2,"asset_state":"available","asset_attributes":["job_run"],"asset_id":"5d9dae4e-0dfe-4e53-9e56-18a4bb44af","asset_category":"USER"},"entity":{"job_run":{"job_ref":"28723316-9373-44ba-9229-7c796f21b099","job_name":"Dummy Job for E2E Testing","job_type":"notebook","state":"Starting","isScheduledRun":false,"configuration":{"env_id":"jupconda37-aff59748-260a-476e-9578-b4f4a93e7a92","version":"ca0b9a58-ba0d-4d28-93d5-3cdc66a5b137","env_type":"notebook","env_variables":[],"job_manager_id":"9268a8a8-72e4-47a1-a851-65f38d8284ce","notebook_job_output":{},"command_line_arguments":[]},"project_name":"Tahoe Migration"}},"href":"/v2/assets/5d9dae4e-0dfe-4e53-9e56-ee18a4bb44af?project_id=aff59748-260a-476e-9578-b4f4a93e7a92"}
I want to extract the value of asset_id from the output stream and store it in a variable within shell script. I tried storing the response in a bash variable and tried to parse with awk, it didn't work. Could one of you please provide me an idea as to how to go about this?
Thank you
jq is much better for processing json, although you need to strip off the "quote Response from job execution is " prefix first. Something like this:
response=$(curl ...)
asset_id=$(echo "${response#*Response from job execution is }" | jq -r '.metadata.asset_id')
If that's the only thing you need, you can solve it with sed:
<YOUR_CURL> | sed 's/.*"asset_id":"\([0-9a-z\-]*\)".*/\1/'

Variable expansion in shell script

I am having some problem expanding a variable.
First I set the value of the key.
KEY=Basic YWRtaW46YWRtaW4=
Then I tried calling it in curl but it was not working. To diagnose the problem, I decided to print the actual command that was being run. This was the result
print curl -H `print "Authorization: " $KEY` "http://192.168.1.1/userRpm/WlanMacFilterRpm.htm?Page=1&exclusive=0"
>> http://192.168.1.1/userRpm/WlanMacFilterRpm.htm?Page=1&exclusive=0
It was only printing the string after the KEY. Whereas, If I replaced it myself, it was printing the correct output.
print curl -H "Authorization: Basic YWRtaW46YWRtaW4=" "http://192.168.1.1/userRpm/WlanMacFilterRpm.htm?Page=1&exclusive=0"
>> curl -H Authorization: Basic YWRtaW46YWRtaW4= http://192.168.1.1/userRpm/WlanMacFilterRpm.htm?Page=1&exclusive=0
You need to enclose the value in quotes because you have a space.
KEY="Basic YWRtaW46YWRtaW4="
Doing KEY=Basic YWRtaW46YWRtaW4= without quotes is like assigning "Basic" to "KEY" and nothing to another variable called "YWRtaW46YWRtaW4".
Then simply use it like so:
curl -H "Authorization: $KEY" "http://192.168.1.1/userRpm/WlanMacFilterRpm.htm?Page=1&exclusive=0"

bash curl POST request: "-d" option using a variable

I am trying to make a POST request. The following works without any problem;
# Get expected response
curl http://localhost:9020/xxx/xxxx/ -H Content-type:application/json
-H Accept:application/json -d '{"LED":{
"language":"CHINESE"}, "text":"1928年11月22日"}'
However, if I try and set the value for the -d option curl cannot submit the request to the server.
var="{\"LED\":{ \"language\":\"CHINESE\"}, \"text\":\"1928年11月22日\"}"
# No Response
curl http://localhost:9020/xxx/xxxx/ -H Content-type:application/json
-H Accept:application/json -d $var
Can anyone explain why the latter does not seem to work? I have tried a variety of escape characters, but no luck with that either.
You need to double-quote the variable:
curl http://localhost:9020/xxx/xxxx/ -H Content-type:application/json \
-H Accept:application/json -d "$var"
Without the double-quoting, the value is split on spaces,
and the shell interprets that as multiple additional arguments to curl,
but you need that value to be a single argument.
That's what double-quoting will achieve.

Resources