I wanna GET request with variable x in command like this
curl 'abc.com' --data '{"songs":{"module":"server","method":"get_songs","param":{"songid":$x}}}'
but the command can't get the value of x, because it's nested too deeply!
Hope you help!
single quotes is hiding the variable from bash, the nesting doesn't matter, it's just a string.
you can escape the variable form single quotes, let bash substitute the value and then join back to the rest of the string.
... '{"songs":{"module":"server","method":"get_songs","param":{"songid":'"$x"'}}}'
The variable $x is not replaced because it is embedded in a string enclosed in apostrophes. The variables are not expanded and the escape sequences do not work in the strings enclosed in apostrophes.
There are multiple solutions for your problem.
The simplest one is to close the string before the variable and use another string after it:
curl 'abc.com' \
--data '{"songs":{"module":"server","method":"get_songs","param":{"songid":'$x'}}}'
However, this produces unexpected results if the value of $x contains spaces or characters that are special to the shell (like * etc).
It has a very simple fix: let the variable $x be outside the strings enclosed in apostrophes but enclose it in quotes:
curl 'abc.com' \
--data '{"songs":{"module":"server","method":"get_songs","param":{"songid":'"$x"'}}}'
However, while this works, it is not easy to read and the quotes around $x can be confused with the quotes from the JSON.
I recommend to use the here-doc syntax to build the JSON:
DATA=$(cat << END
{"songs":{"module":"server","method":"get_songs","param":{"songid":$x}}}
END
)
curl 'abc.com' --data "$DATA"
This way the variable is expanded and there is no need for extra quotes or apostrophes.
This approach (and also your initial approach) has a major drawback. You need to write the JSON manually and a missing or an extra comma or quote will break it. A simple way to prevent this is to use a tool (for example jq) to generate the JSON:
DATA=$(echo -n "$x" | jq -s -R '
{
songs: {
module: "server",
method: "get_songs",
param: { songid: . }
}
}
')
curl 'abc.com' --data "$DATA"
The advantage of using jq (or other tool designed to work with JSON) becomes visible when $x contains characters that must be encoded when used in JSON (quotes, newlines). The variable substitution provided by the shell does not do any encoding for them, the resulting JSON is malformed and the server where you send it will reject your request.
Related
I am trying to write a shell script that invokes two APIs using curl.
One of the keys of JSON output of the first curl is passed to the second curl. In the Bash script below, I am passing token as a command line parameter to the first curl and it works fine.
The output of the first curl is extracted into client_token and I am passing it to the second curl. It is failing.
The reason being, wherever I have $client_token, the value is getting substituted as "value" (with quotes) instead of value (without quotes). Curl expects strings without quotes in the second curl. How can I get rid of double quotes?
echo $1
XVaultToken=`curl -X POST "https://sub.domain.tld:8200/login" -d '{"token":"'"$1"'"}'`
client_token=`echo $XVaultToken|jq '.auth.client_token'
echo $client_token
apiKey=`curl -X GET https://sub.domain.tld:8200/api-key -H 'X-Vault-Token: "'"$client_token"'"'`
#apiKey=`curl -X GET https://sub.domain.tld:8200/api-key -H 'X-Vault-Token: $client_token'`
echo "apikey"
Probably your jq command is outputting the quotes that you don't want. Ask jq for the raw value instead:
client_token=`echo $XVaultToken|jq -r '.auth.client_token'
Recently discovered jq and am using it to format some data.
How do I use it to access fields of a json object that happen to be numeric strings?
For example, the following fails for me with an error:
echo '{"20":"twenty"}' | jq .["20"]
What's the right way to do this?
Immediate Answer: Use More Quotes
In jq .["20"], the double quotes are parsed as shell syntax, not jq syntax (shell quoting is character-by-character: One can switch quoting types within a larger string). Use single quotes to protect that entire string from modification by the shell:
$ echo '{"20":"twenty"}' | jq '.["20"]'
"twenty"
Finding The Problem Yourself
One approach to diagnosing this kind of problem is using the shell's xtrace facility, to tell the shell to echo back to you the command lines it's running:
$ set -x
$ echo '{"20":"twenty"}' | jq .["20"]
+ echo '{"20":"twenty"}'
+ jq '.[20]'
jq: error (at <stdin>:1): Cannot index object with number
As you can see, jq .["20"] was parsed as being identical to jq '.[20]'
This question already has answers here:
Expansion of variables inside single quotes in a command in Bash
(8 answers)
Closed 4 years ago.
I have a very simple bash script with three commands.
The first command strips the first word off of the last git commit, the second command attempts to make a POST call to an api endpoint, with that same variable as part of the call, and the third command just prints that variable, to ensure it was working properly. See the code below
SOMETHING=$(git log -1 --pretty=%B | head -n1 | sed -e 's/\s.*$//' | cut -d ' ' -f1)
curl -X POST \
http://www.someurl.com/ \
-H 'Cache-Control: no-cache' \
-d '{"item":"$SOMETHING"}'
echo "variable was $SOMETHING"
When I run that bash script, I get a response from the service saying that "item was not set properly" in XML, however it does correctly echo the correct variable. So I know that first line is working. If I copy that curl command and paste it into bash, replacing $SOMETHING with the actual value, it works fine.
Single quotes do not expand the $variables inside them.
Try
'{"item":"'"$SOMETHING"'"}'
instead. Brief explanation:
'{"item":"' is a string delimited by single quotes that contains double quotes
"$SOMETHING" is a string delimited by double quotes, that expands the variable $SOMETHING
'"}' is again a ''-delimited string that contains double quotes
Simply writing those strings in a row without gaps is string concatenation
In this way, you get your variable expansion, but don't have to insert any backslashes to escape the double quotes.
How does one format a string leveraging an environment variable within at the command line? For example, I want to curl and pass some variable, i.e.:
curl -X POST --data-urlencode 'payload={"text": "I want to print an environment variable here, eg a path: $PATH"}' https://someapi.com/
The easy answer is to change the kind of quotes you're using:
curl -X POST --data-urlencode \
'payload={"text": "I want to print an environment variable here, eg a path: '"$PATH"'"}' \
https://someapi.com/
Notably, this is still using single-quotes on the outside (so you don't need to change your payload), but then it ends the single quotes, starts double quotes, and embeds your substitution in those double quotes (before ending them and switching back to single quotes, within which literal -- rather than syntactic -- double quotes can be embedded).
A variant on this approach, which avoids the need for syntactic quotes mixed into the document content, is to used an unquoted heredoc, as advised by #chepner in the comments:
curl -X POST --data-urlencode #- https://someapi.com/ <<EOF
payload={"text": "I want to print an environment variable here, eg a path: $PATH"}
EOF
The better answer is to use a tool that knows how to format JSON; jq is a widely popular choice. Consider the following example:
text="I want to print an environment variable here, eg a path: \"$PATH\""
curl -X POST --data-urlencode #- https://someapi.com/ <<EOF
payload=$(printf '%s\n' "$text" | jq -R '{text: .}')
EOF
This way you're guaranteed valid output, even if your environment variable contains backslashes, literal quotes, nonprintable characters, or whatever else may come.
In my bash_profile I created a deploy function. There I read a path from a file:
deployPath=$(jq '.["prod-deploy-path"]' ./package.json)
When I print the path, I get the following:
"/var/www/user/websitefolder/dist"
Now I want to remove the last part of the URL:
pwdPath=$(dirname $deployPath)
but I receive
"/var/www/user/websitefolder/dist
(notice the last quotation mark is missing)
If I do the same thing with a fixed string instead of the variable
dirname "/var/www/user/websitefolder/dist"
I receive what I want to receive:
dirname "/var/www/user/websitefolder/"
What do I need to write in my bash_profile to get the same result with the variable as with a string?
You will need to use the -r option for jq:
deployPath=$(jq -r '.["prod-deploy-path"]' ./package.json)
which will output a raw string.
Your problem is not that you got a variable but that the content in the variable are wrapped in quotes, eg:
with_quotes='"something here"'
without_quotes='something here'
echo "$with_quotes" # "something here"
echo "$without_quotes" # something here
jq will by default print a JSON encoded string. This also means that if your string contains double quotes they will be escaped like this: "hello \"world"
From man jq:
--raw-output / -r
With this option, if the filter's result is a string then it will be written
directly to standard output rather than being formatted as a JSON string
with quotes. This can be useful for making jq filters talk to non-JSON-based
systems