expand variable inside single quotes shell script - shell

I need to be able to use ${var} with preserving the single quote as the command requires it.
I have tried escaping the single quote by concatenating '"${var}"' but the command gave error as the single quotes didn't get preserved.
$COMMAND '[{"name": "john", "tel": ${var}}]'

You should quote this except for "${var}".
var=999
$COMMAND '[{"name": "john", "tel": "'${var}'"}]'

I think that your command does not "require" single quotes around that JSON
string. It's a shell that requires them to treat the entire JSON
string as a single word and pass it to $COMMAND.
There are 2 ways to use it in JSON string (I set var to 999):
$ echo command "[{\"name\": \"john\", \"tel\": ${var}}]"
command [{"name": "john", "tel": 999}]
or:
$ echo command '[{"name": "john", "tel": '"${var}"'}]'
command [{"name": "john", "tel": 999}]
However, if your command really requires JSON string enclosed in single quotes do this:
"'[{\"name\": \"john\", \"tel\": ${var}}]'"
Also, [ and ] might not be needed:
"{\"name\": \"john\", \"tel\": ${var}}"

Related

Replace string with Bash variable in jq command

I realize this is a simple question but I haven't been able to find the answer. Thank you to anyone who may be able to help me understand what I am doing wrong.
Goal: Search and replace a string in a specific key in a JSON file with a string in a Bash variable using jq.
For example, in the following JSON file:
"name": "Welcome - https://docs.mysite.com/",
would become
"name": "Welcome",
Input (file.json)
[
{
"url": "https://docs.mysite.com",
"name": "Welcome - https://docs.mysite.com/",
"Ocurrences": "679"
},
{
"url": "https://docs.mysite.com",
"name": "Welcome",
"Ocurrences": "382"
}
]
Failing script (using variable)
siteUrl="docs.mysite.com"
jq --arg siteUrl "$siteUrl" '.[].name|= (gsub(" - https://$siteUrl/"; ""))' file.json > file1.json`
Desired output (file1.json)
[
{
"url": "https://docs.mysite.com",
"name": "Welcome",
"Ocurrences": "679"
},
{
"url": "https://docs.mysite.com",
"name": "Welcome",
"Ocurrences": "382"
}
]
I've tried various iterations on removing quotes, changing between ' and ", and adding and removing backslashes.
Successful script (not using variable)
jq '.[].name|= (gsub(" - https://docs.mysite.com/"; ""))' file.json > file1.json
More specifically, if it matters, I am processing an export of a website's usage data from Azure App Insights. Unfortunately, the same page may be assigned different names. I sum the Ocurrences of the two objects with the newly identical url later. If it is better to fix this in App Insights I am grateful for that insight, although I know Bash better than Kusto queries. I am grateful for any help or direction.
Almost. Variables are not automatically expanded within a string. You must interpolate them explicitly with \(…):
jq --arg siteUrl 'docs.mysite.com' '.[].name |= (gsub(" - https://\($siteUrl)/"; ""))' file.json
Alternatively, detect a suffix match and extract the prefix by slicing:
jq --arg siteUrl 'docs.mysite.com' '" - https://\($siteUrl)/" as $suffix | (.[].name | select(endswith($suffix))) |= .[:$suffix|-length]' file.json

jq does not show null output

I have the following code in the command line script:
output_json=$(jq -n \
--argjson ID "${id}" \
--arg Title "${title}" \
--argjson like "\"${like}\"" \
'$ARGS.named')
I put the id, title and like variables into the jq. I get the following output:
[
{
"ID": 6,
"Title": "ABC",
"like": ""
},
{
"ID": 22,
"Title": "ABC",
"like": "Yes"
}
]
But, I am trying to get the output in the following format, i.e. with null:
[
{
"ID": 6,
"Title": "ABC",
"like": null
},
{
"ID": 22,
"Title": "ABC",
"like": "Yes"
}
]
I don't quite get it is it possible to do this in general, or is it a problem with my jq command?
And as far as I understood "like": "" is not the same as "like": null. I am also a little confused now, and do not really understand what is the correct choice to use.
By using --argjson you need to provide valid JSON-encoded argument, thus if you want to receive null the value needs to be literally null. Your solution, however, adds quotes around it, so it can never be evaluated to null. (Also, it will only be a valid JSON string if it follows the JSON encoding for special characters such as the quote characters itself).
If you want to have a JSON string in the regular case, and null in the case where it is empty, import the content of ${like} as string using --arg and without the extra quotes (just as you do with ${title}), then use some jq logic to turn the empty string into null. An if statement would do, for example:
like=
jq -n --arg like "${like}" '{like: (if $like == "" then null else $like end)}'
{
"like": null
}

Invalid numeric literal when passing a truncated JSON object to jq

I have a response from cURL which looks like this:
{"username": "bot", "verified": true, "locale": "en-US", "mfa_enabled": false, "bot": true, "id": "123", "flags": 0, "avatar": null, "discriminator": "3114", "email": null} 200
which is stored in variable called auth
Then I want to be able to loop that object doing this:
response=$(jq -c "." <<< "${auth::-3}")
Note that I remove the last 3 characters because those are the status code.
So technically it should work, but it returns: parse error: Invalid numeric literal at line 1, column 11
If I enter the raw JSON as a string, it works. But not like this. Why?
Consider:
response=$(jq -n --argjson auth "${auth% *}" '$auth')
...which will work correctly with versions of bash too old to correctly support ${auth::-3} (a 4.x-only feature), and which will also log enough details to track down any issue caused by the content passed to jq when run with bash -x yourscript.

how do I pass a bash variable to curl?

I'm having difficulty passing a bash variable to curl. My script is as follows:
now=$(date)
curl --data-binary '{"jsonrpc":"1.0","id":"curltext","method":"importprivkey","params":["test",$now,false]}' -H 'content-type:text/plain;' http://chad:password#127.0.0.1:8332/
$now isn't passing to the curl string. What am I missing?
Variables aren't expanded in single-quoted strings ('), only inside double-quoted strings (") and heredoc are variables expanded.
In your case, swapping the single quotes for double quotes will mean that you'll have to backslash escape all the double quotes of your JSON string. Fear not! You can concatenate strings too! You don't need any special operator like + or . for that, just stop quoting and start a new quoted string:
$ curl [..] '...json...'"$now"'..json..'
Full example:
$ now=$(date)
$ curl \
--data-binary \
'{"jsonrpc":"1.0","id":"curltext","method":"importprivkey","params":["test","'"$now"'",false]}' \
-H 'content-type:text/plain;' \
http://httpbin.org/post
{
"args": {},
"data": "{\"jsonrpc\":\"1.0\",\"id\":\"curltext\",\"method\":\"importprivkey\",\"params\":[\"test\",\"Thu Aug 24 10:10:32 BST 2017\",false]}",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Connection": "close",
"Content-Length": "111",
"Content-Type": "text/plain;",
"Host": "httpbin.org",
"User-Agent": "curl/7.55.1"
},
"json": null,
"origin": "34.194.174.91",
"url": "http://httpbin.org/post"
}
Stuff like this looks rather ugly/unreadable in my opinion. This is a good moment to consider using a shell script with the aforementioned heredoc, or switch to a more structured programming language such as Python. Dealing with JSON in shell scripts has not made many people happy ;-)
SOLVED:
now=$(date)
curl --data-binary '{"jsonrpc":"1.0","id":"curltext","method":"importprivkey","params":['"$var3"','"$now"',false]}' -H 'content-type:text/plain;' http://chad:password#127.0.0.1:8332/

cURL operation performs differently when used in batch script

I'm a bit new to batch scripting, so I apologize if this is glaringly obvious, but I couldn't find any similar information.
I'm trying to perform the following cURL call:
curl -H "Content-Type:application/json" -d '{"lt":"f","sort":"n","max":"1500","offset":"1500"}' [API_KEY]#api.nal.usda.gov/ndb/list
When I run that line in a command line (or Cygwin) it does exactly what I want it to.
However, when I try to call it from a bat file, it seems my parameters are getting messed up somehow.
FOR /L %%A IN (0, 1500, 77500) DO (
curl -H "Content-Type:application/json" -d '{"lt":"f","sort":"n","max":"1500","offset":"%%A"}' [API_KEY]#api.nal.usda.gov/ndb/list > %%A.txt
)
I'm getting output into the correct .txt file, but it doesn't seem like the %%A in offset is getting replaced. I'm getting a "bad parameter" exception from the API. From the output on the command line, it looks accurate.
I'm open to using bash scripting instead if it would make more sense, but I was having the same issue using bash.
(Note: I replaced my API key with a placeholder in the example.. that's not the problem)
In bash at least, the problem is that variable expansion does not occur inside single quotes; you need to use double quotes and escape the nested double quotes:
for a in 0 1500 77500; do
curl -H "Content-Type:application/json" -d "{\"lt\":\"f\",\"sort\":\"n\",\"max\":\"1500\",\"offset\":\"$a\"}" [API_KEY]#api.nal.usda.gov/ndb/list > "$a".txt
)
I suspect you need to do the equivalent in a batch file.
You can concatenate adjacent single- and double-quoted strings to minimized the number of escaped quotes:
... -d '{"lt": "f", "sort": "n", "max": "1500", "offset": "'"$a"'"}' ...
but you may want to consider one of two other options. First, read the data from a here document instead of using a hard-coded string:
curl -H "..." -d#- [API_KEY]#api.nal.usda.gov/ndb/list > "$a".txt <<EOF
{"lt": "f", "sort": "n", "max": "1500", "offset": "$a"}
EOF
or use something like jq to generate the JSON for your:
curl -H "..." \
-d "$(jq --arg a "$a" '{lt: "f", sort: "n", max: "1500", offset: $a}') \
[API_KEY]#api.nal.usda.gov/ndb/list > "$a".txt
The jq solution would be preferable in general, since you don't have to worry about pre-escaping any variable values.

Resources