jq format when running from a bash script with variable expansion - bash

I've got a jq command that works when running directly from the shell or from within a shell script, but when I try to add variable expansion, I get jq errors for unexpected format or invalid characters. My goal is to have a quick and easy way to update some json configuration.
Here's a simplified example.
The format of the json I'm modifying:
{
"pets": {
"some-new-pet": {
"PetInfo": {
"name": "my-brand-new-pet",
"toys": [
"toy1-postfix",
"toy2-postfix",
"toy3-postfix"
]
}
}
}
}
The jq without variable expansion:
cat myfile.json | jq '.pets."some-new-pet" += {PetInfo: {name: "my-brand-new-pet"}, toys: ["toy1", "toy2", "toy3"]}}'
The above runs fine, and adds the new pets.some-new-pet entry to my json.
Below is what I'm trying to do with variable expansion that fails.
jq_args = "'.pets.\"${PET}\" += {PetInfo: {name: \"${NAME}\"}, toys: [\"${toy1}-postfix\", \"${toy2}-postfix\", \"${toy3}-postfix\"]}}'"
cat myfile.json | jq $jq_args
The error message I get with the above:
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at <top-level>, line 1: '.pets."some-new-pet"
My file is formatted as utf-8 and uses LF line endings.

I do not recommend constructing a jq filter using variable expansion or printf. It will work for simple cases but will fail if the string contains double quotes, backslashes or control-codes, as they have special meanings inside a JSON string. As an alternative to using printf, jq has a way to pass in variables directly via the command-line, avoiding all these issues.
pet='some-second-pet'
name='my-even-newer'
toy1=toy1
toy2=toy2
toy3=toy3
jq \
--arg pet "$pet" \
--arg name "$name" \
--arg toy1 "$toy1" \
--arg toy2 "$toy2" \
--arg toy3 "$toy3" \
'.pets[$pet] += {
PetInfo: {name: $name},
toys: ["\($toy1)-postfix", "\($toy2)-postfix", "\($toy3)-postfix"]
}' \
myfile.json
Output:
{
"pets": {
"some-new-pet": {
"PetInfo": {
"name": "my-brand-new-pet",
"toys": [
"toy1-postfix",
"toy2-postfix",
"toy3-postfix"
]
}
},
"some-second-pet": {
"PetInfo": {
"name": "my-even-newer-pet"
},
"toys": [
"toy1-postfix",
"toy2-postfix",
"toy3-postfix"
]
}
}
}

It would be cleaner and less error prone to format the string using printf
PET='dog'
NAME='sam'
toy1="t1"
toy2="t2"
toy3="t3"
jq_args=$(printf '.pets."%s" += {PetInfo: {name: "%s"}, toys: ["%s-postfix", "%s-postfix", "%s-postfix"]}}' "${PET}" "${NAME}" "${toy1}" "${toy2}" "${toy3}")
echo "$jq_args"
Result:
.pets."dog" += {PetInfo: {name: "sam"}, toys: ["t1-postfix", "t2-postfix", "t3-postfix"]}
Additionally, redundant quoting could be avoided by quoting the arg on this command
cat myfile.json | jq "$jq_args"

Fix your jq code by removing extra } at end
Fix bash jq call:
Add cotes "..." around your $jq_args
so don't use singles '...' in your jq_args definition
Use printf with -v option to define jq_args:
printf -v jq_args "...format..." value1 value2 ...
So your code became:
PET="some-new-pet"
NAME="my-brand-new-pet"
toy1="toy1"
toy2="toy2"
toy3="toy3"
format='.pets."%s" += {PetInfo: {name: "%s"}, toys: ["%s", "%s", "%s"]}'
printf -v jq_args "${format}" "${PET}" "${NAME}" "${toy1}" "${toy2}" "${toy3}"
cat myfile.json | jq "$jq_args"
Output:
{
"pets": {
"some-new-pet": {
"PetInfo": {
"name": "my-brand-new-pet"
},
"toys": [
"toy1",
"toy2",
"toy3"
]
}
}
}
Notes:
When you define your format, you put it into simple cotes '...'. It's really better to format JSON (or XML) without back-slashes (\\) before each double cotes (")
Use printf -v variable_name. It's more readable than var_name=$(printf ...)

By constructing the jq filter ("code") using outer bash variables ("data") you may run into escaping issues, which could eventually break or even divert your filter. (see https://en.wikipedia.org/wiki/Code_injection)
Instead, use mechanisms by jq to introduce external data through variables (parameter --arg):
jq --arg pet "${PET}" \
--arg name "${NAME}" \
--arg toy1 "${toy1}-postfix" \
--arg toy2 "${toy2}-postfix" \
--arg toy3 "${toy3}-postfix" \
'
.pets[$pet] += {PetInfo: {$name, toys: [$toy1,$toy2,$toy3]}}
' myfile.json
If you have an unknown number of variables to include, check out jq's --args parameter (note the additional s)

Related

How to interpolate several values contained in a variable into a json string

I have a parsed variable obtained after parsing some text:
parsed=$(echo "PA-232 message1 GX-1234 message2 PER-10 message3" | grep -Eo '[A-Z]+-[0-9]+')
parsed contains a bunch of ids:
echo $parsed
PA-232
GX-1234
PER-10
The next thing I have to do in my script is generate a json text and invoke an API with it:
The json text should be
"{\"tasks\": [{\"taskId\": \"PA-232\"}, {\"taskId\": \"GX-1234\"}, {\"taskId\": \"PER-10\"}], \"projectId\": \"$CI_PROJECT_ID\" }"
Notice CI_PROJECT_ID is an envvar that I also have to send, thats why I needed to use double quotes and escape them.
And it would be called with curl:
curl -X POST -H 'Content-Type:application/json' -k -u $CLIENT_ID:$CLIENT_SECRET 'https://somewhere.com/api/tasks' -d "{\"tasks\": [{\"taskId\": \"PA-232\"}, {\"taskId\": \"GX-1234\"}, {\"taskId\": \"PER-10\"}], \"projectId\": \"$CI_PROJECT_ID\"}"
The question is how can I generate a json string like the one shown above from the parsed variable and the additional envvar?
How about doing it with jq?
CI_PROJECT_ID='I want this " to be escaped automatically'
echo 'PA-232 message1 GX-1234 message2 PER-10 message3' |
jq -R --arg ciProjectId "$CI_PROJECT_ID" '
{
tasks: [
capture( "(?<taskId>[[:upper:]]+-[[:digit:]]+)"; "g" )
],
projectId: $ciProjectId
}
'
{
"tasks": [
{
"taskiD": "PA-232"
},
{
"taskiD": "GX-1234"
},
{
"taskiD": "PER-10"
}
],
"projectId": "I want this \" to be escaped automatically"
}
note: you can use jq -c ... for outputting a compact JSON
And here's a solution without jq that doesn't escape the characters in the strings so it might generate invalid JSON:
CI_PROJECT_ID='no escaping needed'
tasks_jsonArr=$(
echo "PA-232 message1 GX-1234 message2 PER-10 message3" |
grep -Eo '[A-Z]+-[0-9]+' |
sed 's/.*/{ "taskiD": "&" }/' |
paste -sd ',' |
sed 's/.*/[ & ]/'
)
curl -k 'https://somewhere.com/api/tasks' \
-X POST \
-H 'Content-Type:application/json' \
-u "$CLIENT_ID:$CLIENT_SECRET" \
-d "{\"tasks\": $tasks_jsonArr, \"projectId\": \"$CI_PROJECT_ID\"}"
N.B. For JSON-escaping strings with standard tools, take a look at function json_stringify in awk

jq: Append JSON object via shell variable

I have a JSON file and I'm trying to add another field to it.
Example JSON File:
{"data":{}}
Looking at other answers the += seems to work:
objectName="objName"
cat $jsonFile | jq --arg objName $objectName '.data[$objName] += {"test": "json"}'
Outputs
{
"data": {
"objName": {
"test": "json"
}
}
}
just as expected.
The problem is that I can't hardcode the JSON so I'm inputting the string as a variable. But I can't get the syntax to work:
objectName="objName"
objJSON='{"test": "json"}'
cat $jsonFile | jq -r --arg objName $objectName --arg jsonString $objJSON '.data[$objName] += $jsonString'
I'm getting the error jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at <top-level>, line 1: "json"}
Use --argjson rather than --arg when passing JSON rather than strings.
When in doubt, quote your strings; that way if the shell executing your code isn't the one you think it is, you won't have your values being munged. (It's also helpful to stay in the habits necessary to write portable code; the whole reason I dropped zsh after spending 6 months learning it back in the mid-2000s is that my code quality when writing for other shells suffered).
objName="objName"
objJSON='{"test": "json"}'
echo '{"data":{}}' |
jq --arg objectName "$objName" \
--argjson jsonString "$objJSON" \
'.data[$objectName] += $jsonString'
...properly emits:
{
"data": {
"objName": {
"test": "json"
}
}
}

how to build json without escaping new line?

I run jq from bash and all my new lines are escaped
release_message="\`\`\`a\na\n\`\`\`"
query=$(jq -n \
--arg message $release_message \
"{text:\$message}"
);
echo "query $query"
result
query {
"text": "```a\\na\\n```"
}
How to prevent extra escape from jq?
You can either encode your input in JSON yourself or let jq do it
option 1 : encode yourself
release_message='"```a\na\n```"'
jq -n --argjson message "$release_message" '{text:$message}'
# or :
# echo "$release_message" | jq '{text:.}'
Have bash produce a valid JSON string (note : quotes-enclosed), pass through the standard input or with --argjson.
option 2 : let jq encode the string
release_message='```a
a
```'
jq --arg message "$release_message" '{text:$message}'
# or :
# echo "$release_message" | jq -R --slurp '{text:.}'
Have bash produce the literal string, pass with --arg or specify --raw-input/-R to have the input encoded in JSON, plus --slurp so that the multiple lines are considered as a single string.
Since you're using bash, it's generally best to use single quotes unless you want string interpolation. Consider, for example:
release_message='```a\na\n```'
query=$(jq -n \
--arg message "$release_message" \
'{text: $message }'
);
echo "query $query"
You might want to consider using $'....':
release_message=$'```a\na\n```'
Using gsub
Depending on what your actual goals are, you might want to use gsub, e.g.
release_message='```a\na\n```'
query=$(jq -n \
--arg message "$release_message" \
'{text: ($message | gsub("\\n";"\n")) }'
);
echo "query $query"
produces:
query {
"text": "```a\\na\\n```"
}

Replace variable in read JSON data in Shell Script [duplicate]

test.sh is not replacing test.json parameter values ($input1 and $input2). result.json has same ParameterValue "$input1/solution/$input2.result"
[
{
"ParameterKey": "Project",
"ParameterValue": [ "$input1/solution/$input2.result" ]
}
]
test.sh
#!/bin/bash
input1="test1"
input2="test2"
echo $input1
echo $input2
cat test.json | jq 'map(if .ParameterKey == "Project" then . + {"ParameterValue" : "$input1/solution/$input2.result" } else . end )' > result.json
shell variables in jq scripts should be interpolated or passed as arguments via --arg name value:
jq --arg inp1 "$input1" --arg inp2 "$input2" \
'map(if .ParameterKey == "Project"
then . + {"ParameterValue" : ($inp1 + "/solution/" + $inp2 + ".result") }
else . end)' test.json
The output:
[
{
"ParameterKey": "Project",
"ParameterValue": "test1/solution/test2.result"
}
]
In your jq program, you have quoted "$input1/solution/$input2.result", and therefore it is a JSON string literal, whereas you evidently want string interpolation; you also need to distinguish between the shell variables ($input1 and $input2) on the one hand, and the corresponding jq dollar-variables (which may or may not have the same name) on the other.
Since your shell variables are strings, you could pass them in using the --arg command-line option (e.g. --arg input1 "$input1" if you chose to name the variables in the same way).
You can read up on string interpolation in the jq manual (see https://stedolan.github.io/jq/manual, but note the links at the top for different versions of jq).
There are other ways to achieve the desired results too, but using string interpolation with same-named variables, you'd write:
"\($input1)/solution/\($input2).result"
Notice that the above string is NOT itself literally a JSON string. Only after string interpolation does it become so.

How to replace a variable inside a string in bash

I have a string env variable which looks like below
data={\"data\":{\"sources\":\"some value\", \"destination\":\"some other value\"}}
I would like to include date (say YEAR) within this env variable. That is, I have another env variable called YEAR (bash: YEAR=2019) and I would like to use this variable (YEAR) inside data. Here is what I need to do
data={\"data\":{\"sources\":\"some value ${YEAR}\", \"destination\":\"some other value\"}}
but it does not work, how can I make it work?
Use jq:
$ echo "$data" | jq --argjson y "$YEAR" '.data.sources += " \($y)"'
{
"data": {
"sources": "some value 2019",
"destination": "some other value"
}
}
or
# Note the -c argument to compress the data to a single line
$ data=$(echo "$data" | jq -c --argjson y "$YEAR" '.data.sources += " \($y)"')
$ echo "$data"
{"data":{"sources":"some value 2019","destination":"some other value"}}
Alternative, using here documents, minimizing the need to escape quotes, while still supporting variable substitutions:
data=$(cat <<DATA
"data": {
"sources":"some value ${YEAR}",
"destination":"some other value"
}
}
DATA
)

Resources