jq: Append JSON object via shell variable - macos

I have a JSON file and I'm trying to add another field to it.
Example JSON File:
{"data":{}}
Looking at other answers the += seems to work:
objectName="objName"
cat $jsonFile | jq --arg objName $objectName '.data[$objName] += {"test": "json"}'
Outputs
{
"data": {
"objName": {
"test": "json"
}
}
}
just as expected.
The problem is that I can't hardcode the JSON so I'm inputting the string as a variable. But I can't get the syntax to work:
objectName="objName"
objJSON='{"test": "json"}'
cat $jsonFile | jq -r --arg objName $objectName --arg jsonString $objJSON '.data[$objName] += $jsonString'
I'm getting the error jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at <top-level>, line 1: "json"}

Use --argjson rather than --arg when passing JSON rather than strings.
When in doubt, quote your strings; that way if the shell executing your code isn't the one you think it is, you won't have your values being munged. (It's also helpful to stay in the habits necessary to write portable code; the whole reason I dropped zsh after spending 6 months learning it back in the mid-2000s is that my code quality when writing for other shells suffered).
objName="objName"
objJSON='{"test": "json"}'
echo '{"data":{}}' |
jq --arg objectName "$objName" \
--argjson jsonString "$objJSON" \
'.data[$objectName] += $jsonString'
...properly emits:
{
"data": {
"objName": {
"test": "json"
}
}
}

Related

jq format when running from a bash script with variable expansion

I've got a jq command that works when running directly from the shell or from within a shell script, but when I try to add variable expansion, I get jq errors for unexpected format or invalid characters. My goal is to have a quick and easy way to update some json configuration.
Here's a simplified example.
The format of the json I'm modifying:
{
"pets": {
"some-new-pet": {
"PetInfo": {
"name": "my-brand-new-pet",
"toys": [
"toy1-postfix",
"toy2-postfix",
"toy3-postfix"
]
}
}
}
}
The jq without variable expansion:
cat myfile.json | jq '.pets."some-new-pet" += {PetInfo: {name: "my-brand-new-pet"}, toys: ["toy1", "toy2", "toy3"]}}'
The above runs fine, and adds the new pets.some-new-pet entry to my json.
Below is what I'm trying to do with variable expansion that fails.
jq_args = "'.pets.\"${PET}\" += {PetInfo: {name: \"${NAME}\"}, toys: [\"${toy1}-postfix\", \"${toy2}-postfix\", \"${toy3}-postfix\"]}}'"
cat myfile.json | jq $jq_args
The error message I get with the above:
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at <top-level>, line 1: '.pets."some-new-pet"
My file is formatted as utf-8 and uses LF line endings.
I do not recommend constructing a jq filter using variable expansion or printf. It will work for simple cases but will fail if the string contains double quotes, backslashes or control-codes, as they have special meanings inside a JSON string. As an alternative to using printf, jq has a way to pass in variables directly via the command-line, avoiding all these issues.
pet='some-second-pet'
name='my-even-newer'
toy1=toy1
toy2=toy2
toy3=toy3
jq \
--arg pet "$pet" \
--arg name "$name" \
--arg toy1 "$toy1" \
--arg toy2 "$toy2" \
--arg toy3 "$toy3" \
'.pets[$pet] += {
PetInfo: {name: $name},
toys: ["\($toy1)-postfix", "\($toy2)-postfix", "\($toy3)-postfix"]
}' \
myfile.json
Output:
{
"pets": {
"some-new-pet": {
"PetInfo": {
"name": "my-brand-new-pet",
"toys": [
"toy1-postfix",
"toy2-postfix",
"toy3-postfix"
]
}
},
"some-second-pet": {
"PetInfo": {
"name": "my-even-newer-pet"
},
"toys": [
"toy1-postfix",
"toy2-postfix",
"toy3-postfix"
]
}
}
}
It would be cleaner and less error prone to format the string using printf
PET='dog'
NAME='sam'
toy1="t1"
toy2="t2"
toy3="t3"
jq_args=$(printf '.pets."%s" += {PetInfo: {name: "%s"}, toys: ["%s-postfix", "%s-postfix", "%s-postfix"]}}' "${PET}" "${NAME}" "${toy1}" "${toy2}" "${toy3}")
echo "$jq_args"
Result:
.pets."dog" += {PetInfo: {name: "sam"}, toys: ["t1-postfix", "t2-postfix", "t3-postfix"]}
Additionally, redundant quoting could be avoided by quoting the arg on this command
cat myfile.json | jq "$jq_args"
Fix your jq code by removing extra } at end
Fix bash jq call:
Add cotes "..." around your $jq_args
so don't use singles '...' in your jq_args definition
Use printf with -v option to define jq_args:
printf -v jq_args "...format..." value1 value2 ...
So your code became:
PET="some-new-pet"
NAME="my-brand-new-pet"
toy1="toy1"
toy2="toy2"
toy3="toy3"
format='.pets."%s" += {PetInfo: {name: "%s"}, toys: ["%s", "%s", "%s"]}'
printf -v jq_args "${format}" "${PET}" "${NAME}" "${toy1}" "${toy2}" "${toy3}"
cat myfile.json | jq "$jq_args"
Output:
{
"pets": {
"some-new-pet": {
"PetInfo": {
"name": "my-brand-new-pet"
},
"toys": [
"toy1",
"toy2",
"toy3"
]
}
}
}
Notes:
When you define your format, you put it into simple cotes '...'. It's really better to format JSON (or XML) without back-slashes (\\) before each double cotes (")
Use printf -v variable_name. It's more readable than var_name=$(printf ...)
By constructing the jq filter ("code") using outer bash variables ("data") you may run into escaping issues, which could eventually break or even divert your filter. (see https://en.wikipedia.org/wiki/Code_injection)
Instead, use mechanisms by jq to introduce external data through variables (parameter --arg):
jq --arg pet "${PET}" \
--arg name "${NAME}" \
--arg toy1 "${toy1}-postfix" \
--arg toy2 "${toy2}-postfix" \
--arg toy3 "${toy3}-postfix" \
'
.pets[$pet] += {PetInfo: {$name, toys: [$toy1,$toy2,$toy3]}}
' myfile.json
If you have an unknown number of variables to include, check out jq's --args parameter (note the additional s)

How to pass environment bash variables into json using jq [duplicate]

I want to populate json template with value "Hello Jack", but the "Hello" part shall remain inside of template, is there are any way of doing that, I've tried code below but it gives me error:
jq -n --arg person "Jack" '{my_key: "Hello "$person}'
jq: error: syntax error, unexpected '$', expecting '}' (Unix shell quoting issues?) at <top-level>, line 1:
Use string interpolation syntax like so:
jq -n --arg person Jack '{my_key: "Hello \($person)"}'
And to load the template from a file, use the -f switch:
$ cat template.json
{
"my_key": "Hello \($person)"
}
$ jq -n --arg person Jack -f template.json
{
"my_key": "Hello Jack"
}

Not able to get value from json using jq when I replace one of the elements with variable

Using jq if I try to get the value of an element using string I am able to get the value. But to make it flexible if I replace it with a variable then it throws error.
data=$(cat <<EOF
{
"my_db_instance_class": {
"sensitive": false,
"type": "string",
"value": "db.t3.medium"
},
"my_db_instance_test": {
"sensitive": false,
"type": "string",
"value": "db.t4.medium"
}
}
EOF
)
#echo $data
keys=$(echo $data | jq -r '. |=keys' | jq '.[]')
#echo $keys
for key in $keys
do
value=$(echo $data | jq --arg k "$key" '.my_db_instance_class.value')
final_value="${final_value}${key}: ${value}"
done
The above code works but it not very static. Below I have replaced my_db_instance_class with a variable
for key in $keys
do
value=$(echo $data | jq --arg k "$key" '.$k.value')
final_value="${final_value}${key}: ${value}"
done
Error message
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.$k.value
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.$k.value
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.$k.value
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.$k.value
jq: 2 compile errors
Expected final output of the whole script
"my_db_instance_class": "db.t3.medium""my_db_instance_test":
"db.t3.medium"
It seems like you're first trying to get all the keys, then loop over then to show the key with the relevant value.
We can simplify this using just jq like so:
to_entries[] | "\(.key): \(.value.value)"
This will produce the following output:
my_db_instance_class: db.t3.medium
my_db_instance_test: db.t4.medium
JqPlay Demo
to_entires documentation
String interpolation (\()) documentation

jq produces `is not defined at <top-level>` error

I'm seeing a is not defined at <top-level> when calling jq like so:
jq ".Changes[0].ResourceRecordSet.Name = word-is-here.domain.com" someFile.json
The error repeats for each word separated by a dash in the second side of the replacement. The full error is like
jq: error: word/0 is not defined at <top-level>, line 1:
.Changes[0].ResourceRecordSet.Name = word-is-here.domain.com
I've tried escaping quotes in many different ways but that didn't help. (what I mean by this is doing "'"'" weird stuff, I'm still learning bash so I'm just trowing stuff at the wall until it sticks)
EDIT:
So I'm trying to run this in a bash script, and both side of the = signs are variables such as jq --arg value "$value" --arg key "$key" '$key = $value' "$path" (what I tried after a suggestion)
and got the error:
Invalid path expression with result ".Changes[0].ResourceRecor...
The json I'm using is as such:
{
"Changes": [
{
"Action": "do something",
"ResourceRecordSet": {
"Name": "some name here to replace",
...
}
}
]
}
jq '.Changes[0].ResourceRecordSet.Name = "word-is-here.domain.com"' file.json
Quote the string you are assigning. Or pass it to jq via an argument:
jq --arg foo 'words-here' '.Changes[0].ResourceRecordSet.Name = $foo' file.json
For passing the path to the key you want as an argument, a suggestion from https://github.com/stedolan/jq/issues/1493 might work:
jq --argjson path '["Changes",0,"ResourceRecordSet","Name"]' \
--arg val 'word-is-here.domain.com' \
'getpath($path) = $val' file.json
The problem (or at least the obvious problem) here is evidently the string: word-is-here.domain.com, since jq is interpreting the dash ("-") as an operation ("minus").
Unfortunately, since you haven't given us many clues, it's not completely clear what specifically needs to be changed, but a reasonable guess is that word-is-here.domain.com is intended as a fixed string. If so, you would have to present it as a JSON string. So in a bash or bash-like environment, you could write:
jq '.Changes[0].ResourceRecordSet.Name = "word-is-here.domain.com"' someFile.json
Specifying the LHS path via a shell variable
If the LHS path must be specified by a shell variable, it should if possible be passed in as a JSON array, e.g. using the --argjson command-line option; one can then use an expression of the form setpath($path; $value) to update the path.
If for some reason a solution allowing the LHS to be specified as a jq path is preferred, then shell string-interpolation could be used, though as with any such interpolation, this should be done with care.

How to use key variables to get a second level value of json file using jq

I need to get a specific value from a json file using jq in a bash script (busybox). The json file looks like this:
{
"example.com": {
"backend": "1.3.7"
}
}
In my script there are two variables: project and app - as there are of course multiple projects and applications. I need to use these variable values to get the version value. In this example project is "example.com" and app is "backend"
I tried this
jq --arg p "$project" --arg a "$app" '.[$p].[$a]' file.json
But I do get the error
jq: error: syntax error, unexpected '[', expecting FORMAT or QQSTRING_START (Unix shell quoting issues?) at <top-level>, line 1:
.[$p].[$a]
jq: 1 compile error
You may use it like this:
jq -r --arg p "$project" --arg a "$app" '.[$p][$a]' file.json
or else:
jq -r --arg p "$project" --arg a "$app" '.[$p] | .[$a]' file.json
1.3.7

Resources