I have a json string like this,
rssample='{ "Changes": [{"Action": "UPSERT","ResourceRecordSet": {"ResourceRecords":[{ "Value":""}], "Type": "TXT","NAME":"","TTL": 300}}]}'
I want to update the NAME and Value in it using the below variables..
name="testname"
newvalue="heritage=external-dns,external-dns/owner=us-east-1:sandbox-newtestowner,external-dns/resource=ingress/monitoring/prometheus-operator-alertmanager"
So I tried using sed,
newrs=$(sed -E "s/"NAME":""/"NAME":"$name"/g" <<< "$rssample")
newrs1=$(sed -E "s/"Value":""/"Value":"\$newvalue"/g" <<< "$newrs")
I am expecting below as output,
{ "Changes": [{"Action": "UPSERT","ResourceRecordSet": {"ResourceRecords":[{ "Value":"\"heritage=external-dns,external-dns/owner=us-east-1:sandbox-newtestowner,external-dns/resource=ingress/monitoring/prometheus-operator-alertmanager\""}], "Type": "TXT","NAME":"testname","TTL": 300}}]}
But I am empty Name.And getting error for value as,
sed: 1: "s/Value:/Value:"heritag ...": bad flag in substitute command: 'o'
My output is,
{ "Changes": [{"Action": "UPSERT","ResourceRecordSet": {"ResourceRecords":[{ "Value":""}], "Type": "TXT","NAME":"","TTL": 300}}]}
Please let me know how to fix this? is using sed good idea or jq?
It's generally safer, and therefore often better, to use a JSON-aware tool rather than sed when editing JSON. Using jq, one possibility would be:
jq --arg name "$name" --arg newvalue "$newvalue" '
.Changes[0].ResourceRecordSet |=
(.NAME=$name
| .ResourceRecords[0].Value = $newvalue)' <<< "$rssample"
A free-form approach
jq --arg name "$name" --arg newvalue "$newvalue" '
walk(if type == "object"
then if has("NAME") then .NAME=$name else . end
| if has("Value") then .Value = $newvalue else . end
else . end)' <<< "$rssample"
Related
I have json file which extract the color value from the file. For some reason, it only fetch only one block of code & for the rest it throws error.
snippet
#!/bin/bash
clear
echo "Add the figma json file path"
read path
figma_json="$(echo -e "${path}" | tr -d '[:space:]')"
echo "*****************************************"
color_values=$(cat $figma_json | jq -r '.color')
color_keys=$(cat $figma_json | jq -r '.color | keys' |sed 's,^ *,,; s, *$,,'| tr -s ' ' | tr ' ' '_')
echo $color_keys
for c_key in $color_keys
do
echo "key string: $c_key"
echo $color_values | jq ".$c_key.value"
echo "*********************************************"
done
Output
trimmed string: "gray1",
{
"description": "",
"type": "color",
"value": "#333333ff",
"extensions": {
"org.lukasoppermann.figmaDesignTokens": {
"styleId": "S:0b49d19e868ec919fac01ec377bb989174094d7e,",
"exportKey": "color"
}
}
}
null
*********************************************
trimmed string: "gray2" //Expected output
"#333333ff"
*********************************************
If we look at the second output it prints the hex value of gray2 which is the expected output
Please use the follow link to get the json file
link
It's quite unclear what you are aiming at, but here's one way how you would read from a JSON file using just one call to jq, and most probably without the need to employ sed or tr. The selection as well as the formatting can easily be adjusted to your liking.
jq -r '.color | to_entries[] | "\(.key): \(.value.value)"' "$figma_json"
gray1: #333333ff
gray2: #4f4f4fff
gray3: #828282ff
gray4: #bdbdbdff
gray5: #e0e0e0ff
gray6: #f2f2f2ff
red: #eb5757ff
orange: #f2994aff
yellow: #f2c94cff
green1: #219653ff
green2: #27ae60ff
green3: #6fcf97ff
blue1: #2f80edff
blue2: #2d9cdbff
blue3: #56ccf2ff
purple1: #9b51e0ff
purple2: #bb6bd9ff
Demo
I've got a jq command that works when running directly from the shell or from within a shell script, but when I try to add variable expansion, I get jq errors for unexpected format or invalid characters. My goal is to have a quick and easy way to update some json configuration.
Here's a simplified example.
The format of the json I'm modifying:
{
"pets": {
"some-new-pet": {
"PetInfo": {
"name": "my-brand-new-pet",
"toys": [
"toy1-postfix",
"toy2-postfix",
"toy3-postfix"
]
}
}
}
}
The jq without variable expansion:
cat myfile.json | jq '.pets."some-new-pet" += {PetInfo: {name: "my-brand-new-pet"}, toys: ["toy1", "toy2", "toy3"]}}'
The above runs fine, and adds the new pets.some-new-pet entry to my json.
Below is what I'm trying to do with variable expansion that fails.
jq_args = "'.pets.\"${PET}\" += {PetInfo: {name: \"${NAME}\"}, toys: [\"${toy1}-postfix\", \"${toy2}-postfix\", \"${toy3}-postfix\"]}}'"
cat myfile.json | jq $jq_args
The error message I get with the above:
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at <top-level>, line 1: '.pets."some-new-pet"
My file is formatted as utf-8 and uses LF line endings.
I do not recommend constructing a jq filter using variable expansion or printf. It will work for simple cases but will fail if the string contains double quotes, backslashes or control-codes, as they have special meanings inside a JSON string. As an alternative to using printf, jq has a way to pass in variables directly via the command-line, avoiding all these issues.
pet='some-second-pet'
name='my-even-newer'
toy1=toy1
toy2=toy2
toy3=toy3
jq \
--arg pet "$pet" \
--arg name "$name" \
--arg toy1 "$toy1" \
--arg toy2 "$toy2" \
--arg toy3 "$toy3" \
'.pets[$pet] += {
PetInfo: {name: $name},
toys: ["\($toy1)-postfix", "\($toy2)-postfix", "\($toy3)-postfix"]
}' \
myfile.json
Output:
{
"pets": {
"some-new-pet": {
"PetInfo": {
"name": "my-brand-new-pet",
"toys": [
"toy1-postfix",
"toy2-postfix",
"toy3-postfix"
]
}
},
"some-second-pet": {
"PetInfo": {
"name": "my-even-newer-pet"
},
"toys": [
"toy1-postfix",
"toy2-postfix",
"toy3-postfix"
]
}
}
}
It would be cleaner and less error prone to format the string using printf
PET='dog'
NAME='sam'
toy1="t1"
toy2="t2"
toy3="t3"
jq_args=$(printf '.pets."%s" += {PetInfo: {name: "%s"}, toys: ["%s-postfix", "%s-postfix", "%s-postfix"]}}' "${PET}" "${NAME}" "${toy1}" "${toy2}" "${toy3}")
echo "$jq_args"
Result:
.pets."dog" += {PetInfo: {name: "sam"}, toys: ["t1-postfix", "t2-postfix", "t3-postfix"]}
Additionally, redundant quoting could be avoided by quoting the arg on this command
cat myfile.json | jq "$jq_args"
Fix your jq code by removing extra } at end
Fix bash jq call:
Add cotes "..." around your $jq_args
so don't use singles '...' in your jq_args definition
Use printf with -v option to define jq_args:
printf -v jq_args "...format..." value1 value2 ...
So your code became:
PET="some-new-pet"
NAME="my-brand-new-pet"
toy1="toy1"
toy2="toy2"
toy3="toy3"
format='.pets."%s" += {PetInfo: {name: "%s"}, toys: ["%s", "%s", "%s"]}'
printf -v jq_args "${format}" "${PET}" "${NAME}" "${toy1}" "${toy2}" "${toy3}"
cat myfile.json | jq "$jq_args"
Output:
{
"pets": {
"some-new-pet": {
"PetInfo": {
"name": "my-brand-new-pet"
},
"toys": [
"toy1",
"toy2",
"toy3"
]
}
}
}
Notes:
When you define your format, you put it into simple cotes '...'. It's really better to format JSON (or XML) without back-slashes (\\) before each double cotes (")
Use printf -v variable_name. It's more readable than var_name=$(printf ...)
By constructing the jq filter ("code") using outer bash variables ("data") you may run into escaping issues, which could eventually break or even divert your filter. (see https://en.wikipedia.org/wiki/Code_injection)
Instead, use mechanisms by jq to introduce external data through variables (parameter --arg):
jq --arg pet "${PET}" \
--arg name "${NAME}" \
--arg toy1 "${toy1}-postfix" \
--arg toy2 "${toy2}-postfix" \
--arg toy3 "${toy3}-postfix" \
'
.pets[$pet] += {PetInfo: {$name, toys: [$toy1,$toy2,$toy3]}}
' myfile.json
If you have an unknown number of variables to include, check out jq's --args parameter (note the additional s)
I'm trying to create 2 variables via bash $lat, $long base on the result of my curl response.
curl ipinfo.io/33.62.137.111 | grep "loc" | awk '{print $2}'
I got.
"42.6334,-71.3162",
I'm trying to get
$lat=42.6334
$long=-71.3162
Can someone give me a little push ?
IFS=, read -r lat long < <(
curl -s ipinfo.io/33.62.137.111 |
jq -r '.loc'
)
printf 'Latitude is: %s\nLongitude is: %s\n' "$lat" "$long"
The ipinfo.io API is returning JSON data, so let parse it with jq:
Here is the JSON as returned by the query from your sample:
{
"ip": "33.62.137.111",
"city": "Columbus",
"region": "Ohio",
"country": "US",
"loc": "39.9690,-83.0114",
"postal": "43218",
"timezone": "America/New_York",
"readme": "https://ipinfo.io/missingauth"
}
We are going to JSON query the loc entry from the main root object ..
curl -s ipinfo.io/33.62.137.111: download the JSON data -s silently without progress.
jq -r '.loc': Process JSON data, query the loc entry of the main object and -r output raw string.
IFS=, read -r lat long < <(: Sets the Internal Field Separator to , and read both lat and long variables from the following command group output stream.
Although the answer from #LeaGris is quite interesting, if you don't want to use an external library or something, you can try this:
Playground: https://repl.it/repls/ThoughtfulImpressiveComputer
coordinates=($(curl ipinfo.io/33.62.137.111 | sed 's/ //g' | grep -P '(?<=\"loc\":").*?(?=\")' -o | tr ',' ' '))
echo "${coordinates[#]}"
echo ${coordinates[0]}
echo ${coordinates[1]}
Example output:
39.9690 -83.0114 # echo "${coordinates[#]}"
39.9690 # ${coordinates[0]}
-83.0114 # ${coordinates[1]}
Explanation:
curl ... get the JSON data
sed 's/ //g' remove all spaces
grep -P ... -o
-P interpret the given pattern as a perl regexp
(?<=\"loc\":").*?(?=\")
(?<=\"loc\":") regex lookbehind
.*? capture the longitude and latitude part with non-greedy search
(?=\") regex lookahead
-o get only the matching part which'ld be e.g. 39.9690,-83.0114
tr ',' ' ' replace , with space
Finally we got something like this: 39.9690 -83.0114
Putting it in parentheses lets us create an array with two values in it (cf. ${coordinates[...]}).
I run jq from bash and all my new lines are escaped
release_message="\`\`\`a\na\n\`\`\`"
query=$(jq -n \
--arg message $release_message \
"{text:\$message}"
);
echo "query $query"
result
query {
"text": "```a\\na\\n```"
}
How to prevent extra escape from jq?
You can either encode your input in JSON yourself or let jq do it
option 1 : encode yourself
release_message='"```a\na\n```"'
jq -n --argjson message "$release_message" '{text:$message}'
# or :
# echo "$release_message" | jq '{text:.}'
Have bash produce a valid JSON string (note : quotes-enclosed), pass through the standard input or with --argjson.
option 2 : let jq encode the string
release_message='```a
a
```'
jq --arg message "$release_message" '{text:$message}'
# or :
# echo "$release_message" | jq -R --slurp '{text:.}'
Have bash produce the literal string, pass with --arg or specify --raw-input/-R to have the input encoded in JSON, plus --slurp so that the multiple lines are considered as a single string.
Since you're using bash, it's generally best to use single quotes unless you want string interpolation. Consider, for example:
release_message='```a\na\n```'
query=$(jq -n \
--arg message "$release_message" \
'{text: $message }'
);
echo "query $query"
You might want to consider using $'....':
release_message=$'```a\na\n```'
Using gsub
Depending on what your actual goals are, you might want to use gsub, e.g.
release_message='```a\na\n```'
query=$(jq -n \
--arg message "$release_message" \
'{text: ($message | gsub("\\n";"\n")) }'
);
echo "query $query"
produces:
query {
"text": "```a\\na\\n```"
}
I'm using jq to read some data from a JSON file.
after=`cat somefile.json | jq '.after[]'`
returns something like this:
"some value" "another value" "something else"
Basically a list of quoted strings. I now need to convert these strings into one string formatted like
"some value; another value; something else;"
I've tried a lot of combinations of for loops to try and get this working and nothing quite works.
Anyone know how this can be done? Cheers!
use sed:
sed -e 's/" /; /g; s/ "/ /g; s/"$/;"/' <<< '"some value" "another value" "something else"'
OUTPUT:
"some value; another value; something else;"
use sed s command for replacing the desire value
Thanks all! I actually decided to dig deeper into the jq docs to see if I could simply leverage it to do what I want.
after=`cat somefile.json | jq -c -r '.after[] + "; "'` | tr -d '\n'
This ended up working very well. Thanks for the sed version though! Always good to see another working solution.
Assuming .after[] returns the list of strings you describe, you can do this entirely with jq using join to format them as follows:
[ .after[] ] | join("; ") + ";"