How to use sed with jq to replace _ in key name with symbol a
{ "product_name":"kl" }
should become
{ "productaname":"kl" }
in a bash script
No need for sed; it's easy to do in just jq:
$ jq '{ productaname: .product_name }' <<<'{ "product_name":"kl" }'
{"productaname":"kl"}
If you want to replace underscores with a's in all keys of an object:
$ jq 'with_entries(.key |= gsub("_"; "a"))' <<<'{ "product_name":"kl", "foo_bar":12 }'
{"productaname":"kl","fooabar":12}
From the documentation for with_entries:
to_entries, from_entries, with_entries
These functions convert between an object and an array of key-value pairs. If to_entries is passed an object, then for each k: v entry in the input, the output array includes {"key": k, "value": v}.
from_entries does the opposite conversion, and with_entries(foo) is a shorthand for to_entries | map(foo) | from_entries, useful for doing some operation to all keys and values of an object. from_entries accepts key, Key, name, Name, value and Value as keys.
Related
I am new to jq and facing an issue while parsing my json
I have a json stored in a variable like this
temp='{ "1": { "my_name": "one" }, "2": { "my_name": "two" } }'
Now I need to get the value of my_name for both other entries
I have tried something like this
echo $temp | jq '.1' //out put 0.1
I was assuming to get { "my_name": "one" }
And similarly to get my_name value I did
echo $temp | jq '.1.my_name' // Its output is giving me error
Can any one please help to identify what is wrong with my syntax and how can I correct it.
Just the number gets interpreted as a float. You need to use it in a context where it is unambiguously a key string.
echo "$temp" | jq '.["1"]["my_name"]'
and
echo "$temp" | jq '.["1"]'
to get the containing dict.
With a sufficiently new jq (I think >= 1.4) you can also say
echo "$temp" | jq '."1".my_name'
Whenever you are trying to reference a key that is not a valid identifier, you have to quote it. See the manual for more details.
To select the item under the key 1, you'd do this:
."1"
For your other question on how to obtain the my_name values, you could do this:
to_entries | map(.value.my_name)
I am reading aws ssm parameters via jq with json, and want to put them into a yaml file after some processing and alias mapping.
I have one associative array of mapping parameter name to some internal name (see process.sh)
My simplified example.json is
{
"Parameters": [
{
"Name": "/dev/applications/web/some_great_key",
"Value": "magicvaluenotrelevant"
},
{
"Name": "/dev/applications/web/api_magic",
"Value": "blabla"
}
]
}
My bash script process.sh is
#!/usr/bin/env bash
declare -A ssmMap
ssmMap[/dev/applications/web/some_great_key]=theGreatKEYAlias
ssmMap[/dev/applications/web/api_magic]=apiKey
jq -r '
.Parameters
| map({
Name: .Name,
parameterValue: .Value
})
| map((${!ssmMap[.Name]} + ": \"" + .parameterValue + "\""))
| join("\n")
' < example.json;
I want/expected the output to be:
ssm_theGreatKEYAlias: "magicvaluenotrelevant"
ssm_apiKey: "singleblabla"
With the process.sh script that I provided I get error because cannot find a way to use the associative array sshMap inside jq map, to transform the parameter name from json into the mapped alias from sshMap.
Any idea how that can be achieved ?
If I change the line 13 of process.sh into
| map((.Name + ": \"" + .parameterValue + "\""))
it works fine but without mapping, uses original parameter name as comes from json file.
You're trying to access a shell var from jq.
Fixed:
ssm_map='
{
"some_great_key": "theGreatKEYAlias",
"api_magic": "apiKey"
}
'
jq -r --argjson ssm_map "$ssm_map" '
.Parameters[] |
"ssm_\( $ssm_map[ .Name ] ): \"\( .Value )\""
'
Demo on jqplay
You could convert the variable you describe into JSON with varying levels of difficulty (depending on what limitations, if any, you're willing to accept). But since you're building the variable from the output of yq, should could simply have it output JSON directly.
Using mikefarah/yq or kislyuk/yq or itchyny/gojq is definitely the better approach if your actual task is to convert from YAML and to process as JSON.
However, to accomplish the task you have described in the original question, one could add the keys and values of the bash associative array using the --args option and access them within jq via the $ARGS.positional array (requires jq version 1.6), then restore the association by transposeing both halves of that array, and create an INDEX using the key from the first half, which can then be used with JOIN to create the desired text lines, which are output as raw text using the -r option.
jq -r '
JOIN(
$ARGS.positional | [.[:length/2], .[length/2:]] | transpose | INDEX(.[0]);
.Parameters[];
.Name;
"ssm_\(.[1][1] // .[0].Name): \(.[0].Value)"
)
' example.json --args "${!ssmMap[#]}" "${ssmMap[#]}"
ssm_theGreatKEYAlias: magicvaluenotrelevant
ssm_apiKey: blabla
The fallback // .[0].Name is unnecessary for the sample file, but was added to use the original key in case there's an input key not covered by the bash array.
I have this below YAML input and I am trying to extract shown output using yq. I want to remove pairs where key name (VAR-A) in value {{a.b.VAR-A}} (after a.b.) matches and If I have more than one {{a.b.VAR-A}} in values separated by - , I want to keep them.
VAR-A: '{{a.b.VAR-A}}'
VAR-B: '{{a.b.VAR-B}}'
VAR-C: v0.0
VAR-D: '{{a.b.VAR-D}}-{{a.b.VAR-A}}'
VAR-E: '{{a.b.VAR-C}}-{{a.b.VAR-B}}-{{a.b.VAR-A}}'
VAR-F: True
Expected Output:
VAR-C: v0.0
VAR-D: '{{a.b.VAR-D}}-{{a.b.VAR-A}}'
VAR-E: '{{a.b.VAR-C}}-{{a.b.VAR-B}}-{{a.b.VAR-A}}'
VAR-F: True
This question works if I have all strings, but it fails when I have boolean value in yaml. Extract Key Value pairs which matches the regex in YAML using yq/sed/grep
I get below error:
Error: cannot substitute with !!bool, can only substitute strings. Hint: Most often you'll want to use '|=' over '=' for this operation.
There are at least two very different extant "yq" projects: a Python-based one, which is the focus of Part 1 below, and a Go-based one, which is the focus of Part 2.
Part 1
python-yq 'del(.[] | select( ( type == "string" and test("^{{a[.]b[.][^}]*}}$" ))))' so-vars.yaml
or
python-yq 'map_values( select( ( type == "string" and test("^{{a[.]b[.][^}]*}}$" )) | not))' so-vars.yaml
Output:
{
"VAR-C": "v0.0",
"VAR-D": "{{a.b.VAR-D}}-{{a.b.VAR-A}}",
"VAR-E": "{{a.b.VAR-C}}-{{a.b.VAR-B}}-{{a.b.VAR-A}}",
"VAR-F": true
}
Part 2
The Go-based version of yq that I have (4.6.3) might not be able to handle your requirements directly, but here's a solution that uses this yq to translate to and from JSON, and jq to do the rest:
yq -j eval . input.yaml |
jq 'del(.[] | select(( type == "string" and test("^{{a[.]b[.][^}]*}}$" ))))' > tmp.json
yq -P eval . tmp.json
The del-free version of the jq program:
map_values( select( type == "string" and test("^{{a[.]b[.][^}]*}}$" | not)
Output:
VAR-C: v0.0
VAR-D: '{{a.b.VAR-D}}-{{a.b.VAR-A}}'
VAR-E: '{{a.b.VAR-C}}-{{a.b.VAR-B}}-{{a.b.VAR-A}}'
VAR-F: true
I am writing a bash script utilizing jq to filter out JSON entries given some bash variables and return some of the key values from each entry into a tab-delimited file. I think the first few lines of this command are okay, but the 4th line I think is causing the problem. I have tried piping each entry in line 4 to tostring but to no avail.
info=`cat $FILE | jq -r \
' .[] \
| map(select(.host| contains(env.A))) \
| [."ip.A",."ts",."ip.B"] \
| #tsv'`
JSON example entry:
{
"ts": "2019-06-19T00:00:00.000000Z",
"ip.A": "0.0.0.0",
"ip.B": "0.0.0.0",
"host": "www.google.com",
}
In these files, there are no brackets surrounding the entire text within the file.
Error Given:
jq: error (at <stdin>:0): Cannot iterate over string ("2019-06-18...)
Do I need to handle ".ts" in some special way?
This code is broken long before the 3rd line.
If there isn't an outer array or object, you can't use .[].
If your data type is an object and not a list, using map() on it throws away data (specifically, it discards the keys, leaving only the values).
...so, .[] iterates over the values in your object, and then map() tries to iterate over each of those values as if it was an iterable object itself, which it's not... hence your error.
A version cut down to remove the broken parts might look like:
a="google.com" jq -r '
if (.host | contains(env.a)) then
[."ip.A",."ts",."ip.B"] | #tsv
else
empty
end
' <<'EOF'
{
"ts": "2019-06-19T00:00:00.000000Z",
"ip.A": "0.0.0.0",
"ip.B": "0.0.0.0",
"host": "www.google.com"
}
EOF
...which works fine.
So I have a file of "keys", for example:
key1
key2
key3
and I have a file of key:value pairs:
key1:value1
key2:value2
key3:value3
I want to replace the keys in my file of keys with their corresponding values in the key:value file. So the file of keys will look like this when complete:
value1
value2
value3
...
What is the best way to do this in bash? Note that a key may appear more than once in the keys file, but should only appear once in the key:values file.
if the join command is available in your environment, the following should work. The addition of an index via the awk command is needed to restore original key order (via a Schwartzian transform).
join -o 1.1,2.2 -t':' -1 2 -2 1 <(awk '{print(NR":"$0)}' key_file | sort -k2,2 -t':') <(sort -k1,1 -t':' key_values_file) | sort -k1,1 -t':' | cut -f2 -d':'
I know you want "bash" , but this is very simply solved with a quick perl script. Assume you have the files pairs.txt and keys.txt :
use strict;
my %keys2values;
# read through the pairs file to get the key:value mapping
open PAIRS, "cat pairs.txt |" ;
while(<PAIRS>) {
chomp $_;
my ($key,$value) = split(":",$_);
$keys2values{$key} = $value;
}
open KEYS, "cat keys.txt |";
while(<KEYS>) {
chomp $_;
my $key = $_;
if(defined $keys2values{$key}) {
print "$keys2values{$key}\n";
}
# if a key:value pair isn't defined, just print the key
else {
print "$key\n";
}
}
Since I have a thing for pure-bash solutions, I'll just post this solution. It will only work in bash 4+, because it uses associative arrays.
#!/bin/bash
while IFS=: read key value; do
declare -A hash[$key]=$value
done < pairfile
while read key; do
printf '%s\n' "${hash[$key]}"
done < keyfile