How to use(read) value from associative array during jq map? - bash

I am reading aws ssm parameters via jq with json, and want to put them into a yaml file after some processing and alias mapping.
I have one associative array of mapping parameter name to some internal name (see process.sh)
My simplified example.json is
{
"Parameters": [
{
"Name": "/dev/applications/web/some_great_key",
"Value": "magicvaluenotrelevant"
},
{
"Name": "/dev/applications/web/api_magic",
"Value": "blabla"
}
]
}
My bash script process.sh is
#!/usr/bin/env bash
declare -A ssmMap
ssmMap[/dev/applications/web/some_great_key]=theGreatKEYAlias
ssmMap[/dev/applications/web/api_magic]=apiKey
jq -r '
.Parameters
| map({
Name: .Name,
parameterValue: .Value
})
| map((${!ssmMap[.Name]} + ": \"" + .parameterValue + "\""))
| join("\n")
' < example.json;
I want/expected the output to be:
ssm_theGreatKEYAlias: "magicvaluenotrelevant"
ssm_apiKey: "singleblabla"
With the process.sh script that I provided I get error because cannot find a way to use the associative array sshMap inside jq map, to transform the parameter name from json into the mapped alias from sshMap.
Any idea how that can be achieved ?
If I change the line 13 of process.sh into
| map((.Name + ": \"" + .parameterValue + "\""))
it works fine but without mapping, uses original parameter name as comes from json file.

You're trying to access a shell var from jq.
Fixed:
ssm_map='
{
"some_great_key": "theGreatKEYAlias",
"api_magic": "apiKey"
}
'
jq -r --argjson ssm_map "$ssm_map" '
.Parameters[] |
"ssm_\( $ssm_map[ .Name ] ): \"\( .Value )\""
'
Demo on jqplay
You could convert the variable you describe into JSON with varying levels of difficulty (depending on what limitations, if any, you're willing to accept). But since you're building the variable from the output of yq, should could simply have it output JSON directly.

Using mikefarah/yq or kislyuk/yq or itchyny/gojq is definitely the better approach if your actual task is to convert from YAML and to process as JSON.
However, to accomplish the task you have described in the original question, one could add the keys and values of the bash associative array using the --args option and access them within jq via the $ARGS.positional array (requires jq version 1.6), then restore the association by transposeing both halves of that array, and create an INDEX using the key from the first half, which can then be used with JOIN to create the desired text lines, which are output as raw text using the -r option.
jq -r '
JOIN(
$ARGS.positional | [.[:length/2], .[length/2:]] | transpose | INDEX(.[0]);
.Parameters[];
.Name;
"ssm_\(.[1][1] // .[0].Name): \(.[0].Value)"
)
' example.json --args "${!ssmMap[#]}" "${ssmMap[#]}"
ssm_theGreatKEYAlias: magicvaluenotrelevant
ssm_apiKey: blabla
The fallback // .[0].Name is unnecessary for the sample file, but was added to use the original key in case there's an input key not covered by the bash array.

Related

jq How to pass key starting with numeral as argument [duplicate]

I am new to jq and facing an issue while parsing my json
I have a json stored in a variable like this
temp='{ "1": { "my_name": "one" }, "2": { "my_name": "two" } }'
Now I need to get the value of my_name for both other entries
I have tried something like this
echo $temp | jq '.1' //out put 0.1
I was assuming to get { "my_name": "one" }
And similarly to get my_name value I did
echo $temp | jq '.1.my_name' // Its output is giving me error
Can any one please help to identify what is wrong with my syntax and how can I correct it.
Just the number gets interpreted as a float. You need to use it in a context where it is unambiguously a key string.
echo "$temp" | jq '.["1"]["my_name"]'
and
echo "$temp" | jq '.["1"]'
to get the containing dict.
With a sufficiently new jq (I think >= 1.4) you can also say
echo "$temp" | jq '."1".my_name'
Whenever you are trying to reference a key that is not a valid identifier, you have to quote it. See the manual for more details.
To select the item under the key 1, you'd do this:
."1"
For your other question on how to obtain the my_name values, you could do this:
to_entries | map(.value.my_name)

bash loop error : Get JSON Object by property with jq / bash

I would like to get the values from Json file. Which is working.
JsonFileToTest:
{
"permissions": [
{
"emailid": "test1#test.com",
"rights": "read"
},
{
"emailid": "test2#test.com",
"rights": "read"
}
]
}
readPermissions=($(jq -r '.permissions' JsonFileToTest))
# The command below works perfectly, But when I Put it in a loop, It does not.
#echo ${readPermissions[#]} | jq 'values[].emailid'
for vals in ${readPermissions[#]}
do
# I would like o extract the email id of the user. The loop is not working atm.
echo ${vals[#]} | jq 'values[].emailid'
done
what am I missing here?
thanks
If you really want to do it this way, that might look like:
readarray -t permissions < <(jq -c '.permissions[]' JsonFileToTest)
for permissionSet in "${permissions[#]}"; do
jq -r '.emailid' <<<"$permissionSet"
done
Note that we're telling jq to print one line per item (with -c), and using readarray -t to read each line into an array element (unlike the array=( $(...command...) ) antipattern, which splits not just on newlines but on other whitespace as well, and expands globs in the process).
But there's no reason whatsoever to do any of that. You'll get the exact same result simply running:
jq -r '.permissions[].emailid' JsonFileToTest

How do I concatenate dummy values in JQ based on field value, and then CSV-aggregate these concatenations?

In my bash script, when I run the following jq against my curl result:
curl -u someKey:someSecret someURL 2>/dev/null | jq -r '.schema' | jq -r -c '.fields'
I get back a JSON array as follows:
[{"name":"id","type":"int","doc":"Documentation for the id field."},{"name":"test_string","type":"string","doc":"Documentation for the test_string field"}]
My goal is to do a call with jq applied to return the following (given the example above):
{"id":1234567890,"test_string":"xxxxxxxxxx"}
NB: I am trying to automatically generate templated values that match the "schema" JSON shown above.
So just to clarify, that is:
all array objects (there could be more than 2 shown above) returned in a single comma-delimited row
doc fields are ignored
the values for "name" (including their surrounding double-quotes) are concatenated with either:
:1234567890 ...when the "type" for that object is "int"
":xxxxxxxxxx" ...when the "type" for that object is "string"
NB: these will be the only types we ever get for now
Can someone show me how I can expand upon my initial jq to return this?
NB: I tried working down the following path but am failing beyond this...
curl -u someKey:someSecret someURL 2>/dev/null | jq -r '.schema' | jq -r -c '.fields' | "\(.name):xxxxxxxxxxx"'
If it's not possible in pure JQ (my preference) I'm also happy for a solution that mixes in a bit of sed/awk magic :)
Cheers,
Stan
Given the JSON shown, you could add the following to your pipeline:
jq -c 'map({(.name): (if .type == "int" then 1234567890 else "xxxxxxxxxx" end)})|add'
With that JSON, the output would be:
{"id":1234567890,"test_string":"xxxxxxxxxx"}
However, it would be far better if you combined the three calls to jq into one.

jq command throws error "cannot iterate over string" when

I am writing a bash script utilizing jq to filter out JSON entries given some bash variables and return some of the key values from each entry into a tab-delimited file. I think the first few lines of this command are okay, but the 4th line I think is causing the problem. I have tried piping each entry in line 4 to tostring but to no avail.
info=`cat $FILE | jq -r \
' .[] \
| map(select(.host| contains(env.A))) \
| [."ip.A",."ts",."ip.B"] \
| #tsv'`
JSON example entry:
{
"ts": "2019-06-19T00:00:00.000000Z",
"ip.A": "0.0.0.0",
"ip.B": "0.0.0.0",
"host": "www.google.com",
}
In these files, there are no brackets surrounding the entire text within the file.
Error Given:
jq: error (at <stdin>:0): Cannot iterate over string ("2019-06-18...)
Do I need to handle ".ts" in some special way?
This code is broken long before the 3rd line.
If there isn't an outer array or object, you can't use .[].
If your data type is an object and not a list, using map() on it throws away data (specifically, it discards the keys, leaving only the values).
...so, .[] iterates over the values in your object, and then map() tries to iterate over each of those values as if it was an iterable object itself, which it's not... hence your error.
A version cut down to remove the broken parts might look like:
a="google.com" jq -r '
if (.host | contains(env.a)) then
[."ip.A",."ts",."ip.B"] | #tsv
else
empty
end
' <<'EOF'
{
"ts": "2019-06-19T00:00:00.000000Z",
"ip.A": "0.0.0.0",
"ip.B": "0.0.0.0",
"host": "www.google.com"
}
EOF
...which works fine.

Get json field value with JQ from different directory

Title may be incorrect as I'm not actually sure where this is failing. I have a bash script running in one directory, and a JSON file I need a value from in a different directory. I want to copy the value from the external directory into an identical JSON file in the current directory.
I'm using jq to grab the value, but I can't figure out how to grab from a directory other than the one the script is running in.
The relevant bits of file structure are as follows;
cloudformation
- parameters_v13.json
environment_files
- prepare_stack_files.json (the script this is run from)
- directory, changes based on where the script is pointed
- created directory where created files are being output
- GREPNAME_parameters.json
The chunk of the JSON file I'm interested in looks like this;
[
{
"ParameterKey": "RTSMEMAIL",
"ParameterValue": "secretemail"
}
]
The script needs to get the "secretemail" from cloudformation/parameters_v13.json and paste it into the matching RTSMEMAIL field in the GREPNAME_parameters.json file.
I've been attempting the following with no luck - nothing is output. No error message either, just blank output. I know the GREPNAME path is correct because it's used elsewhere with no issues.
jq --arg email "$EMAIL" '(.[] | select(.ParameterKey == "RTSMEMAIL") | .ParameterValue) |= $email' ../cloudformation/parameters_v13.json | sponge ${GREPNAME}_parameters.json
This jq filter should help you get secretmail string
jq '.[] | select(.ParameterKey=="RTSMEMAIL") | .ParameterValue' json
"secretemail"
Add a -r file for raw output to remove quotes around the value
jq -r '.[] | select(.ParameterKey=="RTSMEMAIL") | .ParameterValue' json
secretemail
--raw-output / -r:
With this option, if the filter’s result is a string then it will be written directly to standard output rather than being formatted as a JSON string with quotes. This can be useful for making jq filters talk to non-JSON-based systems.
As I could see it you are trying to pass args to jq filter, for extraction you can do something first by setting the variable in bash
email="RTSMEMAIL"
and now pass it to the filter as
jq --arg email "$email" -r '.[] | select(.ParameterKey==$email) | .ParameterValue' json
secretemail
Now to replace the string obtained from parameters_v13.json file to your GREPNAME_parameters.json do the following steps:-
First storing the result from the first file in a variable to re-use later, I have used the file to extract as json, this actually points your parameters_v13.json file in another path.
replacementValue=$(jq --arg email "$email" -r '.[] | select(.ParameterKey==$email) | .ParameterValue' json)
now the $replacementValue will hold the secretmail which you want to update to another file. As you have indicated previously GREPNAME_parameters.json has a similar syntax as of the first file. Something like below,
$ cat GREPNAME_parameters.json
[
{
"ParameterKey": "SOMEJUNK",
"ParameterValue": "somejunkvalue"
}
]
Now I understand your intention is replace "ParameterValue" from the above file to the value obtained from the other file. To achieve that,
jq --arg replace "$replacementValue" '.[] | .ParameterValue = $replace' GREPNAME_parameters.json
{
"ParameterKey": "SOMEJUNK",
"ParameterValue": "secretemail"
}
You can then write this output to the a temp file and move it back as the GREPNAME_parameters.json. Hope this answers your question.
#Alex -
(1) sponge simply provides a convenient way to modify a file without having to manage a temporary file. You could use it like this:
jq ........ input.json | sponge input.json
Here, "input.json" is the file that you want to edit "in place". If you want to avoid overwriting the input file, you would not use sponge. In fact, I would recommend against doing so until you're absolutely sure that's what you want.
(2) There are several strategies for achieving what you have described using jq. They basically fall into two categories: (a) invoke jq twice; (b) invoke jq once.
Ignoring the sponge part:
the pattern for using jq twice would be as follows:
param=$(jq -r '.[]
| select(.ParameterKey == "RTSMEMAIL")|.ParameterValue
' cloudformation/parameters_v13.json )
jq --arg param "$param" -f edit.jq input.json
assuming you have jq 1.5, the pattern for doing everything with just one invocation of jq would be:
jq --argfile p cloudformation/parameters_v13.json -f manage.jq input.json
Here, edit.jq and manage.jq are files containing suitable jq programs.
Based on my understanding of your requirements, edit.jq might look like this:
(.[] | select(.ParameterKey == "RTSMEMAIL")|.ParameterValue) |= $param
And manage.jq might look like this:
($p[] | select(.ParameterKey == "RTSMEMAIL")|.ParameterValue) as $param
| (.[]| select(.ParameterKey == "RTSMEMAIL")|.ParameterValue) |= $param

Resources