I have an array of json objects that I'd like to convert to an associative array in bash with a slight alteration to the key
{
"Parameters": [
{
"Name": "/path/user_management/api_key",
"Type": "SecureString",
"Value": "1234",
"Version": 1
},
{
"Name": "/path/user_management/api_secret",
"Type": "SecureString",
"Value": "5678",
"Version": 1
}
]
}
I know I need to use jq and sed but I just can't quite find the proper combination of doing what I'm looking for. Need to strip out "/path/user_management/" and set the remaining as the key, and use Value for value.
Trying to find a fairly clean one liner piping commands together. What I'd like to end up with is a bash associative array of something like:
myArray[api_key]="1234"
myArray[api_secret]="5678"
Asking for a one-liner code is as good as asking for unreadable code. If you want do this in a proper way, read the output of jq command in a while loop and strip out unwanted characters as required.
#!/usr/bin/env bash
# declare an associative array, the -A defines the array of this type
declare -A _my_Array
# The output of jq is separated by '|' so that we have a valid delimiter
# to read our keys and values. The read command processes one line at a
# time and puts the values in the variables 'key' and 'value'
while IFS='|' read -r key value; do
# Strip out the text until the last occurrence of '/'
strippedKey="${key##*/}"
# Putting the key/value pair in the array
_my_Array["$strippedKey"]="$value"
done< <(jq -r '.Parameters[] | "\(.Name)|\(.Value)"' json)
# Print the array using the '-p' or do one by one
declare -p _my_Array
Or print the array, the traditional way
for key in "${!_my_Array[#]}"; do
printf '%s %s\n' "${key}" "${_my_Array[$key]}"
done
Related
I am reading aws ssm parameters via jq with json, and want to put them into a yaml file after some processing and alias mapping.
I have one associative array of mapping parameter name to some internal name (see process.sh)
My simplified example.json is
{
"Parameters": [
{
"Name": "/dev/applications/web/some_great_key",
"Value": "magicvaluenotrelevant"
},
{
"Name": "/dev/applications/web/api_magic",
"Value": "blabla"
}
]
}
My bash script process.sh is
#!/usr/bin/env bash
declare -A ssmMap
ssmMap[/dev/applications/web/some_great_key]=theGreatKEYAlias
ssmMap[/dev/applications/web/api_magic]=apiKey
jq -r '
.Parameters
| map({
Name: .Name,
parameterValue: .Value
})
| map((${!ssmMap[.Name]} + ": \"" + .parameterValue + "\""))
| join("\n")
' < example.json;
I want/expected the output to be:
ssm_theGreatKEYAlias: "magicvaluenotrelevant"
ssm_apiKey: "singleblabla"
With the process.sh script that I provided I get error because cannot find a way to use the associative array sshMap inside jq map, to transform the parameter name from json into the mapped alias from sshMap.
Any idea how that can be achieved ?
If I change the line 13 of process.sh into
| map((.Name + ": \"" + .parameterValue + "\""))
it works fine but without mapping, uses original parameter name as comes from json file.
You're trying to access a shell var from jq.
Fixed:
ssm_map='
{
"some_great_key": "theGreatKEYAlias",
"api_magic": "apiKey"
}
'
jq -r --argjson ssm_map "$ssm_map" '
.Parameters[] |
"ssm_\( $ssm_map[ .Name ] ): \"\( .Value )\""
'
Demo on jqplay
You could convert the variable you describe into JSON with varying levels of difficulty (depending on what limitations, if any, you're willing to accept). But since you're building the variable from the output of yq, should could simply have it output JSON directly.
Using mikefarah/yq or kislyuk/yq or itchyny/gojq is definitely the better approach if your actual task is to convert from YAML and to process as JSON.
However, to accomplish the task you have described in the original question, one could add the keys and values of the bash associative array using the --args option and access them within jq via the $ARGS.positional array (requires jq version 1.6), then restore the association by transposeing both halves of that array, and create an INDEX using the key from the first half, which can then be used with JOIN to create the desired text lines, which are output as raw text using the -r option.
jq -r '
JOIN(
$ARGS.positional | [.[:length/2], .[length/2:]] | transpose | INDEX(.[0]);
.Parameters[];
.Name;
"ssm_\(.[1][1] // .[0].Name): \(.[0].Value)"
)
' example.json --args "${!ssmMap[#]}" "${ssmMap[#]}"
ssm_theGreatKEYAlias: magicvaluenotrelevant
ssm_apiKey: blabla
The fallback // .[0].Name is unnecessary for the sample file, but was added to use the original key in case there's an input key not covered by the bash array.
How can i get the values inner depends in bash script?
manifest.py
# Commented lines
{
'category': 'Sales/Subscription',
'depends': [
'sale_subscription',
'sale_timesheet',
],
'auto_install': True,
}
Expected response:
sale_subscription sale_timesheet
The major problem is linebreak, i have already tried | grep depends but i can not get the sale_timesheet value.
Im trying to add this values comming from files into a var, like:
DOWNLOADED_DEPS=($(ls -A $DOWNLOADED_APPS | while read -r file; do cat $DOWNLOADED_APPS/$file/__manifest__.py | [get depends value])
Example updated.
If this is your JSON file:
{
"category": "Sales/Subscription",
"depends": [
"sale_subscription",
"sale_timesheet"
],
"auto_install": true
}
You can get the desired result using jq like this:
jq -r '.depends | join(" ")' YOURFILE.json
This uses .depends to extract the value from the depends field, pipes it to join(" ") to join the array with a single space in between, and uses -r for raw (unquoted) output.
If it is not a json file and only string then you can use below Regex to find the values. If it's json file then you can use other methods like Thomas suggested.
^'depends':\s*(?:\[\s*)(.*?)(?:\])$
demo
you can use egrep for this as follows:
% egrep -M '^\'depends\':\s*(?:\[\s*)(.*?)(?:\])$' pathTo\jsonFile.txt
you can read about grep
As #Thomas has pointed out in a comment, the OPs input data is not in JSON format:
$ cat manifest.py
# Commented lines // comments not allowed in JSON
{
'category': 'Sales/Subscription', // single quotes should be replaced by double quotes
'depends': [
'sale_subscription',
'sale_timesheet', // trailing comma at end of section not allowed
],
'auto_install': True, // trailing comma issue; should be lower case "true"
}
And while the title of the question mentions regex, there is no sign of a regex in the question. I'll leave a regex based solution for someone else to come up with and instead ...
One (quite verbose) awk solution based on the input looking exactly like what's in the question:
$ awk -F"'" ' # use single quote as field separator
/depends/ { printme=1 ; next } # if we see the string "depends" then set printme=1
printme && /]/ { printme=0 ; next} # if printme=1 and line contains a right bracket then set printme=0
printme { printf pfx $2; pfx=" " } # if printme=1 then print a prefix + field #2;
# first time around pfx is undefined;
# subsequent passes will find pfx set to a space;
# since using "printf" with no "\n" in sight, all output will stay on a single line
END { print "" } # add a linefeed on the end of our output
' json.dat
This generates:
sale_subscription sale_timesheet
How can I access array elements by string?
What I want to achieve is the following: I have a number of json files in a directory. These should be read in one after the other. All key values ("id", "test") should be configured / replaced. The changes should be saved in the json files.
#a json file example:
{
"id": "<fill-id>",
"test": "<fill-test>"
}
The first key "id" should call the function "create_id".
The second key "test" should call the function "create_test".
#!/bin/sh
declare -a FUNCTION
FUNCTION["id"]="create_id"
FUNCTION["test"]="create_test"
FUNCTION["secret"]="FUNCTION.C"
# read key/value from json file
for filename in *.json; do
while read -r key value; do
declare "$key=$value"
echo name: $key value: $value
${FUNCTION[$key]} $filename $value
done < <(jq -r 'to_entries[] | "\(.key) \(.value)"' $filename)
done
You'll need to use an associative array, declare -A FUNCTION
BUT associative arrays are not a sh feature: declare -A is bash syntax. Change your #! line.
You'll also need bash version 4.0 or greater: the default /bin/bash on MacOSX is too old.
You can also find associative arrays in ksh and zsh.
I have a sample JSON like
"request_id": "016-ae81e5737add",
"renewable": false,
"data": {
"client_decrypted.key": "-----BEGIN KEY-----sdjijhgisdTRjkRD05\n-----END KEY-----",
"passphrase": "password"
}
In my bash script, I am reading the JSON specifically the contents of data key. I would like to create files with the filename as key (e.g: files like client_decrypted.key & passphrase) with the contents of the files as the value of the corresponding key.
I was able to use jq to read the key & value pair
for pair in $( jq -r ".data | to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]"); do
echo $${pair} # OUTPUT=passphrase=password
done
I guess I can use IFS to split each pair on =. Is there a better way of doing this?
One safe way would be to emit the JSON records with a NULL byte padding and then later read it back in a shell loop, delimiting on that byte
while IFS= read -r -d '' key && IFS= read -r -d '' value; do
printf '%s\n' "$value" > "$key"
done < <(jq -j '.data | to_entries[] | (.key, "\u0000", .value, "\u0000")' json)
Since the NULL byte can't be part of a "valid" byte in any of the input records, it is used as the delimit character. The jq expression
(.key, "\u0000", .value, "\u0000")
appends the NULL byte between the key and the value field, which is read with a while loop , two reads at a time to store the key name and the value fields in the respective variables.
Note that <(..) is a bash/ksh93/zsh process substitution technique, which is not POSIX compliant and might not work in pure bourne shells.
I would like to get the values from Json file. Which is working.
JsonFileToTest:
{
"permissions": [
{
"emailid": "test1#test.com",
"rights": "read"
},
{
"emailid": "test2#test.com",
"rights": "read"
}
]
}
readPermissions=($(jq -r '.permissions' JsonFileToTest))
# The command below works perfectly, But when I Put it in a loop, It does not.
#echo ${readPermissions[#]} | jq 'values[].emailid'
for vals in ${readPermissions[#]}
do
# I would like o extract the email id of the user. The loop is not working atm.
echo ${vals[#]} | jq 'values[].emailid'
done
what am I missing here?
thanks
If you really want to do it this way, that might look like:
readarray -t permissions < <(jq -c '.permissions[]' JsonFileToTest)
for permissionSet in "${permissions[#]}"; do
jq -r '.emailid' <<<"$permissionSet"
done
Note that we're telling jq to print one line per item (with -c), and using readarray -t to read each line into an array element (unlike the array=( $(...command...) ) antipattern, which splits not just on newlines but on other whitespace as well, and expands globs in the process).
But there's no reason whatsoever to do any of that. You'll get the exact same result simply running:
jq -r '.permissions[].emailid' JsonFileToTest