I have a yq read command as below,
groups=$(yq read generated/identity-mapping.yaml "iamIdentityMappings.[0].groups")
It reads iamIdentityMappings from below yaml:
iamIdentityMappings:
- groups:
- Appdeployer
- Moregroups
It stores group as below,
- Appdeployer
- Moregroups
But I want to store groups as below.(comma separated values)
groups="Appdeployer","Moregroups"
How to do this in bash?
yq is just a wrapper for jq, which supports CSV output:
$ groups="$(yq -r '.iamIdentifyMappings[0].groups | #csv' generated/identity-mapping.yaml)"
$ echo "$groups"
"Appdeployer","Moregroups"
The yq invocation in your question just causes an error. Note the fixed version.
Use mapfile and format a null delimited list with yq:
mapfile -d '' -t groups < <(
yq -j '.iamIdentityMappings[0].groups[]+"\u0000"' \
generated/identity-mapping.yaml
)
typeset -p groups
Output:
declare -a groups=([0]="Appdeployer" [1]="Moregroups")
And now you can fulfill this second part of your question:
Construct a command based upon a count variable in bash
# Prepare eksctl's arguments into an array
declare -a eksctl_args=(create iamidentitymapping --cluster "$name" --region "$region" --arn "$rolearn" )
# Read the groups from the yml into an array
mapfile -d '' -t groups < <(
yq -j '.iamIdentityMappings[0].groups[]+"\u0000"' \
generated/identity-mapping.yaml
)
# Add arguments per group
for group in "${groups[#]}"; do
eksctl_args+=(--group "$group")
done
# add username argument
eksctl_args+=(--username "$username")
# call eksctl with its arguments
eksctl "${eksctl_args[#]}"
yq 4.16+ now has a built in #csv operator:
yq e '.iamIdentityMappings.[0].groups | #csv' file.yaml
Note that #csv will only wrap values in quotes if needed (e.g. they have a comma).
If you want quotes, then sub then in and join with commas:
yq e '
.iamIdentityMappings.[0].groups |
(.[] |= sub("(.*)", "\"${1}\""))
| join(",")'
Disclaimer: I wrote yq.
yq version 3 is deprecated now and you can achieve the same output using version 4
#!/bin/bash
while IFS= read -r value; do
groups_array+=($value)
done < <(yq eval '.iamIdentityMappings.[0].groups.[]' generated/identity-mapping.yaml)
printf -v comma_seperated '%s,' "${groups_array[#]}"
echo "${comma_seperated%,}"
This code prints the comma seperated values as you wanted
Related
I have a JSON data as follows in data.json file
[
{"original_name":"pdf_convert","changed_name":"pdf_convert_1"},
{"original_name":"video_encode","changed_name":"video_encode_1"},
{"original_name":"video_transcode","changed_name":"video_transcode_1"}
]
I want to iterate through the array and extract the value for each element in a loop. I saw jq. I find it difficult to use it to iterate. How can I do that?
Just use a filter that would return each item in the array. Then loop over the results, just make sure you use the compact output option (-c) so each result is put on a single line and is treated as one item in the loop.
jq -c '.[]' input.json | while read i; do
# do stuff with $i
done
By leveraging the power of Bash arrays, you can do something like:
# read each item in the JSON array to an item in the Bash array
readarray -t my_array < <(jq --compact-output '.[]' input.json)
# iterate through the Bash array
for item in "${my_array[#]}"; do
original_name=$(jq --raw-output '.original_name' <<< "$item")
changed_name=$(jq --raw-output '.changed_name' <<< "$item")
# do your stuff
done
jq has a shell formatting option: #sh.
You can use the following to format your json data as shell parameters:
cat data.json | jq '. | map([.original_name, .changed_name])' | jq #sh
The output will look like:
"'pdf_convert' 'pdf_convert_1'"
"'video_encode' 'video_encode_1'",
"'video_transcode' 'video_transcode_1'"
To process each row, we need to do a couple of things:
Set the bash for-loop to read the entire row, rather than stopping at the first space (default behavior).
Strip the enclosing double-quotes off of each row, so each value can be passed as a parameter to the function which processes each row.
To read the entire row on each iteration of the bash for-loop, set the IFS variable, as described in this answer.
To strip off the double-quotes, we'll run it through the bash shell interpreter using xargs:
stripped=$(echo $original | xargs echo)
Putting it all together, we have:
#!/bin/bash
function processRow() {
original_name=$1
changed_name=$2
# TODO
}
IFS=$'\n' # Each iteration of the for loop should read until we find an end-of-line
for row in $(cat data.json | jq '. | map([.original_name, .changed_name])' | jq #sh)
do
# Run the row through the shell interpreter to remove enclosing double-quotes
stripped=$(echo $row | xargs echo)
# Call our function to process the row
# eval must be used to interpret the spaces in $stripped as separating arguments
eval processRow $stripped
done
unset IFS # Return IFS to its original value
From Iterate over json array of dates in bash (has whitespace)
items=$(echo "$JSON_Content" | jq -c -r '.[]')
for item in ${items[#]}; do
echo $item
# whatever you are trying to do ...
done
Try Build it around this example. (Source: Original Site)
Example:
jq '[foreach .[] as $item ([[],[]]; if $item == null then [[],.[0]] else [(.[0] + [$item]),[]] end; if $item == null then .[1] else empty end)]'
Input [1,2,3,4,null,"a","b",null]
Output [[1,2,3,4],["a","b"]]
None of the answers here worked for me, out-of-the-box.
What did work was a combination of a few:
projectList=$(echo "$projRes" | jq -c '.projects[]')
IFS=$'\n' # Read till newline
for project in ${projectList[#]}; do
projectId=$(jq '.id' <<< "$project")
projectName=$(jq -r '.name' <<< "$project")
...
done
unset IFS
NOTE: I'm not using the same data as the question does, in this example assume projRes is the output from an API that gives us a JSON list of projects, eg:
{
"projects": [
{"id":1,"name":"Project"},
... // array of projects
]
}
An earlier answer in this thread suggested using jq's foreach, but that may be much more complicated than needed, especially given the stated task. Specifically, foreach (and reduce) are intended for certain cases where you need to accumulate results.
In many cases (including some cases where eventually a reduction step is necessary), it's better to use .[] or map(_). The latter is just another way of writing [.[] | _] so if you are going to use jq, it's really useful to understand that .[] simply creates a stream of values.
For example, [1,2,3] | .[] produces a stream of the three values.
To take a simple map-reduce example, suppose you want to find the maximum length of an array of strings. One solution would be [ .[] | length] | max.
Here is a simple example that works in zch shell:
DOMAINS='["google","amazon"]'
arr=$(echo $DOMAINS | jq -c '.[]')
for d in $arr; do
printf "Here is your domain: ${d}\n"
done
I stopped using jq and started using jp, since JMESpath is the same language as used by the --query argument of my cloud service and I find it difficult to juggle both languages at once. You can quickly learn the basics of JMESpath expressions here: https://jmespath.org/tutorial.html
Since you didn't specifically ask for a jq answer but instead, an approach to iterating JSON in bash, I think it's an appropriate answer.
Style points:
I use backticks and those have fallen out of fashion. You can substitute with another command substitution operator.
I use cat to pipe the input contents into the command. Yes, you can also specify the filename as a parameter, but I find this distracting because it breaks my left-to-right reading of the sequence of operations. Of course you can update this from my style to yours.
set -u has no function in this solution, but is important if you are fiddling with bash to get something to work. The command forces you to declare variables and therefore doesn't allow you to misspell a variable name.
Here's how I do it:
#!/bin/bash
set -u
# exploit the JMESpath length() function to get a count of list elements to iterate
export COUNT=`cat data.json | jp "length( [*] )"`
# The `seq` command produces the sequence `0 1 2` for our indexes
# The $(( )) operator in bash produces an arithmetic result ($COUNT minus one)
for i in `seq 0 $((COUNT - 1))` ; do
# The list elements in JMESpath are zero-indexed
echo "Here is element $i:"
cat data.json | jp "[$i]"
# Add or replace whatever operation you like here.
done
Now, it would also be a common use case to pull the original JSON data from an online API and not from a local file. In that case, I use a slightly modified technique of caching the full result in a variable:
#!/bin/bash
set -u
# cache the JSON content in a stack variable, downloading it only once
export DATA=`api --profile foo compute instance list --query "bar"`
export COUNT=`echo "$DATA" | jp "length( [*] )"`
for i in `seq 0 $((COUNT - 1))` ; do
echo "Here is element $i:"
echo "$DATA" | jp "[$i]"
done
This second example has the added benefit that if the data is changing rapidly, you are guaranteed to have a consistent count between the elements you are iterating through, and the elements in the iterated data.
This is what I have done so far
arr=$(echo "$array" | jq -c -r '.[]')
for item in ${arr[#]}; do
original_name=$(echo $item | jq -r '.original_name')
changed_name=$(echo $item | jq -r '.changed_name')
echo $original_name $changed_name
done
I am trying to pass bash variable in yq
test.yml
configuration:
Properties:
corporate-url: https://stackoverflow.com/
temp = '.configuration.Properties.corporate-url'
export $temp
Value1=$(yq '.[env($temp)]' test.yml)
expected output:
https://stackoverflow.com/
but I am getting this error(Actual output)
Error: Value for env variable '$variable1' not provided in env()
Please note:
I am trying to fetch corporate-url value, using a bash variable, constraint is that I cannot pass string directly in yq as the value of temp changes as this snippet is running inside a for loop which changes value of temp every time so cannot hard code for a particular value.
Reference YQ Documentation:
https://mikefarah.gitbook.io/yq/operators/env-variable-operators
ApisDraft folder contains multiple yml files
ApisDraft=$(find drafts/* -maxdepth 1 -type f)
for ApiFixOrgsTags in $ApisDraft
do
my_var=$(yq '.securityDefinitions.[].tokenUrl' $ApiFixOrgsTags)
ConfigProper='.configuration.Properties.'
CatelogProper='.configuration.catalogs.[].Properties.'
variable1=$ConfigProper${my_var}
variable2=$CatelogProper${my_var}
# to remove white all spaces
variable1= echo $variable1 | sed -E 's/(\.) */\1/g'
variable2= echo $variable2 | sed -E 's/(\.) */\1/g'
export $variable1
export $variable2
Value1=$(yq "$variable1" $ApiFixOrgsTags)
Value2=$(yq '.[env($variable2)]' $ApiFixOrgsTags)
done
In this case, you don't need to put it in the environment. Let the shell expand it so yq just sees the value of the variable:
yq "$temp" test.yml # => https://stackoverflow.com/
I have below command:
ExpirationDate=$(date -d '+60 days' +'%Y-%m-%d')
VaultName="abc"
getapp=$(az keyvault secret list --vault-name $VaultName --query "[].{SecretName:name,ExpiryDate:attributes.expires} [?ExpiryDate<='$ExpirationDate']" | jq '.[].SecretName' | tr -d '"')
getserviceprincipal=$(az keyvault secret list --vault-name $VaultName --query "[].{Type:contentType,ExpiryDate:attributes.expires} [?ExpiryDate<='$ExpirationDate']" | jq '.[].Type' | tr -d '"')
## get length of $distro array
len=${#getapp[#]}
## Use bash for loop
for (( i=0; i-le$len-1; i++ ))
do
echo "${getapp[$i]}"
./resetpassword.sh -a ${getapp[$i]} -s ${getserviceprincipal[$i]} -y
echo "${getserviceprincipal[$i]}"
done
in this command I want store all value of vault name getapp and similarly getserviceprincipal. Example If I have more then 2 vault in getapp variable then script is not working due to $getapp is not storing variable in array.
Is anyone help me to put out this simple solutions!! Thanks In Advance..
readarray -t getapp < <( az keyvault ... | tr -d '"' ) should do the trick here.
Note that this requires newlines to be valid delimiters. If there can be newlines in your data then you'll have to pick a different delimiter with the -d delim option. If there isn't any single delimiter that works everywhere then bash may not be the best choice for this.
Since you are using jq, I think you could so something like that:
declare -a getapp=()
declare -a getserviceprincipal=()
# note: be sure to check that the resulting bash is valid!
eval(az keyvault secret list \
--vault-name $VaultName \
--query "[].{SecretName:name,ExpiryDate:attributes.expires} [?ExpiryDate<='$ExpirationDate']" \
| jq --raw-output '.[] | #sh "getapp+=( \(.SecretName) ) ; getserviceprincipal+=( \(.Type) );' ")
If all goes well, this will result in getapp and getserviceprincipal being filled as array: https://jqplay.org/s/BbHMn9i79KB
Note:
as you can see, you don't need to invoke your command (az) twice.
you can also extract the jq expression to a file using the --from-file option, which may help when reading it and handling shell quotes.
I have a yaml file say "test.yaml". The below is the content of yaml file.
...
test:
config:
abc: name1
xyz: name2
...
Now I want to read the value of abc and xyz alone from the yaml inside a shell script and store it in two variables inside shell script. test.yaml file contains additional data apart from the above one which I don't need to bother about that inside this shell script.
Eg: test.sh
var1=name1 //test[config[abc]]
var2=name2 //test[config[xyz]]
How do I read specific data (as key-value) from yaml inside a shell script. It would be really helpful if someone helps me out on this. Thanks in advance!!!
Here's an example with yq. All of the following assumes that the values do not contain newlines.
Given
$ cat test.yaml
---
test:
config:
abc: name1
xyz: name2
then
yq e '.test.config | to_entries | map(.value) | .[]' test.yaml
outputs
name1
name2
You can read them into variables like
{ read -r var1; read -r var2; } < <(yq e '.test.config | to_entries | map(.value) | .[]' test.yaml)
declare -p var1 var2
declare -- var1="name1"
declare -- var2="name2"
I would read them into an associative array with the yaml key though:
declare -A conf
while IFS="=" read -r key value; do conf["$key"]=$value; done < <(
yq e '.test.config | to_entries | map([.key, .value] | join("=")) | .[]' test.yaml
)
declare -p conf
declare -A conf=([abc]="name1" [xyz]="name2" )
Then you can write
echo "test config for abc is ${conf[abc]}"
# or
for var in "${!conf[#]}"; do printf "key %s, value %s\n" "$var" "${conf[$var]}"; done
I'm using "the Go implementation"
$ yq --version
yq (https://github.com/mikefarah/yq/) version 4.16.1
I am trying to use jq to parse information from the TVDB api. I need to pull a couple of fields and assign the values to variables that I can continue to use in my bash script. I know I can easily assign the output to one variable through bash with variable="$(command)" but I need the output to produce multiple variables and I don't want to make to use multiple commands.
I read this documentation:
https://stedolan.github.io/jq/manual/v1.5/#Advancedfeatures
but I don't know if this relevant to what I am trying to do.
jq '.data' produces the following output:
[
{
"absoluteNumber": 51,
"airedEpisodeNumber": 6,
"airedSeason": 4,
"airedSeasonID": 680431,
"dvdEpisodeNumber": 6,
"dvdSeason": 4,
"episodeName": "We Will Rise",
"firstAired": "2017-03-15",
"id": 5939660,
"language": {
"episodeName": "en",
"overview": "en"
},
"lastUpdated": 1490769062,
"overview": "Clarke and Roan must work together in hostile territory in order to deliver an invaluable asset to Abby and her team."
}
]
I tried jq '.data | {episodeName:$name}' and jq '.data | .episodeName as $name' just to try and get one working. I don't understand the documentation or even if it's what I'm looking for. Is there a way to do what I am trying to do?
You can use separate variables with read :
read var1 var2 var3 < <(echo $(curl -s 'https://api.github.com/repos/torvalds/linux' |
jq -r '.id, .name, .full_name'))
echo "id : $var1"
echo "name : $var2"
echo "full_name : $var3"
Using array :
read -a arr < <(echo $(curl -s 'https://api.github.com/repos/torvalds/linux' |
jq -r '.id, .name, .full_name'))
echo "id : ${arr[0]}"
echo "name : ${arr[1]}"
echo "full_name : ${arr[2]}"
Also you can split jq output with some character :
IFS='|' read var1 var2 var3 var4 < <(curl '......' | jq -r '.data |
map([.absoluteNumber, .airedEpisodeNumber, .episodeName, .overview] |
join("|")) | join("\n")')
Or use an array like :
set -f; IFS='|' data=($(curl '......' | jq -r '.data |
map([.absoluteNumber, .airedEpisodeNumber, .episodeName, .overview] |
join("|")) | join("\n")')); set +f
absoluteNumber, airedEpisodeNumber, episodeName & overview are respectively ${data[0]}, ${data[1]}, ${data[2]}, ${data[3]}. set -f and set +f are used to respectively disable & enable globbing.
For the jq part, all your required fields are mapped and delimited with a '|' character with join("|")
If your are using jq < 1.5, you'll have to convert Number to String with tostring for each Number fields eg:
IFS='|' read var1 var2 var3 var4 < <(curl '......' | jq -r '.data |
map([.absoluteNumber|tostring, .airedEpisodeNumber|tostring, .episodeName, .overview] |
join("|")) | join("\n")')
jq always produces a stream of zero or more values. For example, to produce the two values corresponding to "episodeName" and "id"' you could write:
.data[] | ( .episodeName, .id )
For your purposes, it might be helpful to use the -c command-line option, to ensure each JSON output value is presented on a single line. You might also want to use the -r command-line option, which removes the outermost quotation marks from each output value that is a JSON string.
For further variations, please see the jq FAQ https://github.com/stedolan/jq/wiki/FAQ, e.g. the question:
Q: How can a stream of JSON texts produced by jq be converted into a bash array of corresponding values?
Experimental conversion of quoted OP input, (tv.dat), to a series of bash variables, (and an array). The jq code is mostly borrowed from here and there, but I don't know how to get jq to unroll an array within an array, so the sed code does that, (that's only good for one level, but so are bash arrays):
jq -r ".[] | to_entries | map(\"DAT_\(.key) \(.value|tostring)\") | .[]" tv.dat |
while read a b ; do echo "${a,,}='$b'" ; done |
sed -e '/{.*}/s/"\([^"]*\)":/[\1]=/g;y/{},/() /' -e "s/='(/=(/;s/)'$/)/"
Output:
dat_absolutenumber='51'
dat_airedepisodenumber='6'
dat_airedseason='4'
dat_airedseasonid='680431'
dat_dvdepisodenumber='6'
dat_dvdseason='4'
dat_episodename='We Will Rise'
dat_firstaired='2017-03-15'
dat_id='5939660'
dat_language=([episodeName]="en" [overview]="en")
dat_lastupdated='1490769062'
dat_overview='Clarke and Roan must work together in hostile territory in order to deliver an invaluable asset to Abby and her team.'