How to read a specific data from a yaml file inside a shell script - bash

I have a yaml file say "test.yaml". The below is the content of yaml file.
...
test:
config:
abc: name1
xyz: name2
...
Now I want to read the value of abc and xyz alone from the yaml inside a shell script and store it in two variables inside shell script. test.yaml file contains additional data apart from the above one which I don't need to bother about that inside this shell script.
Eg: test.sh
var1=name1 //test[config[abc]]
var2=name2 //test[config[xyz]]
How do I read specific data (as key-value) from yaml inside a shell script. It would be really helpful if someone helps me out on this. Thanks in advance!!!

Here's an example with yq. All of the following assumes that the values do not contain newlines.
Given
$ cat test.yaml
---
test:
config:
abc: name1
xyz: name2
then
yq e '.test.config | to_entries | map(.value) | .[]' test.yaml
outputs
name1
name2
You can read them into variables like
{ read -r var1; read -r var2; } < <(yq e '.test.config | to_entries | map(.value) | .[]' test.yaml)
declare -p var1 var2
declare -- var1="name1"
declare -- var2="name2"
I would read them into an associative array with the yaml key though:
declare -A conf
while IFS="=" read -r key value; do conf["$key"]=$value; done < <(
yq e '.test.config | to_entries | map([.key, .value] | join("=")) | .[]' test.yaml
)
declare -p conf
declare -A conf=([abc]="name1" [xyz]="name2" )
Then you can write
echo "test config for abc is ${conf[abc]}"
# or
for var in "${!conf[#]}"; do printf "key %s, value %s\n" "$var" "${conf[$var]}"; done
I'm using "the Go implementation"
$ yq --version
yq (https://github.com/mikefarah/yq/) version 4.16.1

Related

Pass bash variable in yq

I am trying to pass bash variable in yq
test.yml
configuration:
Properties:
corporate-url: https://stackoverflow.com/
temp = '.configuration.Properties.corporate-url'
export $temp
Value1=$(yq '.[env($temp)]' test.yml)
expected output:
https://stackoverflow.com/
but I am getting this error(Actual output)
Error: Value for env variable '$variable1' not provided in env()
Please note:
I am trying to fetch corporate-url value, using a bash variable, constraint is that I cannot pass string directly in yq as the value of temp changes as this snippet is running inside a for loop which changes value of temp every time so cannot hard code for a particular value.
Reference YQ Documentation:
https://mikefarah.gitbook.io/yq/operators/env-variable-operators
ApisDraft folder contains multiple yml files
ApisDraft=$(find drafts/* -maxdepth 1 -type f)
for ApiFixOrgsTags in $ApisDraft
do
my_var=$(yq '.securityDefinitions.[].tokenUrl' $ApiFixOrgsTags)
ConfigProper='.configuration.Properties.'
CatelogProper='.configuration.catalogs.[].Properties.'
variable1=$ConfigProper${my_var}
variable2=$CatelogProper${my_var}
# to remove white all spaces
variable1= echo $variable1 | sed -E 's/(\.) */\1/g'
variable2= echo $variable2 | sed -E 's/(\.) */\1/g'
export $variable1
export $variable2
Value1=$(yq "$variable1" $ApiFixOrgsTags)
Value2=$(yq '.[env($variable2)]' $ApiFixOrgsTags)
done
In this case, you don't need to put it in the environment. Let the shell expand it so yq just sees the value of the variable:
yq "$temp" test.yml # => https://stackoverflow.com/

convert yaml to environment variable by selecting a KEY

I have a config file:
name: test
configmap:
foo1: bar1
foo2: bar2
secrets:
secrets1: value1
whatever comes under configmap should be converted to ENV Variables. Like in above example, only 2 values will be converted to ENV variables:
export foo1=bar1
export foo2=bar2
I followed this https://unix.stackexchange.com/questions/539009/export-environment-variables-parsed-from-yaml-text-file
but could not get it working.
You can use a tool like yq to create an output like:
key1=value1 key2=value2
Passing that to export will create the env vars.
export $(yq e '.configmap | to_entries | map(.key + "=" + .value) | join(" ")' input)
Local shell example, were your input is stored in a file called input:
$ echo $foo1 $foo2
$
$ export $(yq e '.configmap | to_entries | map(.key + "=" + .value) | join(" ")' input)
$
$ echo $foo1 $foo2
bar1 bar2
$
Using awk:
While I would recommend using a proper YAML parser as in the answer by #OstoneO, you could use awk to accomplish this task:
$ unset foo1; unset foo2
$ export $(awk -F": " '/configmap:/{f=1} {if(f && NF==2) {printf "export %s=%s ", $1, $2}} /secrets:/{f=0}' test.yaml);
$ echo $foo1 $foo2
bar1 bar2

How can I replace the value inside a yaml file from another yaml file in bash

I would like to replace a single value inside a yaml file (file1.yml) with the value of another yaml file (file2.yml). The key inside both file:
app:
key: 12345
So far here is what I have done using sed with no success:
#!/bin/bash
old_value=`cat file1.yml | grep key | head -n1 | cut -f 2 -d ':'`
new_value=`cat file2.yml | grep key | head -n1 | cut -f 2 -d ':'`
sed "s/$old_value/$new_value/g;" file1.yml
I guess I should not be sending the standard output with sed and should be using awk.
To manipulate yaml files, you should employ a yaml processor, like mikefarah/yq or kislyuk/yq.
Using mikefarah/yq:
new="$(yq '.key' file2.yml)" yq -i '.key = env(new)' file1.yml
Using kislyuk/yq:
yml="$(yq -y -s '.[0].key = .[1].key | .[0]' file1.yml file2.yml)"
cat <<< "$yml" > file1.yml
Because in a yaml file the same value may exist in multiple places you want to use sed to perform a full search and then replace the value you are looking for; or use an specialized tool like yq (a jq wrapper)
For example, this yaml file is valid
app1:
key: "1234"
app2:
key: "1234"
with sed you will run the following to change key: "1234" to key: "5678" in just app2
sed '/^app2:/{n;s/key:.*/key: "5678"/;}' file.yaml
But doing the same using yq using in-place edit would look like:
yq -i -y '.app2.key = "5678"' file.yml

Convery yaml array to string array

I have a yq read command as below,
groups=$(yq read generated/identity-mapping.yaml "iamIdentityMappings.[0].groups")
It reads iamIdentityMappings from below yaml:
iamIdentityMappings:
- groups:
- Appdeployer
- Moregroups
It stores group as below,
- Appdeployer
- Moregroups
But I want to store groups as below.(comma separated values)
groups="Appdeployer","Moregroups"
How to do this in bash?
yq is just a wrapper for jq, which supports CSV output:
$ groups="$(yq -r '.iamIdentifyMappings[0].groups | #csv' generated/identity-mapping.yaml)"
$ echo "$groups"
"Appdeployer","Moregroups"
The yq invocation in your question just causes an error. Note the fixed version.
Use mapfile and format a null delimited list with yq:
mapfile -d '' -t groups < <(
yq -j '.iamIdentityMappings[0].groups[]+"\u0000"' \
generated/identity-mapping.yaml
)
typeset -p groups
Output:
declare -a groups=([0]="Appdeployer" [1]="Moregroups")
And now you can fulfill this second part of your question:
Construct a command based upon a count variable in bash
# Prepare eksctl's arguments into an array
declare -a eksctl_args=(create iamidentitymapping --cluster "$name" --region "$region" --arn "$rolearn" )
# Read the groups from the yml into an array
mapfile -d '' -t groups < <(
yq -j '.iamIdentityMappings[0].groups[]+"\u0000"' \
generated/identity-mapping.yaml
)
# Add arguments per group
for group in "${groups[#]}"; do
eksctl_args+=(--group "$group")
done
# add username argument
eksctl_args+=(--username "$username")
# call eksctl with its arguments
eksctl "${eksctl_args[#]}"
yq 4.16+ now has a built in #csv operator:
yq e '.iamIdentityMappings.[0].groups | #csv' file.yaml
Note that #csv will only wrap values in quotes if needed (e.g. they have a comma).
If you want quotes, then sub then in and join with commas:
yq e '
.iamIdentityMappings.[0].groups |
(.[] |= sub("(.*)", "\"${1}\""))
| join(",")'
Disclaimer: I wrote yq.
yq version 3 is deprecated now and you can achieve the same output using version 4
#!/bin/bash
while IFS= read -r value; do
groups_array+=($value)
done < <(yq eval '.iamIdentityMappings.[0].groups.[]' generated/identity-mapping.yaml)
printf -v comma_seperated '%s,' "${groups_array[#]}"
echo "${comma_seperated%,}"
This code prints the comma seperated values as you wanted

using jq to assign multiple output variables

I am trying to use jq to parse information from the TVDB api. I need to pull a couple of fields and assign the values to variables that I can continue to use in my bash script. I know I can easily assign the output to one variable through bash with variable="$(command)" but I need the output to produce multiple variables and I don't want to make to use multiple commands.
I read this documentation:
https://stedolan.github.io/jq/manual/v1.5/#Advancedfeatures
but I don't know if this relevant to what I am trying to do.
jq '.data' produces the following output:
[
{
"absoluteNumber": 51,
"airedEpisodeNumber": 6,
"airedSeason": 4,
"airedSeasonID": 680431,
"dvdEpisodeNumber": 6,
"dvdSeason": 4,
"episodeName": "We Will Rise",
"firstAired": "2017-03-15",
"id": 5939660,
"language": {
"episodeName": "en",
"overview": "en"
},
"lastUpdated": 1490769062,
"overview": "Clarke and Roan must work together in hostile territory in order to deliver an invaluable asset to Abby and her team."
}
]
I tried jq '.data | {episodeName:$name}' and jq '.data | .episodeName as $name' just to try and get one working. I don't understand the documentation or even if it's what I'm looking for. Is there a way to do what I am trying to do?
You can use separate variables with read :
read var1 var2 var3 < <(echo $(curl -s 'https://api.github.com/repos/torvalds/linux' |
jq -r '.id, .name, .full_name'))
echo "id : $var1"
echo "name : $var2"
echo "full_name : $var3"
Using array :
read -a arr < <(echo $(curl -s 'https://api.github.com/repos/torvalds/linux' |
jq -r '.id, .name, .full_name'))
echo "id : ${arr[0]}"
echo "name : ${arr[1]}"
echo "full_name : ${arr[2]}"
Also you can split jq output with some character :
IFS='|' read var1 var2 var3 var4 < <(curl '......' | jq -r '.data |
map([.absoluteNumber, .airedEpisodeNumber, .episodeName, .overview] |
join("|")) | join("\n")')
Or use an array like :
set -f; IFS='|' data=($(curl '......' | jq -r '.data |
map([.absoluteNumber, .airedEpisodeNumber, .episodeName, .overview] |
join("|")) | join("\n")')); set +f
absoluteNumber, airedEpisodeNumber, episodeName & overview are respectively ${data[0]}, ${data[1]}, ${data[2]}, ${data[3]}. set -f and set +f are used to respectively disable & enable globbing.
For the jq part, all your required fields are mapped and delimited with a '|' character with join("|")
If your are using jq < 1.5, you'll have to convert Number to String with tostring for each Number fields eg:
IFS='|' read var1 var2 var3 var4 < <(curl '......' | jq -r '.data |
map([.absoluteNumber|tostring, .airedEpisodeNumber|tostring, .episodeName, .overview] |
join("|")) | join("\n")')
jq always produces a stream of zero or more values. For example, to produce the two values corresponding to "episodeName" and "id"' you could write:
.data[] | ( .episodeName, .id )
For your purposes, it might be helpful to use the -c command-line option, to ensure each JSON output value is presented on a single line. You might also want to use the -r command-line option, which removes the outermost quotation marks from each output value that is a JSON string.
For further variations, please see the jq FAQ https://github.com/stedolan/jq/wiki/FAQ, e.g. the question:
Q: How can a stream of JSON texts produced by jq be converted into a bash array of corresponding values?
Experimental conversion of quoted OP input, (tv.dat), to a series of bash variables, (and an array). The jq code is mostly borrowed from here and there, but I don't know how to get jq to unroll an array within an array, so the sed code does that, (that's only good for one level, but so are bash arrays):
jq -r ".[] | to_entries | map(\"DAT_\(.key) \(.value|tostring)\") | .[]" tv.dat |
while read a b ; do echo "${a,,}='$b'" ; done |
sed -e '/{.*}/s/"\([^"]*\)":/[\1]=/g;y/{},/() /' -e "s/='(/=(/;s/)'$/)/"
Output:
dat_absolutenumber='51'
dat_airedepisodenumber='6'
dat_airedseason='4'
dat_airedseasonid='680431'
dat_dvdepisodenumber='6'
dat_dvdseason='4'
dat_episodename='We Will Rise'
dat_firstaired='2017-03-15'
dat_id='5939660'
dat_language=([episodeName]="en" [overview]="en")
dat_lastupdated='1490769062'
dat_overview='Clarke and Roan must work together in hostile territory in order to deliver an invaluable asset to Abby and her team.'

Resources