I would like to replace a single value inside a yaml file (file1.yml) with the value of another yaml file (file2.yml). The key inside both file:
app:
key: 12345
So far here is what I have done using sed with no success:
#!/bin/bash
old_value=`cat file1.yml | grep key | head -n1 | cut -f 2 -d ':'`
new_value=`cat file2.yml | grep key | head -n1 | cut -f 2 -d ':'`
sed "s/$old_value/$new_value/g;" file1.yml
I guess I should not be sending the standard output with sed and should be using awk.
To manipulate yaml files, you should employ a yaml processor, like mikefarah/yq or kislyuk/yq.
Using mikefarah/yq:
new="$(yq '.key' file2.yml)" yq -i '.key = env(new)' file1.yml
Using kislyuk/yq:
yml="$(yq -y -s '.[0].key = .[1].key | .[0]' file1.yml file2.yml)"
cat <<< "$yml" > file1.yml
Because in a yaml file the same value may exist in multiple places you want to use sed to perform a full search and then replace the value you are looking for; or use an specialized tool like yq (a jq wrapper)
For example, this yaml file is valid
app1:
key: "1234"
app2:
key: "1234"
with sed you will run the following to change key: "1234" to key: "5678" in just app2
sed '/^app2:/{n;s/key:.*/key: "5678"/;}' file.yaml
But doing the same using yq using in-place edit would look like:
yq -i -y '.app2.key = "5678"' file.yml
Related
I have a source file that looks like
{
"admin-user" : {
"index.md": "Index",
"user_profil.md": "User Profil"
}
}
By help of bash and jq I like to concat a string of the second level keys index.md and user_profile.md. Further I like replace .md with .html (key admin-user is unknown an can change)
This is where I hang:
KEYS=$(cat test.json | jq -r '.[]|keys')
concat_string=""
for i in $KEYS
do
md_name=${i/.md/.html}
concat_string="$concat_string$md_name"
done
echo $concat_string
Result:
["index.html","user_profil.html"]
So the result is an array. How can I concat a string with blanks between strings?
All of it can be done from within jq:
map(keys_unsorted[] | sub("\\.md$"; ".html"))
Demo
Alternatively, you can use to_entries to access each .key:
map(to_entries[].key | sub("\\.md$"; ".html"))
Demo
Both will give you an array of strings
["index.html","user_profil.html"]
which you can then -still in jq- concatenate using join and a glue character:
jq -r 'map(keys_unsorted[] | sub("\\.md$"; ".html")) | join(" ")' test.json
Demo
or using the alternative approach:
jq -r 'map(to_entries[].key | sub("\\.md$"; ".html")) | join(" ")' test.json
Demo
Note: Using keys instead of keys_unsorted (as you did in your attempt) will additionally sort the keys.
I have a yaml file say "test.yaml". The below is the content of yaml file.
...
test:
config:
abc: name1
xyz: name2
...
Now I want to read the value of abc and xyz alone from the yaml inside a shell script and store it in two variables inside shell script. test.yaml file contains additional data apart from the above one which I don't need to bother about that inside this shell script.
Eg: test.sh
var1=name1 //test[config[abc]]
var2=name2 //test[config[xyz]]
How do I read specific data (as key-value) from yaml inside a shell script. It would be really helpful if someone helps me out on this. Thanks in advance!!!
Here's an example with yq. All of the following assumes that the values do not contain newlines.
Given
$ cat test.yaml
---
test:
config:
abc: name1
xyz: name2
then
yq e '.test.config | to_entries | map(.value) | .[]' test.yaml
outputs
name1
name2
You can read them into variables like
{ read -r var1; read -r var2; } < <(yq e '.test.config | to_entries | map(.value) | .[]' test.yaml)
declare -p var1 var2
declare -- var1="name1"
declare -- var2="name2"
I would read them into an associative array with the yaml key though:
declare -A conf
while IFS="=" read -r key value; do conf["$key"]=$value; done < <(
yq e '.test.config | to_entries | map([.key, .value] | join("=")) | .[]' test.yaml
)
declare -p conf
declare -A conf=([abc]="name1" [xyz]="name2" )
Then you can write
echo "test config for abc is ${conf[abc]}"
# or
for var in "${!conf[#]}"; do printf "key %s, value %s\n" "$var" "${conf[$var]}"; done
I'm using "the Go implementation"
$ yq --version
yq (https://github.com/mikefarah/yq/) version 4.16.1
I have a yq read command as below,
groups=$(yq read generated/identity-mapping.yaml "iamIdentityMappings.[0].groups")
It reads iamIdentityMappings from below yaml:
iamIdentityMappings:
- groups:
- Appdeployer
- Moregroups
It stores group as below,
- Appdeployer
- Moregroups
But I want to store groups as below.(comma separated values)
groups="Appdeployer","Moregroups"
How to do this in bash?
yq is just a wrapper for jq, which supports CSV output:
$ groups="$(yq -r '.iamIdentifyMappings[0].groups | #csv' generated/identity-mapping.yaml)"
$ echo "$groups"
"Appdeployer","Moregroups"
The yq invocation in your question just causes an error. Note the fixed version.
Use mapfile and format a null delimited list with yq:
mapfile -d '' -t groups < <(
yq -j '.iamIdentityMappings[0].groups[]+"\u0000"' \
generated/identity-mapping.yaml
)
typeset -p groups
Output:
declare -a groups=([0]="Appdeployer" [1]="Moregroups")
And now you can fulfill this second part of your question:
Construct a command based upon a count variable in bash
# Prepare eksctl's arguments into an array
declare -a eksctl_args=(create iamidentitymapping --cluster "$name" --region "$region" --arn "$rolearn" )
# Read the groups from the yml into an array
mapfile -d '' -t groups < <(
yq -j '.iamIdentityMappings[0].groups[]+"\u0000"' \
generated/identity-mapping.yaml
)
# Add arguments per group
for group in "${groups[#]}"; do
eksctl_args+=(--group "$group")
done
# add username argument
eksctl_args+=(--username "$username")
# call eksctl with its arguments
eksctl "${eksctl_args[#]}"
yq 4.16+ now has a built in #csv operator:
yq e '.iamIdentityMappings.[0].groups | #csv' file.yaml
Note that #csv will only wrap values in quotes if needed (e.g. they have a comma).
If you want quotes, then sub then in and join with commas:
yq e '
.iamIdentityMappings.[0].groups |
(.[] |= sub("(.*)", "\"${1}\""))
| join(",")'
Disclaimer: I wrote yq.
yq version 3 is deprecated now and you can achieve the same output using version 4
#!/bin/bash
while IFS= read -r value; do
groups_array+=($value)
done < <(yq eval '.iamIdentityMappings.[0].groups.[]' generated/identity-mapping.yaml)
printf -v comma_seperated '%s,' "${groups_array[#]}"
echo "${comma_seperated%,}"
This code prints the comma seperated values as you wanted
I am using JQ module the parse some of the data and then running the final loop over it to parse few more data.
cluster_list=`databricks --profile hq_dev clusters list --output JSON | jq 'select(.clusters != null) | .clusters[] | [.cluster_name,.autotermination_minutes,.state,.cluster_id] | #csv' | grep -v "job-"`
for cluster in ${cluster_list[#]}
do
cluster_id=`echo $cluster| cut -d "," -f 4 | sed 's/\"//g' | sed 's/\\\//g'`
cluster_name=`echo "${cluster}"| cut -d "," -f 1| sed 's/\"//g' | sed 's/\\\//g'`
echo $cluster_name
done
cluster_list contains following value.
"\"Test Space Cluster\",15,\"TERMINATED\",\"ddd-dese23-can858\""
"\"GatewayCluster\",15,\"TERMINATED\",\"ddd-ddsd-ddsds\""
"\"delete_later\",15,\"TERMINATED\",\"1120-195800-93839\""
"\"GatewayCluster_old\",15,\"TERMINATED\",\"0108-2y7272-393893\""
it prints following.
Test
Space
Cluster
GatewayCluster
delete_later
GatewayCluster_old
Desired output
it shouldn't break to newline if there is a space, I am doing few more action by the name I am getting here.
Test Space Cluster
GatewayCluster
delete_later
GatewayCluster_old
Your script seems a bit overly complex to achieve your goal. Better use read to store each value in a separate variable, and set a comma for the input field separator IFS:
databricks --profile hq_dev clusters list --output JSON |
jq 'select(.clusters != null) | .clusters[] |
[.cluster_name,.autotermination_minutes,.state,.cluster_id] | #csv' |
grep -v "job-" |
sed 's/\\\?"//g' |
while IFS=, read name autotermination_minutes state id ; do
echo $name
done
Note: I didn't touch your jq command. The sed line I put aims to remove quotes, protected or not. You can tune jq to remove these quotes with -r, as said in the man page:
INVOKING JQ
[...]
--raw-output / -r::
With this option, if the filter´s result is a string then it will be written directly to standard output rather than being formatted as a JSON string with quotes. This can be useful for making jq filters talk to non-JSON-based systems.
Looking for a one liner with csvkit.
From a plain json object
{
"whatever": 2342,
"otherwise": 119,
"and": 1,
"so": 2,
"on": 3
}
Want this csv
whatever,2342
otherwise,119
and,1
so,2
on,3
I basically want this command to work, but it doesn't.
echo $the_json | in2csv -f json
> When converting a JSON document with a top-level dictionary element, a key must be specified.
Seems like something csvkit can do, and I just haven't found the right options.
short answer
variant A: in2csv (csvkit) + csvtool
wrap your json in brackets
use in2csv's -I option to avoid unexpected behavior
use a command to transpose the two-row CSV, e.g. csvtool
echo "[$the_json]" | in2csv -I -f json | csvtool transpose -
variant B: use jq instead
This is a solution using only jq: (https://stedolan.github.io/jq/)
echo "$the_json" | jq -r 'to_entries[] | [.key, .value] | #csv'
taken from How to map an object to arrays so it can be converted to csv?
long answer (csvkit + csvtool)
the input
in2csv -f json expects a list of JSON objects, so you need to wrap the single object ({...}) into square brackets ([{...}]).
On POSIX compatible shells, write
echo "[$the_json]"
which will print
[{
"whatever": 2342,
"otherwise": 119,
"and": 1,
"so": 2,
"on": 3
}]
the csvkit command
You may pipe the above data directly into in2csv. However, you might run into issues with the ”type inference“ (CSV data interpretation) feature of csvkit:
$ echo "[$the_json]" | in2csv -f json
whatever,otherwise,and,so,on
2342,119,True,2,3
1 has become True. For details, see the Tips and Troubleshooting part of the docs. It's suggested to turn off type inference using the -I option:
$ echo "[$the_json]" | in2csv -I -f json
whatever,otherwise,and,so,on
2342,119,1,2,3
Now the result is as expected
transpose the data
Still, you need to transpose the data. The csvkit docs say:
To transpose CSVs, consider csvtool.
(csvtool is available on github, opam, debian and probably other distribution channels.)
Using csvkit + csvtool, your final command looks like this:
echo "[$the_json]" | in2csv -I -f json | csvtool transpose -
with the hyphen (-) meaning to take the data from stdin. This is the result:
whatever,2342
otherwise,119
and,1
so,2
on,3
that's it.
I think there is no one-liner solution with csvtool only, you'll need in2csv. You may, however, use jq instead, see the short answer.
FTR, I'm using csvkit version 1.0.3.
Tested the first posted answer works! But it is a bit confusing because "[$the_json]" means the raw content of the json. So an example of command could be this:
echo '[{"a":"b","c":"d"}]' | in2csv -I -f json | csvtool transpose -
and if you want to do it with a file name instead, for instance myfile.json one can add the brackets with a sed command and pipe it to in2csv:
sed -e '1s/^/[/' -e 's/$/,/' -e '$s/,$/]/' myfile.json | in2csv -I -f json > myfile.csv
Example with the full transposition command:
sed -e '1s/^/[/' -e 's/$/,/' -e '$s/,$/]/' myfile.json | in2csv -I -f json | csvtool transpose - > myfile.csv
source: How to add bracket at beginning and ending in text on UNIX