Let say I have YAML file that looks like this:
FOO: somefoo
BAR: somebar
I would like to convert this (using yq) into the following so that I can source the contents into environment variables:
export BAR='somebar'
export FOO='somefoo'
I can do it it with jq by converting the input to JSON first, but I can't seem to figure out how to do it with yq only. (I am using yq 4.x, <4.18).
So, concretely, how could I do the following using just yq?
INPUT="FOO: somefoo
BAR: somebar"
echo "$INPUT" | yq e 'to_json' - | jq -r 'keys[] as $k | "export \($k)='\''\(.[$k])'\''"'
You could switch to kislyuk's yq which uses native jq under the hood. Then, you would just need to_entries to access key and value, string interpolation in combination with the -r flag to produce the output, and #sh to escape for shell compliance:
yq -r 'to_entries[] | "export \(.key)=\(.value | #sh)"'
export FOO='somefoo'
export BAR='somebar'
Use the key operator and string concatenation:
$ echo "$INPUT" | yq $'.[] | "export " + key + "=\'" + . + "\'"'
export FOO='somefoo'
export BAR='somebar'
Tested with yq 4.27.2
The #sh operator has been added in yq v4.31.1 (with my humble contribution). Now you can do it pretty much the same way as in jq:
yq '.[] | "export " + key + "=" + #sh'
The quoting algorithm is a bit different from jq as it starts to quote only at characters that need quoting, so the literal output will likely differ, but will be later parsed equally.
# input
FOO: somefoo
BAR: somebar and some
# result
export FOO=somefoo
export BAR=somebar' and some'
With older yq versions you can still implement a primitive but safe quoting algorithm using other yq functions (the quoting is a little lovely nightmare, though):
# POSIX shell quoting starting "
yq ".[] | \"export \" + key + \"='\" + sub(\"'\", \"'\''\") + \"'\""
# POSIX shell quoting starting '
yq '.[] | "export " + key + "='\''" + sub("'\''", "'\'\\\'\''") + "'\''"'
# BASH "dollar" quoting
yq $'.[] | "export " + key + "=\'" + sub("\'", "\'\\\'\'") + "\'"'
Actually this is exactly what jq does with its #sh. In all cases this ends up as:
export FOO='somefoo'
export BAR='somebar and some'
Related
I have a yq read command as below,
groups=$(yq read generated/identity-mapping.yaml "iamIdentityMappings.[0].groups")
It reads iamIdentityMappings from below yaml:
iamIdentityMappings:
- groups:
- Appdeployer
- Moregroups
It stores group as below,
- Appdeployer
- Moregroups
But I want to store groups as below.(comma separated values)
groups="Appdeployer","Moregroups"
How to do this in bash?
yq is just a wrapper for jq, which supports CSV output:
$ groups="$(yq -r '.iamIdentifyMappings[0].groups | #csv' generated/identity-mapping.yaml)"
$ echo "$groups"
"Appdeployer","Moregroups"
The yq invocation in your question just causes an error. Note the fixed version.
Use mapfile and format a null delimited list with yq:
mapfile -d '' -t groups < <(
yq -j '.iamIdentityMappings[0].groups[]+"\u0000"' \
generated/identity-mapping.yaml
)
typeset -p groups
Output:
declare -a groups=([0]="Appdeployer" [1]="Moregroups")
And now you can fulfill this second part of your question:
Construct a command based upon a count variable in bash
# Prepare eksctl's arguments into an array
declare -a eksctl_args=(create iamidentitymapping --cluster "$name" --region "$region" --arn "$rolearn" )
# Read the groups from the yml into an array
mapfile -d '' -t groups < <(
yq -j '.iamIdentityMappings[0].groups[]+"\u0000"' \
generated/identity-mapping.yaml
)
# Add arguments per group
for group in "${groups[#]}"; do
eksctl_args+=(--group "$group")
done
# add username argument
eksctl_args+=(--username "$username")
# call eksctl with its arguments
eksctl "${eksctl_args[#]}"
yq 4.16+ now has a built in #csv operator:
yq e '.iamIdentityMappings.[0].groups | #csv' file.yaml
Note that #csv will only wrap values in quotes if needed (e.g. they have a comma).
If you want quotes, then sub then in and join with commas:
yq e '
.iamIdentityMappings.[0].groups |
(.[] |= sub("(.*)", "\"${1}\""))
| join(",")'
Disclaimer: I wrote yq.
yq version 3 is deprecated now and you can achieve the same output using version 4
#!/bin/bash
while IFS= read -r value; do
groups_array+=($value)
done < <(yq eval '.iamIdentityMappings.[0].groups.[]' generated/identity-mapping.yaml)
printf -v comma_seperated '%s,' "${groups_array[#]}"
echo "${comma_seperated%,}"
This code prints the comma seperated values as you wanted
How can we mix different input sources when using jq ?
For a specific usecase, I'd like to add some data from a file into a feed that was pipe in stdout.
$ echo '[{"a": 1}]' > /tmp/a1
$ echo '[{"a": 2}]' > /tmp/a2
$ jq --slurp '.[0] + .[1]' /tmp/a1 /tmp/a2
[
{
"a": 1
},
{
"a": 2
}
]
$ cat /tmp/a1 | jq --slurp '.[0] + .[1]' /tmp/a2 # Expecting the same result
[
{
"a": 2
}
]
As you can see, the last command didn't interpret the piped data.
Right now, I'm forced to save the output from the first operation into a temporary file, so that I can do the jq merging operation, before sending it back to the network. Having a single stream would be much more efficient
I'd like to add some data from a file into a feed that was pipe in stdout.
There are various ways to do this, depending on the shell and also the version of jq you are using.
Assuming your jq supports the --argfile option, you might find that quite congenial:
cat /tmp/a1 | jq --argfile a2 /tmp/a2 '. + $a2'
Here is another variation that suggests some of the other possibilities:
jq -n --argfile a1 <(cat /tmp/a1) --argfile a2 <(cat /tmp/a2) '$a1 + $a2'
More interestingly:
(cat /tmp/a1 ; cat /tmp/a2) | jq '. + input'
You might also wish to consider using the --slurpfile option instead of --argfile, but note that --slurpfile always "slurps" the file.
And finally an approach that should work for every version of jq:
jq -s '.[0] + .[1]' <(cat /tmp/a1) /tmp/a2
In general, though, it's best to avoid the -s option.
A note on slurping
If you compare the outputs produced by:
echo '1 2' |
jq -s --debug-dump-disasm --debug-trace '.[0], .[1]'
and
echo '1 2' |
jq --debug-dump-disasm --debug-trace '., input'
you'll notice the former has to PUSHK_UNDER to store the entire array [1,2],
whereas the latter program just reads the two inputs separately.
In the first program, the memory for the array cannot be freed until after
all the pointers into it have been processed, whereas in the second program,
the memory for . can be freed after the first RET.
You could do this, where cat forwards its stdin followed by a2:
<GENERATE a1> | cat - /tmp/a2 | jq --slurp '.[0] + .[1]'
Or this, which is a compound statement passing the results of two separate commands into a pipe:
{ <GENERATE a1> ; cat /tmp/a2; } | jq --slurp '.[0] + .[1]'
Take care to have spaces beside the curly braces and to have a semi-colon before the final one.
Completing peak's answer, you actually don't need redirections:
jq -n --argfile a1 /tmp/a1 --argfile a2 /tmp/a2 '$a1 + $a2'
Given the following input:
J='{"a":1,"b":10,"c":100}
{"a":2,"b":20,"c":200}
{"a":3,"b":30,"c":300}'
The command
SELECT='a,b'; echo $J | jq -c -s --arg P1 $SELECT '.[]|{a,b}'
produces
{"a":1,"b":10}
{"a":2,"b":20}
{"a":3,"b":30}
but this command produces unexpected results:
SELECT='a,b'; echo $J | jq -c -s --arg P1 $SELECT '.[]|{$P1}'
{"P1":"a,b"}
{"P1":"a,b"}
{"P1":"a,b"}
How does one get jq to treat an arg string literally?
Using tostring gives an error
SELECT='a,b'; echo $J | jq -c -s --arg P1 $SELECT '.[]|{$P1|tostring}'
jq: error: syntax error, unexpected '|', expecting '}' (Unix shell quoting
issues?) at <top-level>, line 1:
.[]|{$SELECT|tostring}
jq: 1 compile error
SELECT needs to be a variable and not hardcoded in the script.
SELECT needs to be a variable and not hardcoded in the script.
Assuming you want to avoid the risks of "code injection" and that you want the shell variable SELECT to be a simple string such as "a,b", then consider this reduce-free solution along the lines you were attempting:
J='{"a":1,"b":10,"c":100}'
SELECT='a,b'
echo "$J" |
jq -c --arg P1 "$SELECT" '
. as $in | $P1 | split(",") | map( {(.): $in[.]} ) | add'
Output:
{"a":1,"b":10}
If you really want your data to be parsed as syntax...
This is not an appropriate use case for --arg. Instead, substitute into the code:
select='a,b'; jq -c -s '.[]|{'"$select"'}' <<<"$j"
Note that this has all the usual caveats of code injection: If the input is uncontrolled, the output (or other behavior of the script, particularly if jq gains more capable I/O features in the future) should be considered likewise.
If you want to split the literal string into a list of keys...
Here, we take your select_str (of the form a,b), and generate a map: {'a': 'a', 'b': 'b'}; then, we can break each data item into entries, select only the items in the map, and there's our output.
jq --arg select_str "$select" '
($select_str
| split(",")
| reduce .[] as $item ({}; .[$item]=$item)) as $select_map
| with_entries(select($select_map[.key]))' <<<"$j"
I am trying to use jq to parse information from the TVDB api. I need to pull a couple of fields and assign the values to variables that I can continue to use in my bash script. I know I can easily assign the output to one variable through bash with variable="$(command)" but I need the output to produce multiple variables and I don't want to make to use multiple commands.
I read this documentation:
https://stedolan.github.io/jq/manual/v1.5/#Advancedfeatures
but I don't know if this relevant to what I am trying to do.
jq '.data' produces the following output:
[
{
"absoluteNumber": 51,
"airedEpisodeNumber": 6,
"airedSeason": 4,
"airedSeasonID": 680431,
"dvdEpisodeNumber": 6,
"dvdSeason": 4,
"episodeName": "We Will Rise",
"firstAired": "2017-03-15",
"id": 5939660,
"language": {
"episodeName": "en",
"overview": "en"
},
"lastUpdated": 1490769062,
"overview": "Clarke and Roan must work together in hostile territory in order to deliver an invaluable asset to Abby and her team."
}
]
I tried jq '.data | {episodeName:$name}' and jq '.data | .episodeName as $name' just to try and get one working. I don't understand the documentation or even if it's what I'm looking for. Is there a way to do what I am trying to do?
You can use separate variables with read :
read var1 var2 var3 < <(echo $(curl -s 'https://api.github.com/repos/torvalds/linux' |
jq -r '.id, .name, .full_name'))
echo "id : $var1"
echo "name : $var2"
echo "full_name : $var3"
Using array :
read -a arr < <(echo $(curl -s 'https://api.github.com/repos/torvalds/linux' |
jq -r '.id, .name, .full_name'))
echo "id : ${arr[0]}"
echo "name : ${arr[1]}"
echo "full_name : ${arr[2]}"
Also you can split jq output with some character :
IFS='|' read var1 var2 var3 var4 < <(curl '......' | jq -r '.data |
map([.absoluteNumber, .airedEpisodeNumber, .episodeName, .overview] |
join("|")) | join("\n")')
Or use an array like :
set -f; IFS='|' data=($(curl '......' | jq -r '.data |
map([.absoluteNumber, .airedEpisodeNumber, .episodeName, .overview] |
join("|")) | join("\n")')); set +f
absoluteNumber, airedEpisodeNumber, episodeName & overview are respectively ${data[0]}, ${data[1]}, ${data[2]}, ${data[3]}. set -f and set +f are used to respectively disable & enable globbing.
For the jq part, all your required fields are mapped and delimited with a '|' character with join("|")
If your are using jq < 1.5, you'll have to convert Number to String with tostring for each Number fields eg:
IFS='|' read var1 var2 var3 var4 < <(curl '......' | jq -r '.data |
map([.absoluteNumber|tostring, .airedEpisodeNumber|tostring, .episodeName, .overview] |
join("|")) | join("\n")')
jq always produces a stream of zero or more values. For example, to produce the two values corresponding to "episodeName" and "id"' you could write:
.data[] | ( .episodeName, .id )
For your purposes, it might be helpful to use the -c command-line option, to ensure each JSON output value is presented on a single line. You might also want to use the -r command-line option, which removes the outermost quotation marks from each output value that is a JSON string.
For further variations, please see the jq FAQ https://github.com/stedolan/jq/wiki/FAQ, e.g. the question:
Q: How can a stream of JSON texts produced by jq be converted into a bash array of corresponding values?
Experimental conversion of quoted OP input, (tv.dat), to a series of bash variables, (and an array). The jq code is mostly borrowed from here and there, but I don't know how to get jq to unroll an array within an array, so the sed code does that, (that's only good for one level, but so are bash arrays):
jq -r ".[] | to_entries | map(\"DAT_\(.key) \(.value|tostring)\") | .[]" tv.dat |
while read a b ; do echo "${a,,}='$b'" ; done |
sed -e '/{.*}/s/"\([^"]*\)":/[\1]=/g;y/{},/() /' -e "s/='(/=(/;s/)'$/)/"
Output:
dat_absolutenumber='51'
dat_airedepisodenumber='6'
dat_airedseason='4'
dat_airedseasonid='680431'
dat_dvdepisodenumber='6'
dat_dvdseason='4'
dat_episodename='We Will Rise'
dat_firstaired='2017-03-15'
dat_id='5939660'
dat_language=([episodeName]="en" [overview]="en")
dat_lastupdated='1490769062'
dat_overview='Clarke and Roan must work together in hostile territory in order to deliver an invaluable asset to Abby and her team.'
I want to make a script that is looking for special numbers.
numbers like this 153 = 1^3+5^3+3^3
bash script 153 3
153
In my script I have this kinda thing
echo "$1" | tr -d " " | sed -e 's/\([[:digit:]]\)/\1+/g' | tr '+' '^"$2"+'
That last command doesn't work, it does change something, it changes 1+5+3+ to 1^+5^+3^+
So my question is: how can I use variables in tr?
tr replaces one character with another one. It can't replace one character with a longer string. That's sed's job:
set -- 153 3
echo "$1" | \
tr -d " " | \
sed -e 's/\([[:digit:]]\)/\1^'"$2"'+/g; s/\+$//'
The answer by choroba is correct. Here is a python based one-liner:
$ set -- 153 3
$ python -c "print '+'.join([x+'^$2' for x in list('$1')])"
1^3+5^3+3^3
Explanation:
list will convert the string "153" to ['1', '5', '3']
[ x+'^$2' for x in <list> ] is called list comprehension. Effectively it returns another list: ['1^3', '5^3', '3^3']
Then join them with '+'
NOTE: Only reason I added this answer was because, this does not require to adjust the completed string after processing by build-in functions.
Below are the other common approaches:
$ python -c "print '^$2+'.join(list('$1')) + '^$2'" # Add "^3" after join returns "1^3+5^3+3"
$ echo $1 | sed "s/./&^$2+/g; s/+$//" # Remove last '+' sign from "1^3+5^3+3^3+"