Unable to find match with yq in for loop - bash

I have a yaml file called teams.yml that looks like this:
test:
bonjour: hello
loop:
bonjour: allo
I want to loop over an array and get itsvalues matching a key in the yaml. I have tried yq e 'keys | .[] | select(. == "test")' .github/teams.yml and it returns test whereas yq e 'keys | .[] | select(. == "abc")' .github/teams.yml returns nothing which is enough to get the information I am interested in.
The issue is that, when using the same logic in a for loop, yq returns nothing:
#!/bin/bash
yq e 'keys | .[] | select(. == "abc")' .github/teams.yml # Prints nothing
yq e 'keys | .[] | select(. == "test")' .github/teams.yml # Prints 'test'
ar=( "abc" "test" "xyz" )
for i in "${ar[#]}" # Prints nothing
do
yq e 'keys | .[] | select(. == "$i")' .github/teams.yml
done
What explains the lack of output in the for loop?

Call yq with the loop variable set in the environment, and use env within yq to read environment variables. See also the section Env Variable Operators in the manual.
ar=( "abc" "test" "xyz" )
for i in "${ar[#]}" # Prints nothing
do
i="$i" yq e 'keys | .[] | select(. == env(i))' .github/teams.yml
done
Another way could be to import the (preformatted) array from the environment and just make the array subtractions within yq (saving you the looping and calling yq multiple times):
ar='["abc","test","xyz"]' yq e 'env(ar) - (env(ar) - keys) | .[]' .github/teams.yml

Related

How to get the index of element using jq

I want to get the index of a element from the array.
Link: https://api.github.com/repos/checkstyle/checkstyle/releases
I want to fetch the tag_name.
I generally use this command to get my latest release tag:
LATEST_RELEASE_TAG=$(curl -s https://api.github.com/repos/checkstyle/checkstyle/releases/latest \
| jq ".tag_name")
But now I want previous releases too so I want the index of a particular tag name. For eg: tag_name = "10.3.1"
Also, I am thinking to use mathematical reduction to get the other previous release if I get a particular index number like:
( 'index of 10.3.1' - 1) Any thought regarding this?
Just index of "checkstyle-10.3.1":
curl -s https://api.github.com/repos/checkstyle/checkstyle/releases | jq '[.[].tag_name] | to_entries | .[] | select(.value=="checkstyle-10.3.1") | .key'
Release before "checkstyle-10.3.1":
curl -s https://api.github.com/repos/checkstyle/checkstyle/releases | jq '([.[].tag_name] | to_entries | .[] | select(.value=="checkstyle-10.3.1") | .key) as $index | .[$index+1]'
Release "checkstyle-10.3.1"
curl -s https://api.github.com/repos/checkstyle/checkstyle/releases | jq '.[] | select(.tag_name=="checkstyle-10.3.1")'
Here's a general-purpose function to get the index of something in an array:
# Return the 0-based index of the first item in the array input
# for which f is truthy, else null
def index_by(f):
first(foreach .[] as $x (-1;
.+1;
if ($x|f) then . else empty end)) // null;
This can be used to retrieve the item immediately before a target as follows:
jq '
def index_by(f): first(foreach .[] as $x (-1; .+1; if ($x|f) then . else empty end)) // null;
index_by(.tag_name == "checkstyle-10.3.1") as $i
| if $i then .[$i - 1] else empty end
'

yq v4: print all key value pairs with full key path

I'm trying to determine the correct syntax for using yq to print all key/value pairs from a given yaml input using yq v4 - with the desired output having the full key "path". This was possible using v3 such as this:
$ cat << EOF | yq r -p pv - '**'
> a:
> b: foo
> c: bar
> EOF
a.b: foo
a.c: bar
but I'm having difficulty wrapping my head around the new syntax.
Any help is greatly appreciated.
$ cat << EOF | yq e '.. | select(. == "*") | {(path | join(".")): .} ' -
> a:
> b: foo
> c: bar
> EOF
a.b: foo
a.c: bar
What does this do? Let's go over it:
.. recursively select all values
select(. == "*") filter for scalar values (i.e. filter out the value of a)
(path | join(".")) gets the path as array and joins the elements with .
{…: .} create a mapping, having the joined paths as keys and their values as values
Edit: to get sequence indexes in square brackets ([0] etc), do
$ cat << EOF | yq e '.. | select(. == "*") | {(path | . as $x | (.[] | select((. | tag) == "!!int") |= (["[", ., "]"] | join(""))) | $x | join(".") | sub(".\[", "[")): .} ' -
This seems like there should be a simpler way to do it, but I don't know yq well enough to figure it out.

yq v4 get root keys based on existence of deeper keys

I have this structure:
foo:
image: 123
bar:
image: 456
baz:
config: "my config"
and I'd like to print the root keys (i.e. foo, bar, baz) based on the existence of the child "image"
In yq version 3 I could do this:
$ yq read test.yaml --printMode p "*.image" | awk -F'.' '{print $1}'
foo
bar
But I can't find the equivalent in v4. The yq + jq solution would be:
$ yq -j e test.yaml | jq -r 'to_entries[] | select(.value | has("image")) | [.key][]'
foo
bar
Any idea how to do this with yq v4?
You can use the path operator to get the path of the matching object containing the tag image
yq e '.[] | select(has("image")) | path | .[]' yaml

JQ - Argument list too long error - Large Input

I use Jq to perform some filtering on a large json file using :
paths=$(jq '.paths | to_entries | map(select(.value[].tags | index("Filter"))) | from_entries' input.json)
and write the result to a new file using :
jq --argjson prefix "$paths" '.paths=$prefix' input.json > output.json
But this ^ fails as $paths has a very high line count (order of 100,000).
Error :
jq: Argument list too long
I also went through : /usr/bin/jq: Argument list too long error bash , understood the same problem there, but did not get the solution.
In general, assuming your jq allows it, you could use —argfile or —slurpfile but in your case you can simply avoid the issue by invoking jq just once instead of twice. For example, to keep things clear:
( .paths | to_entries | map(select(.value[].tags | index("Filter"))) | from_entries ) as $prefix
| .paths=$prefix
Even better, simply use |=:
.paths |= ( to_entries | map(select(.value[].tags | index("Filter"))) | from_entries)
or better yet, use with_entries.

Strip a particular word from the beginning of string in jq output

I am getting list of values as below using the curl command:
curl -s http://internal.registry.com/v2/_catalog | jq -r '.repositories[0:5] | to_entries | map( .value )[]'
Output:
centos
containersol/consul-server
containersol/mesos-agent
containersol/mesos-master
cybs/address-api
I want to make sure that output should not have the prefix cybs/ in it. for example, cybs/address-api should just be address-api
Just use sub:
curl ... | jq -r '.repositories[0:5][] | sub("^cybs/"; "")'
Also note that to_entries | map( .value ) is a NOP and should be removed.
Output:
centos
containersol/consul-server
containersol/mesos-agent
containersol/mesos-master
address-api

Resources