I am using the following command to list all the images within my private registry:
curl -s http://internal.private.registry.com/v2/_catalog | jq -r '.repositories[0:2] | to_entries | map( .value )[]'
Output:
centos
containersol/consul-server
I am using the following command to list all the tags of a particular image within my private registry:
curl -s GET http://internal.private.registry.com/v2/centos/tags/list | jq -r '.tags | to_entries | map( .value )[]'
Output:
6.6
6
7.1.1503
Now, I am trying to list all tags for every image present in catalog using the following bash script:
#!/bin/sh
image_name=$(curl -s http://internal.private.registry.com/v2/_catalog | jq -r '.repositories[0:2] | to_entries | map( .value )[]')
while read -r line; do
${line}$_image_taglist=$(curl -s GET http://internal.private.registry.com/v2/cybs/${line}/tags/list | jq -r '.tags | to_entries | map( .value )[]')
while read -r tag; do
echo "$tag"
done <<< "${line}$_image_taglist"
done <<< "$image_name"
However, I am getting the following error while executing ${line}$_image_taglist=$(curl -s GET http://internal.private.registry.com/v2/cybs/${line}/tags/list | jq -r '.tags | to_entries | map( .value )[]') :
jq: error (at <stdin>:1): null (null) has no keys
Also, please note that I am expecting that dynamic variable ${line}$_image_taglist should resolve to centos_image_taglist and containersol/consul-server_image_taglist
jq: error (at :1): null (null) has no keys
Is caused by . being null. In this case . refers to stdin which means that you get no response from http://internal.private.registry.com/v2/cybs/${line}/tags/list.
You cannot define dynamic named variables like this: ${line}$_image_taglist. You should use an associative array if your bash version supports it:
declare -A arr
arr["$line"]=$(curl ... | jq ...)
But in your case I can't really see why you need it, since you are just iteration over it. Why not use a pipe or process substitution?
while read -r tag; do
echo "$tag"
done < <(curl ... | jq ...)
Related
I have a source file that looks like
{
"admin-user" : {
"index.md": "Index",
"user_profil.md": "User Profil"
}
}
By help of bash and jq I like to concat a string of the second level keys index.md and user_profile.md. Further I like replace .md with .html (key admin-user is unknown an can change)
This is where I hang:
KEYS=$(cat test.json | jq -r '.[]|keys')
concat_string=""
for i in $KEYS
do
md_name=${i/.md/.html}
concat_string="$concat_string$md_name"
done
echo $concat_string
Result:
["index.html","user_profil.html"]
So the result is an array. How can I concat a string with blanks between strings?
All of it can be done from within jq:
map(keys_unsorted[] | sub("\\.md$"; ".html"))
Demo
Alternatively, you can use to_entries to access each .key:
map(to_entries[].key | sub("\\.md$"; ".html"))
Demo
Both will give you an array of strings
["index.html","user_profil.html"]
which you can then -still in jq- concatenate using join and a glue character:
jq -r 'map(keys_unsorted[] | sub("\\.md$"; ".html")) | join(" ")' test.json
Demo
or using the alternative approach:
jq -r 'map(to_entries[].key | sub("\\.md$"; ".html")) | join(" ")' test.json
Demo
Note: Using keys instead of keys_unsorted (as you did in your attempt) will additionally sort the keys.
I have a number of ec2 instances running in AWS, and I've extracted this information into a file.
aws ec2 describe-instances > instances.json
I also have another file ipAddressList
cat ipAddressList
10.100.39.4
10.100.56.20
10.100.78.11
10.100.78.12
I would like to extract the ImageId for these 4 instances.
I'm able to get the ImageId for individual ip addresses using this command
cat instances.json | jq '.Reservations[] | .Instances[] | select(.PrivateIpAddress == "10.100.39.41") | .ImageId'
But I would like to put this into a bash loop to extract the ImageId's for all 4 instances at once.
I've tried
for i in `cat ipAddressList` ; do jq '.Reservations[] | .Instances[] | select(.PrivateIpAddress == \$i) | .ImageId' instances.json ; done
But it throws an error.
What am I doing wrong please?
Don't inject data into code, use the --arg and --argjson options provided by jq to do that safely:
for i in `cat ipAddressList`
do jq --arg i "$i" '
.Reservations[] | .Instances[]
| select(.PrivateIpAddress == $i) | .ImageId
' instances.json
done
On top of that, jq provides options to read in files as raw text, so you could shift the entire loop into jq logic, resulting in just one invocation of jq:
jq --rawfile ips ipAddressList '
.Reservations[].Instances[]
| select(IN(.PrivateIpAddress; ($ips / "\n")[])).ImageId
' instances.json
You're really close with your solution.
What you need is
for i in `cat ipAddressList` ; do jq '.Reservations[] | .Instances[] | select(.PrivateIpAddress == "'$i'") | .ImageId' instances.json ; done
And you should be fine.
I am using JQ module the parse some of the data and then running the final loop over it to parse few more data.
cluster_list=`databricks --profile hq_dev clusters list --output JSON | jq 'select(.clusters != null) | .clusters[] | [.cluster_name,.autotermination_minutes,.state,.cluster_id] | #csv' | grep -v "job-"`
for cluster in ${cluster_list[#]}
do
cluster_id=`echo $cluster| cut -d "," -f 4 | sed 's/\"//g' | sed 's/\\\//g'`
cluster_name=`echo "${cluster}"| cut -d "," -f 1| sed 's/\"//g' | sed 's/\\\//g'`
echo $cluster_name
done
cluster_list contains following value.
"\"Test Space Cluster\",15,\"TERMINATED\",\"ddd-dese23-can858\""
"\"GatewayCluster\",15,\"TERMINATED\",\"ddd-ddsd-ddsds\""
"\"delete_later\",15,\"TERMINATED\",\"1120-195800-93839\""
"\"GatewayCluster_old\",15,\"TERMINATED\",\"0108-2y7272-393893\""
it prints following.
Test
Space
Cluster
GatewayCluster
delete_later
GatewayCluster_old
Desired output
it shouldn't break to newline if there is a space, I am doing few more action by the name I am getting here.
Test Space Cluster
GatewayCluster
delete_later
GatewayCluster_old
Your script seems a bit overly complex to achieve your goal. Better use read to store each value in a separate variable, and set a comma for the input field separator IFS:
databricks --profile hq_dev clusters list --output JSON |
jq 'select(.clusters != null) | .clusters[] |
[.cluster_name,.autotermination_minutes,.state,.cluster_id] | #csv' |
grep -v "job-" |
sed 's/\\\?"//g' |
while IFS=, read name autotermination_minutes state id ; do
echo $name
done
Note: I didn't touch your jq command. The sed line I put aims to remove quotes, protected or not. You can tune jq to remove these quotes with -r, as said in the man page:
INVOKING JQ
[...]
--raw-output / -r::
With this option, if the filter´s result is a string then it will be written directly to standard output rather than being formatted as a JSON string with quotes. This can be useful for making jq filters talk to non-JSON-based systems.
Given the following input:
J='{"a":1,"b":10,"c":100}
{"a":2,"b":20,"c":200}
{"a":3,"b":30,"c":300}'
The command
SELECT='a,b'; echo $J | jq -c -s --arg P1 $SELECT '.[]|{a,b}'
produces
{"a":1,"b":10}
{"a":2,"b":20}
{"a":3,"b":30}
but this command produces unexpected results:
SELECT='a,b'; echo $J | jq -c -s --arg P1 $SELECT '.[]|{$P1}'
{"P1":"a,b"}
{"P1":"a,b"}
{"P1":"a,b"}
How does one get jq to treat an arg string literally?
Using tostring gives an error
SELECT='a,b'; echo $J | jq -c -s --arg P1 $SELECT '.[]|{$P1|tostring}'
jq: error: syntax error, unexpected '|', expecting '}' (Unix shell quoting
issues?) at <top-level>, line 1:
.[]|{$SELECT|tostring}
jq: 1 compile error
SELECT needs to be a variable and not hardcoded in the script.
SELECT needs to be a variable and not hardcoded in the script.
Assuming you want to avoid the risks of "code injection" and that you want the shell variable SELECT to be a simple string such as "a,b", then consider this reduce-free solution along the lines you were attempting:
J='{"a":1,"b":10,"c":100}'
SELECT='a,b'
echo "$J" |
jq -c --arg P1 "$SELECT" '
. as $in | $P1 | split(",") | map( {(.): $in[.]} ) | add'
Output:
{"a":1,"b":10}
If you really want your data to be parsed as syntax...
This is not an appropriate use case for --arg. Instead, substitute into the code:
select='a,b'; jq -c -s '.[]|{'"$select"'}' <<<"$j"
Note that this has all the usual caveats of code injection: If the input is uncontrolled, the output (or other behavior of the script, particularly if jq gains more capable I/O features in the future) should be considered likewise.
If you want to split the literal string into a list of keys...
Here, we take your select_str (of the form a,b), and generate a map: {'a': 'a', 'b': 'b'}; then, we can break each data item into entries, select only the items in the map, and there's our output.
jq --arg select_str "$select" '
($select_str
| split(",")
| reduce .[] as $item ({}; .[$item]=$item)) as $select_map
| with_entries(select($select_map[.key]))' <<<"$j"
I am trying to use jq to parse information from the TVDB api. I need to pull a couple of fields and assign the values to variables that I can continue to use in my bash script. I know I can easily assign the output to one variable through bash with variable="$(command)" but I need the output to produce multiple variables and I don't want to make to use multiple commands.
I read this documentation:
https://stedolan.github.io/jq/manual/v1.5/#Advancedfeatures
but I don't know if this relevant to what I am trying to do.
jq '.data' produces the following output:
[
{
"absoluteNumber": 51,
"airedEpisodeNumber": 6,
"airedSeason": 4,
"airedSeasonID": 680431,
"dvdEpisodeNumber": 6,
"dvdSeason": 4,
"episodeName": "We Will Rise",
"firstAired": "2017-03-15",
"id": 5939660,
"language": {
"episodeName": "en",
"overview": "en"
},
"lastUpdated": 1490769062,
"overview": "Clarke and Roan must work together in hostile territory in order to deliver an invaluable asset to Abby and her team."
}
]
I tried jq '.data | {episodeName:$name}' and jq '.data | .episodeName as $name' just to try and get one working. I don't understand the documentation or even if it's what I'm looking for. Is there a way to do what I am trying to do?
You can use separate variables with read :
read var1 var2 var3 < <(echo $(curl -s 'https://api.github.com/repos/torvalds/linux' |
jq -r '.id, .name, .full_name'))
echo "id : $var1"
echo "name : $var2"
echo "full_name : $var3"
Using array :
read -a arr < <(echo $(curl -s 'https://api.github.com/repos/torvalds/linux' |
jq -r '.id, .name, .full_name'))
echo "id : ${arr[0]}"
echo "name : ${arr[1]}"
echo "full_name : ${arr[2]}"
Also you can split jq output with some character :
IFS='|' read var1 var2 var3 var4 < <(curl '......' | jq -r '.data |
map([.absoluteNumber, .airedEpisodeNumber, .episodeName, .overview] |
join("|")) | join("\n")')
Or use an array like :
set -f; IFS='|' data=($(curl '......' | jq -r '.data |
map([.absoluteNumber, .airedEpisodeNumber, .episodeName, .overview] |
join("|")) | join("\n")')); set +f
absoluteNumber, airedEpisodeNumber, episodeName & overview are respectively ${data[0]}, ${data[1]}, ${data[2]}, ${data[3]}. set -f and set +f are used to respectively disable & enable globbing.
For the jq part, all your required fields are mapped and delimited with a '|' character with join("|")
If your are using jq < 1.5, you'll have to convert Number to String with tostring for each Number fields eg:
IFS='|' read var1 var2 var3 var4 < <(curl '......' | jq -r '.data |
map([.absoluteNumber|tostring, .airedEpisodeNumber|tostring, .episodeName, .overview] |
join("|")) | join("\n")')
jq always produces a stream of zero or more values. For example, to produce the two values corresponding to "episodeName" and "id"' you could write:
.data[] | ( .episodeName, .id )
For your purposes, it might be helpful to use the -c command-line option, to ensure each JSON output value is presented on a single line. You might also want to use the -r command-line option, which removes the outermost quotation marks from each output value that is a JSON string.
For further variations, please see the jq FAQ https://github.com/stedolan/jq/wiki/FAQ, e.g. the question:
Q: How can a stream of JSON texts produced by jq be converted into a bash array of corresponding values?
Experimental conversion of quoted OP input, (tv.dat), to a series of bash variables, (and an array). The jq code is mostly borrowed from here and there, but I don't know how to get jq to unroll an array within an array, so the sed code does that, (that's only good for one level, but so are bash arrays):
jq -r ".[] | to_entries | map(\"DAT_\(.key) \(.value|tostring)\") | .[]" tv.dat |
while read a b ; do echo "${a,,}='$b'" ; done |
sed -e '/{.*}/s/"\([^"]*\)":/[\1]=/g;y/{},/() /' -e "s/='(/=(/;s/)'$/)/"
Output:
dat_absolutenumber='51'
dat_airedepisodenumber='6'
dat_airedseason='4'
dat_airedseasonid='680431'
dat_dvdepisodenumber='6'
dat_dvdseason='4'
dat_episodename='We Will Rise'
dat_firstaired='2017-03-15'
dat_id='5939660'
dat_language=([episodeName]="en" [overview]="en")
dat_lastupdated='1490769062'
dat_overview='Clarke and Roan must work together in hostile territory in order to deliver an invaluable asset to Abby and her team.'