I have a source file that looks like
{
"admin-user" : {
"index.md": "Index",
"user_profil.md": "User Profil"
}
}
By help of bash and jq I like to concat a string of the second level keys index.md and user_profile.md. Further I like replace .md with .html (key admin-user is unknown an can change)
This is where I hang:
KEYS=$(cat test.json | jq -r '.[]|keys')
concat_string=""
for i in $KEYS
do
md_name=${i/.md/.html}
concat_string="$concat_string$md_name"
done
echo $concat_string
Result:
["index.html","user_profil.html"]
So the result is an array. How can I concat a string with blanks between strings?
All of it can be done from within jq:
map(keys_unsorted[] | sub("\\.md$"; ".html"))
Demo
Alternatively, you can use to_entries to access each .key:
map(to_entries[].key | sub("\\.md$"; ".html"))
Demo
Both will give you an array of strings
["index.html","user_profil.html"]
which you can then -still in jq- concatenate using join and a glue character:
jq -r 'map(keys_unsorted[] | sub("\\.md$"; ".html")) | join(" ")' test.json
Demo
or using the alternative approach:
jq -r 'map(to_entries[].key | sub("\\.md$"; ".html")) | join(" ")' test.json
Demo
Note: Using keys instead of keys_unsorted (as you did in your attempt) will additionally sort the keys.
Related
Background
I want to be able to pass a json file to WP CLI, to iteratively create posts.
So I thought I could create a JSON file:
[
{
"post_type": "post",
"post_title": "Test",
"post_content": "[leaflet-map][leaflet-marker]",
"post_status": "publish"
},
{
"post_type": "post",
"post_title": "Number 2",
"post_content": "[leaflet-map fitbounds][leaflet-circle]",
"post_status": "publish"
}
]
and iterate the array with jq:
cat posts.json | jq --raw-output .[]
I want to be able to iterate these to execute a similar function:
wp post create \
--post_type=post \
--post_title='Test Map' \
--post_content='[leaflet-map] [leaflet-marker]' \
--post_status='publish'
Is there a way I can do this with jq, or similar?
The closest I've gotten so far is this:
> for i in $(cat posts.json | jq -c .[]); do echo $i; done
But this seems to take issue with the (valid) spaces in the strings. Output:
{"post_type":"post","post_title":"Test","post_content":"[leaflet-map][leaflet-marker]","post_status":"publish"}
{"post_type":"post","post_title":"Number
2","post_content":"[leaflet-map
fitbounds][leaflet-circle]","post_status":"publish"}
Am I way off with this approach, or can it be done?
Use a while to read entire lines, rather than iterating over the words resulting from the command substitution.
while IFS= read -r obj; do
...
done < <(jq -c '.[]' posts.json)
Maybe this would work for you:
Make a bash executable, maybe call it wpfunction.sh
#!/bin/bash
wp post create \
--post_type="$1"\
--post_title="$2" \
--post_content="$3" \
--post_status="$4"
Then run jq on your posts.json and pipe it into xargs
jq -M -c '.[] | [.post_type, .post_title, .post_content, .post_status][]' \
posts.json | xargs -n4 ./wpfunction`
I am experimenting to see how this would handle post_content that contained quotes...
First generate an array of the arguments you wish to pass then convert to a shell compatible form using #sh. Then you could pass to xargs to invoke the command.
$ jq -r '.[] | ["post", "create", (to_entries[] | "--\(.key)=\(.value|tojson)")] | #sh' input.json | xargs wp
I'm trying to create 2 variables via bash $lat, $long base on the result of my curl response.
curl ipinfo.io/33.62.137.111 | grep "loc" | awk '{print $2}'
I got.
"42.6334,-71.3162",
I'm trying to get
$lat=42.6334
$long=-71.3162
Can someone give me a little push ?
IFS=, read -r lat long < <(
curl -s ipinfo.io/33.62.137.111 |
jq -r '.loc'
)
printf 'Latitude is: %s\nLongitude is: %s\n' "$lat" "$long"
The ipinfo.io API is returning JSON data, so let parse it with jq:
Here is the JSON as returned by the query from your sample:
{
"ip": "33.62.137.111",
"city": "Columbus",
"region": "Ohio",
"country": "US",
"loc": "39.9690,-83.0114",
"postal": "43218",
"timezone": "America/New_York",
"readme": "https://ipinfo.io/missingauth"
}
We are going to JSON query the loc entry from the main root object ..
curl -s ipinfo.io/33.62.137.111: download the JSON data -s silently without progress.
jq -r '.loc': Process JSON data, query the loc entry of the main object and -r output raw string.
IFS=, read -r lat long < <(: Sets the Internal Field Separator to , and read both lat and long variables from the following command group output stream.
Although the answer from #LeaGris is quite interesting, if you don't want to use an external library or something, you can try this:
Playground: https://repl.it/repls/ThoughtfulImpressiveComputer
coordinates=($(curl ipinfo.io/33.62.137.111 | sed 's/ //g' | grep -P '(?<=\"loc\":").*?(?=\")' -o | tr ',' ' '))
echo "${coordinates[#]}"
echo ${coordinates[0]}
echo ${coordinates[1]}
Example output:
39.9690 -83.0114 # echo "${coordinates[#]}"
39.9690 # ${coordinates[0]}
-83.0114 # ${coordinates[1]}
Explanation:
curl ... get the JSON data
sed 's/ //g' remove all spaces
grep -P ... -o
-P interpret the given pattern as a perl regexp
(?<=\"loc\":").*?(?=\")
(?<=\"loc\":") regex lookbehind
.*? capture the longitude and latitude part with non-greedy search
(?=\") regex lookahead
-o get only the matching part which'ld be e.g. 39.9690,-83.0114
tr ',' ' ' replace , with space
Finally we got something like this: 39.9690 -83.0114
Putting it in parentheses lets us create an array with two values in it (cf. ${coordinates[...]}).
I am using JQ module the parse some of the data and then running the final loop over it to parse few more data.
cluster_list=`databricks --profile hq_dev clusters list --output JSON | jq 'select(.clusters != null) | .clusters[] | [.cluster_name,.autotermination_minutes,.state,.cluster_id] | #csv' | grep -v "job-"`
for cluster in ${cluster_list[#]}
do
cluster_id=`echo $cluster| cut -d "," -f 4 | sed 's/\"//g' | sed 's/\\\//g'`
cluster_name=`echo "${cluster}"| cut -d "," -f 1| sed 's/\"//g' | sed 's/\\\//g'`
echo $cluster_name
done
cluster_list contains following value.
"\"Test Space Cluster\",15,\"TERMINATED\",\"ddd-dese23-can858\""
"\"GatewayCluster\",15,\"TERMINATED\",\"ddd-ddsd-ddsds\""
"\"delete_later\",15,\"TERMINATED\",\"1120-195800-93839\""
"\"GatewayCluster_old\",15,\"TERMINATED\",\"0108-2y7272-393893\""
it prints following.
Test
Space
Cluster
GatewayCluster
delete_later
GatewayCluster_old
Desired output
it shouldn't break to newline if there is a space, I am doing few more action by the name I am getting here.
Test Space Cluster
GatewayCluster
delete_later
GatewayCluster_old
Your script seems a bit overly complex to achieve your goal. Better use read to store each value in a separate variable, and set a comma for the input field separator IFS:
databricks --profile hq_dev clusters list --output JSON |
jq 'select(.clusters != null) | .clusters[] |
[.cluster_name,.autotermination_minutes,.state,.cluster_id] | #csv' |
grep -v "job-" |
sed 's/\\\?"//g' |
while IFS=, read name autotermination_minutes state id ; do
echo $name
done
Note: I didn't touch your jq command. The sed line I put aims to remove quotes, protected or not. You can tune jq to remove these quotes with -r, as said in the man page:
INVOKING JQ
[...]
--raw-output / -r::
With this option, if the filter´s result is a string then it will be written directly to standard output rather than being formatted as a JSON string with quotes. This can be useful for making jq filters talk to non-JSON-based systems.
I am trying to use jq to parse information from the TVDB api. I need to pull a couple of fields and assign the values to variables that I can continue to use in my bash script. I know I can easily assign the output to one variable through bash with variable="$(command)" but I need the output to produce multiple variables and I don't want to make to use multiple commands.
I read this documentation:
https://stedolan.github.io/jq/manual/v1.5/#Advancedfeatures
but I don't know if this relevant to what I am trying to do.
jq '.data' produces the following output:
[
{
"absoluteNumber": 51,
"airedEpisodeNumber": 6,
"airedSeason": 4,
"airedSeasonID": 680431,
"dvdEpisodeNumber": 6,
"dvdSeason": 4,
"episodeName": "We Will Rise",
"firstAired": "2017-03-15",
"id": 5939660,
"language": {
"episodeName": "en",
"overview": "en"
},
"lastUpdated": 1490769062,
"overview": "Clarke and Roan must work together in hostile territory in order to deliver an invaluable asset to Abby and her team."
}
]
I tried jq '.data | {episodeName:$name}' and jq '.data | .episodeName as $name' just to try and get one working. I don't understand the documentation or even if it's what I'm looking for. Is there a way to do what I am trying to do?
You can use separate variables with read :
read var1 var2 var3 < <(echo $(curl -s 'https://api.github.com/repos/torvalds/linux' |
jq -r '.id, .name, .full_name'))
echo "id : $var1"
echo "name : $var2"
echo "full_name : $var3"
Using array :
read -a arr < <(echo $(curl -s 'https://api.github.com/repos/torvalds/linux' |
jq -r '.id, .name, .full_name'))
echo "id : ${arr[0]}"
echo "name : ${arr[1]}"
echo "full_name : ${arr[2]}"
Also you can split jq output with some character :
IFS='|' read var1 var2 var3 var4 < <(curl '......' | jq -r '.data |
map([.absoluteNumber, .airedEpisodeNumber, .episodeName, .overview] |
join("|")) | join("\n")')
Or use an array like :
set -f; IFS='|' data=($(curl '......' | jq -r '.data |
map([.absoluteNumber, .airedEpisodeNumber, .episodeName, .overview] |
join("|")) | join("\n")')); set +f
absoluteNumber, airedEpisodeNumber, episodeName & overview are respectively ${data[0]}, ${data[1]}, ${data[2]}, ${data[3]}. set -f and set +f are used to respectively disable & enable globbing.
For the jq part, all your required fields are mapped and delimited with a '|' character with join("|")
If your are using jq < 1.5, you'll have to convert Number to String with tostring for each Number fields eg:
IFS='|' read var1 var2 var3 var4 < <(curl '......' | jq -r '.data |
map([.absoluteNumber|tostring, .airedEpisodeNumber|tostring, .episodeName, .overview] |
join("|")) | join("\n")')
jq always produces a stream of zero or more values. For example, to produce the two values corresponding to "episodeName" and "id"' you could write:
.data[] | ( .episodeName, .id )
For your purposes, it might be helpful to use the -c command-line option, to ensure each JSON output value is presented on a single line. You might also want to use the -r command-line option, which removes the outermost quotation marks from each output value that is a JSON string.
For further variations, please see the jq FAQ https://github.com/stedolan/jq/wiki/FAQ, e.g. the question:
Q: How can a stream of JSON texts produced by jq be converted into a bash array of corresponding values?
Experimental conversion of quoted OP input, (tv.dat), to a series of bash variables, (and an array). The jq code is mostly borrowed from here and there, but I don't know how to get jq to unroll an array within an array, so the sed code does that, (that's only good for one level, but so are bash arrays):
jq -r ".[] | to_entries | map(\"DAT_\(.key) \(.value|tostring)\") | .[]" tv.dat |
while read a b ; do echo "${a,,}='$b'" ; done |
sed -e '/{.*}/s/"\([^"]*\)":/[\1]=/g;y/{},/() /' -e "s/='(/=(/;s/)'$/)/"
Output:
dat_absolutenumber='51'
dat_airedepisodenumber='6'
dat_airedseason='4'
dat_airedseasonid='680431'
dat_dvdepisodenumber='6'
dat_dvdseason='4'
dat_episodename='We Will Rise'
dat_firstaired='2017-03-15'
dat_id='5939660'
dat_language=([episodeName]="en" [overview]="en")
dat_lastupdated='1490769062'
dat_overview='Clarke and Roan must work together in hostile territory in order to deliver an invaluable asset to Abby and her team.'
I am using the following command to list all the images within my private registry:
curl -s http://internal.private.registry.com/v2/_catalog | jq -r '.repositories[0:2] | to_entries | map( .value )[]'
Output:
centos
containersol/consul-server
I am using the following command to list all the tags of a particular image within my private registry:
curl -s GET http://internal.private.registry.com/v2/centos/tags/list | jq -r '.tags | to_entries | map( .value )[]'
Output:
6.6
6
7.1.1503
Now, I am trying to list all tags for every image present in catalog using the following bash script:
#!/bin/sh
image_name=$(curl -s http://internal.private.registry.com/v2/_catalog | jq -r '.repositories[0:2] | to_entries | map( .value )[]')
while read -r line; do
${line}$_image_taglist=$(curl -s GET http://internal.private.registry.com/v2/cybs/${line}/tags/list | jq -r '.tags | to_entries | map( .value )[]')
while read -r tag; do
echo "$tag"
done <<< "${line}$_image_taglist"
done <<< "$image_name"
However, I am getting the following error while executing ${line}$_image_taglist=$(curl -s GET http://internal.private.registry.com/v2/cybs/${line}/tags/list | jq -r '.tags | to_entries | map( .value )[]') :
jq: error (at <stdin>:1): null (null) has no keys
Also, please note that I am expecting that dynamic variable ${line}$_image_taglist should resolve to centos_image_taglist and containersol/consul-server_image_taglist
jq: error (at :1): null (null) has no keys
Is caused by . being null. In this case . refers to stdin which means that you get no response from http://internal.private.registry.com/v2/cybs/${line}/tags/list.
You cannot define dynamic named variables like this: ${line}$_image_taglist. You should use an associative array if your bash version supports it:
declare -A arr
arr["$line"]=$(curl ... | jq ...)
But in your case I can't really see why you need it, since you are just iteration over it. Why not use a pipe or process substitution?
while read -r tag; do
echo "$tag"
done < <(curl ... | jq ...)