jq and bash: object construction with --arg is not working - bash

Given the following input:
J='{"a":1,"b":10,"c":100}
{"a":2,"b":20,"c":200}
{"a":3,"b":30,"c":300}'
The command
SELECT='a,b'; echo $J | jq -c -s --arg P1 $SELECT '.[]|{a,b}'
produces
{"a":1,"b":10}
{"a":2,"b":20}
{"a":3,"b":30}
but this command produces unexpected results:
SELECT='a,b'; echo $J | jq -c -s --arg P1 $SELECT '.[]|{$P1}'
{"P1":"a,b"}
{"P1":"a,b"}
{"P1":"a,b"}
How does one get jq to treat an arg string literally?
Using tostring gives an error
SELECT='a,b'; echo $J | jq -c -s --arg P1 $SELECT '.[]|{$P1|tostring}'
jq: error: syntax error, unexpected '|', expecting '}' (Unix shell quoting
issues?) at <top-level>, line 1:
.[]|{$SELECT|tostring}
jq: 1 compile error
SELECT needs to be a variable and not hardcoded in the script.

SELECT needs to be a variable and not hardcoded in the script.
Assuming you want to avoid the risks of "code injection" and that you want the shell variable SELECT to be a simple string such as "a,b", then consider this reduce-free solution along the lines you were attempting:
J='{"a":1,"b":10,"c":100}'
SELECT='a,b'
echo "$J" |
jq -c --arg P1 "$SELECT" '
. as $in | $P1 | split(",") | map( {(.): $in[.]} ) | add'
Output:
{"a":1,"b":10}

If you really want your data to be parsed as syntax...
This is not an appropriate use case for --arg. Instead, substitute into the code:
select='a,b'; jq -c -s '.[]|{'"$select"'}' <<<"$j"
Note that this has all the usual caveats of code injection: If the input is uncontrolled, the output (or other behavior of the script, particularly if jq gains more capable I/O features in the future) should be considered likewise.
If you want to split the literal string into a list of keys...
Here, we take your select_str (of the form a,b), and generate a map: {'a': 'a', 'b': 'b'}; then, we can break each data item into entries, select only the items in the map, and there's our output.
jq --arg select_str "$select" '
($select_str
| split(",")
| reduce .[] as $item ({}; .[$item]=$item)) as $select_map
| with_entries(select($select_map[.key]))' <<<"$j"

Related

How to store dynamic value in Variable For Bash?

I have below command:
ExpirationDate=$(date -d '+60 days' +'%Y-%m-%d')
VaultName="abc"
getapp=$(az keyvault secret list --vault-name $VaultName --query "[].{SecretName:name,ExpiryDate:attributes.expires} [?ExpiryDate<='$ExpirationDate']" | jq '.[].SecretName' | tr -d '"')
getserviceprincipal=$(az keyvault secret list --vault-name $VaultName --query "[].{Type:contentType,ExpiryDate:attributes.expires} [?ExpiryDate<='$ExpirationDate']" | jq '.[].Type' | tr -d '"')
## get length of $distro array
len=${#getapp[#]}
## Use bash for loop
for (( i=0; i-le$len-1; i++ ))
do
echo "${getapp[$i]}"
./resetpassword.sh -a ${getapp[$i]} -s ${getserviceprincipal[$i]} -y
echo "${getserviceprincipal[$i]}"
done
in this command I want store all value of vault name getapp and similarly getserviceprincipal. Example If I have more then 2 vault in getapp variable then script is not working due to $getapp is not storing variable in array.
Is anyone help me to put out this simple solutions!! Thanks In Advance..
readarray -t getapp < <( az keyvault ... | tr -d '"' ) should do the trick here.
Note that this requires newlines to be valid delimiters. If there can be newlines in your data then you'll have to pick a different delimiter with the -d delim option. If there isn't any single delimiter that works everywhere then bash may not be the best choice for this.
Since you are using jq, I think you could so something like that:
declare -a getapp=()
declare -a getserviceprincipal=()
# note: be sure to check that the resulting bash is valid!
eval(az keyvault secret list \
--vault-name $VaultName \
--query "[].{SecretName:name,ExpiryDate:attributes.expires} [?ExpiryDate<='$ExpirationDate']" \
| jq --raw-output '.[] | #sh "getapp+=( \(.SecretName) ) ; getserviceprincipal+=( \(.Type) );' ")
If all goes well, this will result in getapp and getserviceprincipal being filled as array: https://jqplay.org/s/BbHMn9i79KB
Note:
as you can see, you don't need to invoke your command (az) twice.
you can also extract the jq expression to a file using the --from-file option, which may help when reading it and handling shell quotes.

Unable to loop through the JSON internal Array having spaces in values using Bash script JQ [duplicate]

Background
I want to be able to pass a json file to WP CLI, to iteratively create posts.
So I thought I could create a JSON file:
[
{
"post_type": "post",
"post_title": "Test",
"post_content": "[leaflet-map][leaflet-marker]",
"post_status": "publish"
},
{
"post_type": "post",
"post_title": "Number 2",
"post_content": "[leaflet-map fitbounds][leaflet-circle]",
"post_status": "publish"
}
]
and iterate the array with jq:
cat posts.json | jq --raw-output .[]
I want to be able to iterate these to execute a similar function:
wp post create \
--post_type=post \
--post_title='Test Map' \
--post_content='[leaflet-map] [leaflet-marker]' \
--post_status='publish'
Is there a way I can do this with jq, or similar?
The closest I've gotten so far is this:
> for i in $(cat posts.json | jq -c .[]); do echo $i; done
But this seems to take issue with the (valid) spaces in the strings. Output:
{"post_type":"post","post_title":"Test","post_content":"[leaflet-map][leaflet-marker]","post_status":"publish"}
{"post_type":"post","post_title":"Number
2","post_content":"[leaflet-map
fitbounds][leaflet-circle]","post_status":"publish"}
Am I way off with this approach, or can it be done?
Use a while to read entire lines, rather than iterating over the words resulting from the command substitution.
while IFS= read -r obj; do
...
done < <(jq -c '.[]' posts.json)
Maybe this would work for you:
Make a bash executable, maybe call it wpfunction.sh
#!/bin/bash
wp post create \
--post_type="$1"\
--post_title="$2" \
--post_content="$3" \
--post_status="$4"
Then run jq on your posts.json and pipe it into xargs
jq -M -c '.[] | [.post_type, .post_title, .post_content, .post_status][]' \
posts.json | xargs -n4 ./wpfunction`
I am experimenting to see how this would handle post_content that contained quotes...
First generate an array of the arguments you wish to pass then convert to a shell compatible form using #sh. Then you could pass to xargs to invoke the command.
$ jq -r '.[] | ["post", "create", (to_entries[] | "--\(.key)=\(.value|tojson)")] | #sh' input.json | xargs wp

Bash loop is not working when word contains space

I am using JQ module the parse some of the data and then running the final loop over it to parse few more data.
cluster_list=`databricks --profile hq_dev clusters list --output JSON | jq 'select(.clusters != null) | .clusters[] | [.cluster_name,.autotermination_minutes,.state,.cluster_id] | #csv' | grep -v "job-"`
for cluster in ${cluster_list[#]}
do
cluster_id=`echo $cluster| cut -d "," -f 4 | sed 's/\"//g' | sed 's/\\\//g'`
cluster_name=`echo "${cluster}"| cut -d "," -f 1| sed 's/\"//g' | sed 's/\\\//g'`
echo $cluster_name
done
cluster_list contains following value.
"\"Test Space Cluster\",15,\"TERMINATED\",\"ddd-dese23-can858\""
"\"GatewayCluster\",15,\"TERMINATED\",\"ddd-ddsd-ddsds\""
"\"delete_later\",15,\"TERMINATED\",\"1120-195800-93839\""
"\"GatewayCluster_old\",15,\"TERMINATED\",\"0108-2y7272-393893\""
it prints following.
Test
Space
Cluster
GatewayCluster
delete_later
GatewayCluster_old
Desired output
it shouldn't break to newline if there is a space, I am doing few more action by the name I am getting here.
Test Space Cluster
GatewayCluster
delete_later
GatewayCluster_old
Your script seems a bit overly complex to achieve your goal. Better use read to store each value in a separate variable, and set a comma for the input field separator IFS:
databricks --profile hq_dev clusters list --output JSON |
jq 'select(.clusters != null) | .clusters[] |
[.cluster_name,.autotermination_minutes,.state,.cluster_id] | #csv' |
grep -v "job-" |
sed 's/\\\?"//g' |
while IFS=, read name autotermination_minutes state id ; do
echo $name
done
Note: I didn't touch your jq command. The sed line I put aims to remove quotes, protected or not. You can tune jq to remove these quotes with -r, as said in the man page:
INVOKING JQ
[...]
--raw-output / -r::
With this option, if the filterĀ“s result is a string then it will be written directly to standard output rather than being formatted as a JSON string with quotes. This can be useful for making jq filters talk to non-JSON-based systems.

jq with multiple inputs from different sources

How can we mix different input sources when using jq ?
For a specific usecase, I'd like to add some data from a file into a feed that was pipe in stdout.
$ echo '[{"a": 1}]' > /tmp/a1
$ echo '[{"a": 2}]' > /tmp/a2
$ jq --slurp '.[0] + .[1]' /tmp/a1 /tmp/a2
[
{
"a": 1
},
{
"a": 2
}
]
$ cat /tmp/a1 | jq --slurp '.[0] + .[1]' /tmp/a2 # Expecting the same result
[
{
"a": 2
}
]
As you can see, the last command didn't interpret the piped data.
Right now, I'm forced to save the output from the first operation into a temporary file, so that I can do the jq merging operation, before sending it back to the network. Having a single stream would be much more efficient
I'd like to add some data from a file into a feed that was pipe in stdout.
There are various ways to do this, depending on the shell and also the version of jq you are using.
Assuming your jq supports the --argfile option, you might find that quite congenial:
cat /tmp/a1 | jq --argfile a2 /tmp/a2 '. + $a2'
Here is another variation that suggests some of the other possibilities:
jq -n --argfile a1 <(cat /tmp/a1) --argfile a2 <(cat /tmp/a2) '$a1 + $a2'
More interestingly:
(cat /tmp/a1 ; cat /tmp/a2) | jq '. + input'
You might also wish to consider using the --slurpfile option instead of --argfile, but note that --slurpfile always "slurps" the file.
And finally an approach that should work for every version of jq:
jq -s '.[0] + .[1]' <(cat /tmp/a1) /tmp/a2
In general, though, it's best to avoid the -s option.
A note on slurping
If you compare the outputs produced by:
echo '1 2' |
jq -s --debug-dump-disasm --debug-trace '.[0], .[1]'
and
echo '1 2' |
jq --debug-dump-disasm --debug-trace '., input'
you'll notice the former has to PUSHK_UNDER to store the entire array [1,2],
whereas the latter program just reads the two inputs separately.
In the first program, the memory for the array cannot be freed until after
all the pointers into it have been processed, whereas in the second program,
the memory for . can be freed after the first RET.
You could do this, where cat forwards its stdin followed by a2:
<GENERATE a1> | cat - /tmp/a2 | jq --slurp '.[0] + .[1]'
Or this, which is a compound statement passing the results of two separate commands into a pipe:
{ <GENERATE a1> ; cat /tmp/a2; } | jq --slurp '.[0] + .[1]'
Take care to have spaces beside the curly braces and to have a semi-colon before the final one.
Completing peak's answer, you actually don't need redirections:
jq -n --argfile a1 /tmp/a1 --argfile a2 /tmp/a2 '$a1 + $a2'

using jq to assign multiple output variables

I am trying to use jq to parse information from the TVDB api. I need to pull a couple of fields and assign the values to variables that I can continue to use in my bash script. I know I can easily assign the output to one variable through bash with variable="$(command)" but I need the output to produce multiple variables and I don't want to make to use multiple commands.
I read this documentation:
https://stedolan.github.io/jq/manual/v1.5/#Advancedfeatures
but I don't know if this relevant to what I am trying to do.
jq '.data' produces the following output:
[
{
"absoluteNumber": 51,
"airedEpisodeNumber": 6,
"airedSeason": 4,
"airedSeasonID": 680431,
"dvdEpisodeNumber": 6,
"dvdSeason": 4,
"episodeName": "We Will Rise",
"firstAired": "2017-03-15",
"id": 5939660,
"language": {
"episodeName": "en",
"overview": "en"
},
"lastUpdated": 1490769062,
"overview": "Clarke and Roan must work together in hostile territory in order to deliver an invaluable asset to Abby and her team."
}
]
I tried jq '.data | {episodeName:$name}' and jq '.data | .episodeName as $name' just to try and get one working. I don't understand the documentation or even if it's what I'm looking for. Is there a way to do what I am trying to do?
You can use separate variables with read :
read var1 var2 var3 < <(echo $(curl -s 'https://api.github.com/repos/torvalds/linux' |
jq -r '.id, .name, .full_name'))
echo "id : $var1"
echo "name : $var2"
echo "full_name : $var3"
Using array :
read -a arr < <(echo $(curl -s 'https://api.github.com/repos/torvalds/linux' |
jq -r '.id, .name, .full_name'))
echo "id : ${arr[0]}"
echo "name : ${arr[1]}"
echo "full_name : ${arr[2]}"
Also you can split jq output with some character :
IFS='|' read var1 var2 var3 var4 < <(curl '......' | jq -r '.data |
map([.absoluteNumber, .airedEpisodeNumber, .episodeName, .overview] |
join("|")) | join("\n")')
Or use an array like :
set -f; IFS='|' data=($(curl '......' | jq -r '.data |
map([.absoluteNumber, .airedEpisodeNumber, .episodeName, .overview] |
join("|")) | join("\n")')); set +f
absoluteNumber, airedEpisodeNumber, episodeName & overview are respectively ${data[0]}, ${data[1]}, ${data[2]}, ${data[3]}. set -f and set +f are used to respectively disable & enable globbing.
For the jq part, all your required fields are mapped and delimited with a '|' character with join("|")
If your are using jq < 1.5, you'll have to convert Number to String with tostring for each Number fields eg:
IFS='|' read var1 var2 var3 var4 < <(curl '......' | jq -r '.data |
map([.absoluteNumber|tostring, .airedEpisodeNumber|tostring, .episodeName, .overview] |
join("|")) | join("\n")')
jq always produces a stream of zero or more values. For example, to produce the two values corresponding to "episodeName" and "id"' you could write:
.data[] | ( .episodeName, .id )
For your purposes, it might be helpful to use the -c command-line option, to ensure each JSON output value is presented on a single line. You might also want to use the -r command-line option, which removes the outermost quotation marks from each output value that is a JSON string.
For further variations, please see the jq FAQ https://github.com/stedolan/jq/wiki/FAQ, e.g. the question:
Q: How can a stream of JSON texts produced by jq be converted into a bash array of corresponding values?
Experimental conversion of quoted OP input, (tv.dat), to a series of bash variables, (and an array). The jq code is mostly borrowed from here and there, but I don't know how to get jq to unroll an array within an array, so the sed code does that, (that's only good for one level, but so are bash arrays):
jq -r ".[] | to_entries | map(\"DAT_\(.key) \(.value|tostring)\") | .[]" tv.dat |
while read a b ; do echo "${a,,}='$b'" ; done |
sed -e '/{.*}/s/"\([^"]*\)":/[\1]=/g;y/{},/() /' -e "s/='(/=(/;s/)'$/)/"
Output:
dat_absolutenumber='51'
dat_airedepisodenumber='6'
dat_airedseason='4'
dat_airedseasonid='680431'
dat_dvdepisodenumber='6'
dat_dvdseason='4'
dat_episodename='We Will Rise'
dat_firstaired='2017-03-15'
dat_id='5939660'
dat_language=([episodeName]="en" [overview]="en")
dat_lastupdated='1490769062'
dat_overview='Clarke and Roan must work together in hostile territory in order to deliver an invaluable asset to Abby and her team.'

Resources