how to get multiple values from a JSON array/object using jq in a single line?
here curl request and its response
curl -s -k --location --request GET "https://${HOSTNAME}/api/v1/projects/myProjects?page=1&pageSize=2" --header "Authorization: Bearer "$token"" | jq -r '.["data"]'
# Response
[
{
"id": "8a70803f8045722601804f62d54c5d9d",
"createdBy": "8a70802980325cdc0180326b5fe60006",
"createdDate": "2022-04-22T03:48:38.860+0000",
"modifiedBy": "8a70802980325cdc0180326b5fe60006",
"modifiedDate": "2022-04-22T03:48:38.860+0000",
"version": null,
"inactive": false,
"name": "Netbank734113",
},
{
"id": "8a70801c804568ae01804f625a923f8d",
"createdBy": "8a70802980325cdc0180326b5fe60006",
"createdDate": "2022-04-22T03:48:07.442+0000",
"modifiedBy": "8a70802980325cdc0180326b5fe60006",
"modifiedDate": "2022-04-22T03:48:07.442+0000",
"version": null,
"inactive": false,
"name": "Netbank734112",
}
]
Now try the below command to get the id and name in one line but duplicate results are coming.
result=$(curl -s -k --location --request GET "https://${HOSTNAME}/api/v1/projects/myProjects?page=1&pageSize=2" --header "Authorization: Bearer "$token"" | jq -r '.["data"]|.[].name +" "+ .[].id')
echo "$result"
# response
Netbank734113 8a70803f8045722601804f62d54c5d9d
Netbank734112 8a70803f8045722601804f62d54c5d9d
Netbank734113 8a70801c804568ae01804f625a923f8d
Netbank734112 8a70801c804568ae01804f625a923f8d
how do I get rid of those duplicated values and basically get the below response?
Netbank734113 8a70803f8045722601804f62d54c5d9d
Netbank734112 8a70801c804568ae01804f625a923f8d
Tried this command with your inputs getting the below errors, how do I modify this curl command to get desired results.
result=$(curl -s -k --location --request GET "https://${HOSTNAME}/api/v1/projects/myProjects?page=1&pageSize=2" --header "Authorization: Bearer "$token"" | jq -r '["data"]|.[].unique_by(.name, .id) | map([.name, .id])[] | #tsv')
# error response
jq: error: syntax error, unexpected '(', expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
["data"]|.[].unique_by(.name, .id) | map([.name, .id])[] | #tsv
jq: 1 compile error
# This one worked
result=$(curl -s -k --location --request GET "https://${HOSTNAME}/api/v1/projects/myProjects?page=1&pageSize=2" --header "Authorization: Bearer "$token"" | jq -r '.["data"]|.[]|.| .name + " " + .id')
With .[].name + " " + .[].id' you iterate twice over the array. Iterate once and extract your data in one go:
curl … | jq -r '.data[] | .name + " " + .id'
Netbank734113 8a70803f8045722601804f62d54c5d9d
Netbank734112 8a70801c804568ae01804f625a923f8d
Demo
You might also be interested in using string interpolation:
curl … | jq -r '.data[] | "\(.name) \(.id)"'
Demo
Related
Trying to solve "argument list too long"
I have been searching for a solution and found the closest one to my issue
curl: argument list too long
however the response is not clear as I am still having the issue "argument list too long"
curl -X POST -d #data.txt \
https://Path/to/attachments \
-H 'content-type: application/vnd.api+json' \
-H 'x-api-key: KEY' \
-d '{
"data": {
"type": "attachments",
"attributes": {
"attachment": {
"content": "'$(cat data.txt | base64 --wrap=0)'",
"file_name": "'"$FileName"'"
}
}
}
}'
thank you
Use jq to format your base64 encoded data string into a proper JSON string, and then pass the JSON data as standard input to the curl command.
#!/usr/bin/env sh
attached_file='img.png'
# Pipe the base64 encoded content of attached_file
base64 --wrap=0 "$attached_file" |
# into jq to make it a proper JSON string within the
# JSON data structure
jq --slurp --raw-input --arg FileName "$attached_file" \
'{
"type": "attachments",
"attributes": {
"attachment": {
"content": .,
"file_name": $FileName
}
}
}
' |
# Get the resultant JSON piped into curl
# that will read the data from the standard input
# using -d #-
curl -X POST -d #- \
'https://Path/to/attachments' \
-H 'content-type: application/vnd.api+json' \
-H 'x-api-key: KEY'
Per the linked answer
you are trying to pass the entirety of the base64'd content on the command line
This is a limitation of the shell, not curl. That is, the shell is responding with error argument list too long. The program curl is never even started.
The recommendation is
curl has the ability to load in data to POST from a file
Write the json data to some file /tmp/data.json using piping.(the commands will use piping | and file redirection > >> which can handle arbitrarily large amounts of data. Whereas, you cannot place arbitrarily large amounts of data into a single command, there is a limit).
echo -n '
{
"data": {
"type": "attachments",
"attributes": {
"attachment": {
"content": "' > /tmp/data.json
cat data.txt | base64 --wrap=0 >> /tmp/data.json
echo -n '",
"file_name": "'"$FileName"'"
}
}
}
}' >> /tmp/data.json
Pass that file path /tmp/data.json to the curl command using # so curl knows it's a file path.
curl -X POST -d #/tmp/data.json \
"https://Path/to/attachments" \
-H 'content-type: application/vnd.api+json' \
-H 'x-api-key: KEY'
Consider this simple bash script : (I am writing myurl as a placeholder for the correct URL; and myhost as the placeholder for the correct hostname for privacy here.)
for a in `seq 400 450`; do
curl --request POST \
--url myurl \
--header 'Content-Type: application/json' \
--header 'Host: myhost' \
--data '{
"devID": "21010010",
"count": 0,
"fileEnd": 1,
"line": "",
"file": "$a"
}'
done
This is rejected as bad request. I want, the output of seq, (e.g. the number 410) to be a quote delimated string. Notice the line "file": "$a". If I write "file": "410", this works perfectly, and also for any other file between 400 and 450 - as long as it is surrounded by "" quotes.
Question :
I want, that in each iteration of the loop, the curl request should include the number coming from the seq command surrounded by "". So the first call should be :
curl --request POST \
--url myurl \
--header 'Content-Type: application/json' \
--header 'Host: myhost' \
--data '{
"devID": "21010010",
"count": 0,
"fileEnd": 1,
"line": "",
"file": "400"
}'
The next one to be :
curl --request POST \
--url myurl \
--header 'Content-Type: application/json' \
--header 'Host: myhost' \
--data '{
"devID": "21010010",
"count": 0,
"fileEnd": 1,
"line": "",
"file": "401"
}'
and so on.
How can I do this? Thank you.
The --data portion is wrapped in single quotes ...
'{"devID": ... "file": "$a"}'
... which disables the expansion of variable references inside said single quotes.
Consider:
$ a=410
$ echo '"devID": "21010010", "file": "${a}"'
"devID": "21010010", "file": "${a}" # ${a} not expanded
You could wrap the --data section in double quotes, but then you need to also escape all of the nested double quotes.
Another idea is to break the --data section into 2x sections wrapped in single quotes and place the ${a} reference between the 2x sections, eg:
$ echo '"devID": "21010010", "file": "'${a}'"' # change "${a}" to "'${a}'"
"devID": "21010010", "file": "410" # ${a} successfully expanded
NOTE: There are several posts regarding the difference between single and double quotes, eg, this link
I use curl to test an user account creation API as follows:
curl -s -X POST "https://$APISERVER/users" \
-H 'Content-Type: application/json' \
-d '{ \
"username": "'$NEWUSERNAME'", \
"firstName": "'$NEWUSERFIRSTNAME'", \
"lastName": "'$NEWUSERLASTNAME'", \
"displayName": "'$NEWUSERDISPLAYNAME'", \
"password": "'$NEWUSERPASSWORD'" \
}'
and the variables are supplied via command line arguments:
APISERVER=http://localhost:8080
NEWUSERNAME=$1
NEWUSERPASSWORD=$2
NEWUSERFIRSTNAME=$3
NEWUSERLASTNAME=$4
# Calculated variable
NEWUSERDISPLAYNAME="${NEWUSERFIRSTNAME} ${NEWUSERLASTNAME}"
An example invocation of the script is as follows: ./test-new-user.sh jdoe Hello123 John Doe, resulting in the following variable values:
NEWUSERNAME=jdoe
NEWUSERPASSWORD=Hello123
NEWUSERFIRSTNAME=John
NEWUSERLASTNAME=Doe
(I intended NEWUSERDISPLAYNAME to be set to "John Doe")
But I get back an exception from the server, because the payload in the curl command appears to be cut-off, incomplete or malformed.
JSON parse error: Unexpected end-of-input in VALUE_STRING\n at [Source:
java.io.PushbackInputStream#2eda6052; line: 1, column: 293]; nested
exception is com.fasterxml.jackson.databind.JsonMappingException:
Unexpected end-of-input in VALUE_STRING\n at [Source:
java.io.PushbackInputStream#2eda6052; line: 1, column: 293]\n at
[Source: java.io.PushbackInputStream#2eda6052; line: 1, column: 142]
(through reference chain:
com.mycompany.api.pojos.NewUser[\"displayName\"])"
If I hard code value for displayName in the above curl command (as below), the user creation request goes through and works perfectly.
"displayName": "John Doe", \
I suspect it has to do with the space in displayName and how I insert the value for displayName using "'$NEWUSERDISPLAYNAME'". Is there a safe way to perform variable substitution in the curl command's POST request payload?
You need to quote shell variables:
curl -s -X POST "https://$APISERVER/users" \
-H 'Content-Type: application/json' \
-d '{ \
"username": "'"$NEWUSERNAME"'", \
"firstName": "'"$NEWUSERFIRSTNAME"'", \
"lastName": "'"$NEWUSERLASTNAME"'", \
"displayName": "'"$NEWUSERDISPLAYNAME"'", \
"password": "'"$NEWUSERPASSWORD"'" \
}'
In order to avoid excessive quoting, try this printf:
printf -v json -- '{ "username": "%s", "firstName": "%s", "lastName": "%s", "displayName": "%s", "password": "%s" }' \
"$NEWUSERNAME" "$NEWUSERFIRSTNAME" "$NEWUSERLASTNAME" "$NEWUSERDISPLAYNAME" "$NEWUSERPASSWORD"
curl -s -X POST "https://$APISERVER/users" \
-H 'Content-Type: application/json' \
-d "$json"
just use
$(echo $varname)
works for me
I'm using curl with IBM Watson to produce a transcript but I can't seem to get an output where just the transcript is shown as shown below
Another method might be just to grep for the text in transcript":""
curl
curl -u user:password -X POST --header "Content-Type: audio/wav" --header "Transfer-Encoding: chunked" --data-binary #test.wav "https://stream.watsonplatform.net/speech-to-text/api/v1/recognize?continuous=true" > demo.txt
{
"results": [
{
"alternatives": [
{
"confidence": 0.302,
"transcript": "when to stop announced "
}
],
"final": true
},
{
"alternatives": [
{
"confidence": 0.724,
"transcript": "Russia is destroying western cheese and considering a ban Weston condoms and infection internet is reacting "
}
],
"final": true
To store the output into a file you can use the option -o
curl -u user:password -X POST --data-binary #test.wav
-o transcript.txt
--header "Content-Type: audio/wav" --header "Transfer-Encoding: chunked"
"https://stream.watsonplatform.net/speech-to-text/api/v1/recognize?continuous=true"
-o, --output <file>
Write output to <file> instead of stdout. Like in:
curl http://ibm.com.com -o "html_output.txt"
More info.
grep alone can't do quite what you're asking for because it doesn't get more granular than a single line. However, grep + sed can - sed can be used to perform regex replacement on the lines that grep spits out.
First pipe to grep: grep transcript, then pipe to sed: sed 's/.*"transcript": "\(.*\)".*/\1/g'
Here's the complete command:
curl -u user:password -X POST --header "Content-Type: audio/wav" --header "Transfer-Encoding: chunked" --data-binary #test.wav "https://stream.watsonplatform.net/speech-to-text/api/v1/recognize?continuous=true" | grep transcript | sed 's/.*"transcript": "\(.*\)".*/\1/g'
Note that by default, curl displays several lines of status when piping it's output. You can disable those with -s:
curl -s -u user:password -X POST --header "Content-Type: audio/wav" --header "Transfer-Encoding: chunked" --data-binary #test.wav "https://stream.watsonplatform.net/speech-to-text/api/v1/recognize?continuous=true" | grep transcript | sed 's/.*"transcript": "\(.*\)".*/\1/g'
Here's more info on sed if you're interested: http://www.grymoire.com/Unix/Sed.html
Update: I should also mention that there are SDKs available for a number of languages if you're interested in doing something more complex - https://github.com/watson-developer-cloud
I want to upload a file automatically to rackspace files which requires an auth-token that is updated daily, so I want to create a script which gets the auth token and then uses that in the script to upload the file.
This is the command to get the auth token which outputs the key perfectly:
curl -s -X POST https://auth.api.rackspacecloud.com/v2.0/tokens\
-d '{ "auth":{ "RAX-KSKEY:apiKeyCredentials":{ "username":"USER", "apiKey":"KEY" } } }'\
-H "Content-type: application/json" | python -mjson.tool |\
python -c 'import sys, json;\
print json.load(sys.stdin)[sys.argv[1]][sys.argv[2]][sys.argv[3]]'\
access token id
This is the command to upload the file:
curl -X PUT -T file.xml -D - \
-H "Content-Type: text/xml" \
-H "X-Auth-Token: TOKENGOESHERE" \
URL
I need to get the token from the first command into the TOKENGOESHERE place in the second command.
What I have tried so far is:
token = curl -s -X POST https://auth.api.rackspacecloud.com/v2.0/tokens -d '{ "auth":{ "RAX-KSKEY:apiKeyCredentials":{ "username":"USER", "apiKey":"KEY" } } }' -H "Content-type: application/json" | python -mjson.tool | python -c 'import sys, json; print json.load(sys.stdin)[sys.argv[1]][sys.argv[2]][sys.argv[3]]' access token id
curl -X PUT -T file.xml -D - \
-H "Content-Type: text/xml" \
-H "X-Auth-Token: $token" \
URL
but it didn't work and I am guessing it has something to do with the quotes but I don't know enough about bash to know what the problem is.
Thanks!
This should work:
token=$(curl -s -X POST https://auth.api.rackspacecloud.com/v2.0/tokens \
-d '{ "auth":{ "RAX-KSKEY:apiKeyCredentials":{ "username":"USER", "apiKey":"KEY" } } }' \
-H "Content-type: application/json" \
| python -mjson.tool \
| python -c 'import sys, json; print json.load(sys.stdin)["access"]["token"]["id"]')
curl -X PUT -T file.xml -D - \
-H "Content-Type: text/xml" \
-H "X-Auth-Token: $token" \
URL
I know it's a bit off topic, but I wanted to share my 'workflow' which may help a lot of people.
If you download these two cool toys (replacement for curl and python's json):
https://github.com/jkbr/httpie
http://stedolan.github.io/jq/
Then you can do all these fun things:
(Just replace USER and KEY with your real user and key in the 1st line, and all the others are copy and paste-able.
Get the json:
json=$(echo '{ "auth":{ "RAX-KSKEY:apiKeyCredentials":{ "username":"USER", "apiKey":"KEY" } } }' | http POST https://auth.api.rackspacecloud.com/v2.0/tokens)
Get token with http:
token=$(echo $json | jq '.access | .token | .id' | sed s/\"//g)
Easy token usage for later:
auth="X-Auth-Token:$token"
Get endpoint for Sydney cloud files (change SYD for your favorite Datacenter) (change publicURL to internalURL if you're running from inside the DC):
url=$(echo $json | jq '.access | .serviceCatalog | .[] | select(.name == "cloudFiles") | .endpoints | .[] | select(.region == "SYD") | .publicURL' | sed s/\"//g)
-- Hard work is done. Now it gets easy --
Get list of containers:
http "$url" $auth
Create a container:
http PUT "$url/my_container" $auth
Upload a file:
cat python1.JPG | http PUT "$url/my_container/python1.jpg" $auth
List files:
http "$url/my_container"
Get CDN API URL (not the one for downloading, that's later):
cdn_url=$(echo $json | jq ' .access | .serviceCatalog | .[] | select(.name == "cloudFilesCDN") | .endpoints | .[] | select(.region == "SYD") | .publicURL' | sed s/\"//g)
CDN enable the container:
http PUT "$cdn_url/my_container" $auth "X-Cdn-Enabled: True"
Get public CDN url for my_container:
pub_url=$(http -h HEAD "$cdn_url/my_container" $auth | awk '/X-Cdn-Uri/{print $2;}')
View your file:
firefox "$pub_url/python1.jpg"
All the API docs are here: http://docs.rackspace.com/files/api/v1/cf-devguide/content/API_Operations_for_Storage_Services-d1e942.html
Enjoy :)
This is the pattern which you should be using:
token=`cat /etc/passwd`
echo "file contents: $token"
Note, as triplee points out, that you must not have spaces on either side of the = sign.
I highly recommend skipping curl and using one of the language specific SDKs found on http://developer.rackspace.com
They all handle authentication easily and reauthentication for long lived processes. They all have examples of how to upload files too.