I would like to filter the json object while iterating through it and run curl command over each item from the output.
JSON object.
{
"repo": "releases",
"path": "/apps/releases",
"created": "2021-04-01T10:12:23.496-01:00",
"children": [
{
"uri": "/Image1",
"folder": true,
"created": 2022-08-09T17.12.22.987.04.000
},
{
"uri": "/Image2",
"folder": true,
"created": 2022-06-10T10.12.22.412.10.000
},
{
"uri": "/Image3",
"folder": true,
"created": 2022-10-10T07.03.14.742.01.000
},
{
"uri": "/Image4",
"folder": true,
"created": 2022-10-10T07.010.11.542.08.000
}
]
}
Looking for some logic that will iterate through the uri under children and that is passed through curl command as $i which would be Image1, Image2 and Image3.
curl -k -s --user user:password -X GET "https://artifactory.com/api/releases/baseimage/${i}"
While I was running this below command and the output is as follows
for i in $(curl -k -s --user user:password -X GET "https://artifactory.com/api/releases/baseimage/" | jq -c ".children[] |.uri)
Output: ["/Image1", "/Image2", "/Image3"]
I tried the following command but in the output it replaces ${i} with only Image3, somehow it is not taking Image1 and Image2.
for i in $(curl -k -s --user user:password -X GET "https://artifactory.com/api/releases/baseimage/" | jq -r ".children[] |.uri); do curl -k -s --user user:password -X GET "https://artifactory.com/api/releases/baseimage/${i}"; done
I tried the following command but in the output it replaces ${i} with only Image3, somehow it is not taking Image1 and Image2.
curl can read URLs to fetch from a file, which you can generate with jq. Something like
base=https://artifactory.com/api/releases/baseimage
curl -k -s --user user:password -X GET "$base/" |
jq -r --arg b "$base" '.children[].uri | "url = \"\($b)/\(.)\""' |
curl -k -s --user user:password -X GET --config -
Just one curl process to fetch all the individual images, and no shell loop needed.
You might find it easier to construct the for loop along the following lines:
for uri in $( echo "$json" | jq '.children[].uri') ; do
echo curl ... ${uri}...
done
Related
I want to be use a number of values from jq piped to curl. I have json data with a structure that looks something like this
{
"apps": {
"code": "200",
"name": "app1",
},
"result": "1"
}
{
"apps": {
"code": "200",
"name": "app2",
},
"result": "5"
}
...
...
...
What I want to do is to take the values of every apps.name and result and pass that to curl:
curl -XPOST http://localhost --data-raw 'app=$appsName result=$result'
The curl command will then run X amount of times depending on how many objects there are.
In jq I came up with this but I don't know how or the best way of passing this to curl in a loop.
jq -r '.result as $result | .apps as $apps | $apps.name + " " + $result myfile.json
Maybe something like this?
jq < data.json '[.apps.name, .result]|#tsv' -r | while read appsName result; do
curl -XPOST http://localhost --data-raw "$appsName $result"
done
jq -r '"\( .apps.name ) \( .result )"' |
while read -r app result; do
curl -XPOST http://localhost --data-raw "app=$app result=$result"
done
or
jq -r '"app=\( .apps.name ) result=\( .result )"' |
while read -r data; do
curl -XPOST http://localhost --data-raw "$data"
done
The first one assumes the name can't contain whitespace.
Would this do it?
while read -r line; do
curl -XPOST http://localhost --data-raw "$line"
done < <(jq -r '"app=\(.apps.name) result=\(.result)"' myfile.json)
I think xargs is better than while loops. Maybe just personal preference.
jq -r '"app=\( .apps.name ) result=\( .result )"' \
| xargs -r -I{} curl -XPOST http://localhost --data-raw "{}"
I am trying to run this shell script but the value of val1 is not being replaced in the CURL command
gcs_ip="172.16.0.81:8080"
run(){
val1=$1
echo "Migration with clickhouse worker thread: $1, batchsize: , consumer thread: , pagesize:" | tee -a log_migr.log result_migr.log
echo "curl -X PUT "http://$gcs_ip/gcs/api/v1/config""
curl -X PUT "http://$gcs_ip/gcs/api/v1/config" -H "accept: application/json" -H "Content-Type: application/json" -d '[ { "application": "pm", "instance": "-", "partition": "clickhouse", "name": "migration.clickhouse.workers", "value": "$val1" }]'
}
run 10
I have a curl request that accepts JSON as its payload. The JSON payload is a multi-line string. I'm having trouble piping the output of this curl after the EOF.
curl https://foo.bar/v1/baz \
-H "FOO-BAR: BAZ" \
-X POST -d #- <<'EOF'
{
"foo" : "foo_foo",
"bar": {}
}
EOF
This works, but if I want to pipe the output of this to something, for example python -m json.tool, I have a problem. The following doesn't work:
curl https://foo.bar/v1/baz \
-H "FOO-BAR: BAZ" \
-X POST -d #- <<'EOF'
{
"foo" : "foo_foo",
"bar": {}
}
EOF | python -m json.tool
The here document doesn't begin until the next (logical) line after the command itself. Like any other redirection operator, <<'EOF' does not need to be the final token on the command line.
curl https://foo.bar/v1/baz \
-H "FOO-BAR: BAZ" \
-X POST -d #- <<'EOF' | python -m json.tool
{
"foo" : "foo_foo",
"bar": {}
}
EOF
Trying to send some data via API call (curl) but having issue with the format of the curl command:
serviceMsg="$(cat /tmp/response_time)"
perfData=$(/bin/echo "${serviceMsg}" | /bin/sed 's/,//g')
StatusCode=0
curl -k -s -v -u user:password -H 'Accept: application/json' -X POST 'https://localhost:5665/v1/actions/process-check-result?service=ExternalURL!ResponseTime' -d "{ \"exit_status\": $StatusCode, \"plugin_output\": \"${perfData}\", \"performance_data\": [ ${perfData} ]}"
Here is the error I am getting:
* Connection #0 to host localhost left intact
{
"error": 400.0,
"status": "Invalid request body: Error: lexical error: invalid character inside string.\n { \"exit_status\": 0, \"plugin_ou\n (right here) ------^\n\n"
}
what I am missing ?
Don't hand-code JSON: use jq to generate it for you:
perfData=$(sed 's/,//g' /tmp/response_time)
json=$(jq -n --arg sc "$StatusCode" --arg pd "$perfData" \
'{exit_status: $sc, plugin_output: $pd, performance_data: [ $pd ]}')
url='https://localhost:5665/v1/actions/process-check-result?service=ExternalURL!ResponseTime'
curl -ksv -u user:password -H 'Accept: application/json' -X POST -d "$json" "$url"
So, i'm trying to write a script to create a server on Rackspace Cloud. I've been able to get the script to successfully create a server. Although, after the server is created i need to get a couple of pieces of data out of the output and store them in a variable for later use in the same script (going to do a couple things on the new server after creation).
Here is the script i'm using to create the server:
#!/bin/bash
# Ask user to continue or not
read -p "You are about to setup a new Rackspace cloud server. Would you like to continue (y/n)? " cont
if [ "$cont" == "y" ]; then
echo "Starting server setup script..."
# Ask questions to get login creds
read -p "Rackspace Username? " username
read -p "Rackspace API Key? " apikey
# Get Rackspace token
echo "Getting token..."
token=$(curl -s -X POST https://auth.api.rackspacecloud.com/v2.0/tokens \
-d '{ "auth":{ "RAX-KSKEY:apiKeyCredentials":{ "username":"'"$username"'", "apiKey":"'"$apikey"'" } } }' \
-H "Content-type: application/json" \
| python -m json.tool \
| python -c 'import sys, json; print json.load(sys.stdin)["access"]["token"]["id"]')
echo "...done!"
# Get Rackspace account id
echo "Getting account id..."
account=$(curl -s -X POST https://auth.api.rackspacecloud.com/v2.0/tokens \
-d '{ "auth":{ "RAX-KSKEY:apiKeyCredentials":{ "username":"'"$username"'", "apiKey":"'"$apikey"'" } } }' \
-H "Content-type: application/json" \
| python -m json.tool \
| python -c 'import sys, json; print json.load(sys.stdin)["access"]["token"]["tenant"]["id"]')
echo "...done!"
# Create a new Rackspace cloud server
echo "Creating a new Rackspace cloud server..."
serverpassword=$(curl -s -X POST https://dfw.servers.api.rackspacecloud.com/v2/$account/servers \
-H "Content-Type: application/json" \
-H "X-Auth-Token: $token" \
-H "X-Auth-Project-Id: test-project" \
-T server_build.json \
| python -m json.tool \
| python -c 'import sys, json; print json.load(sys.stdin)["server"]["adminPass"]')
echo "...done!"
echo "Your new server has been created!"
read -p "Would you like to continue on to setting up your new server (y/n)? " cont2
if [ "$cont2" == "y" ]; then
exit
else
echo "Here is your root password for this server (you must write this down now): $serverpassword"
exit
fi
else
echo "You have chosen not to setup a new Rackspace server."
exit
fi
You can see that the token and account variables are being set and used in the creation curl command. The last curl command outputs something like this: (removed real values)
{
"server": {
"OS-DCF:diskConfig": "AUTO",
"adminPass": "************",
"id": "************",
"links": [
{
"href": "https://dfw.servers.api.rackspacecloud.com/v2/...",
"rel": "self"
},
{
"href": "https://dfw.servers.api.rackspacecloud.com/...",
"rel": "bookmark"
}
]
}
}
I'd like to be able to store both the adminPass and id in two different variables. How can i do this? I also know that by solving this issue i will be able to refactor the script when getting token and account values by using one curl command instead of two different ones.
Thanks for any help!
I appreciate all the help. I was able to figure out what i needed to do. Maybe my original question wasn't worded in a way to get the best answers but hopefully this resolution will explain better what i was trying to accomplish.
I ended up using these commands for the 3rd curl command and to set the variables i needed:
serverinfo=$(curl -s https://dfw.servers.api.rackspacecloud.com/v2/731174/servers \
-X POST \
-H "Content-Type: application/json" \
-H "X-Auth-Token: $token" \
-H "X-Auth-Project-Id: test-project" \
-T server_build.json | python -m json.tool)
serverpass=$(echo $serverinfo | python -c 'import sys, json; print json.load(sys.stdin)["server"]["adminPass"]')
This allowed me to run the curl command to setup the server and then grab the adminPass from the output of that command and set it to the serverpass variable for later use.
Using this concept i was also able to remove the 2nd curl command since it was a duplicate command just to set a variable. I can now use these commands to rub the curl once and set both variables from its output:
tokensinfo=$(curl -s -X POST https://auth.api.rackspacecloud.com/v2.0/tokens \
-d '{ "auth":{ "RAX-KSKEY:apiKeyCredentials":{ "username":"'"$username"'", "apiKey":"'"$apikey"'" } } }' \
-H "Content-type: application/json")
token=$(echo $tokensinfo | python -c 'import sys, json; print json.load(sys.stdin)["access"]["token"]["id"]')
account=$(echo $tokensinfo | python -c 'import sys, json; print json.load(sys.stdin)["access"]["token"]["tenant"]["id"]')
Hope that better explains what i was going for. Thanks for the help, it did help me arrive at this solution.
you can grab adminpass and id from the output and store in new variable like
var1=$(grep "adminPass" output.txt | cut -d: --complement -f1)
var2=$(grep "id" output.txt | cut -d: --complement -f1)
echo -e $var1
echo -e $var2
for here i assume the third command output you redirected to output.txt file.