I want to start a number of parallel jobs and I want the result outputs in sequential order. The jobs in my case are HTTP requests sent with curl and I'm interested in the response code only. Here is what I have so far:
for i in {1..6}
do
curl -H "Content-Type: application/json" -X POST \
-d 'some data' \
-s -o /dev/null -w "%{http_code}\n" \
<url of service> &
done
wait
This prints the result code of each request, but not in the correct order. Any way I can correct the order of the output?
It is necessary that the requests are actually sent in parallel.
Store the results to files, then print them out once everything is complete:
for i in {1..6}
do
curl -H "Content-Type: application/json" -X POST \
-d 'some data' \
-s -o /dev/null -w "%{http_code}\n" \
<url of service> > result_$i &
done
wait
for i in {1..6}
do
cat result_$i
rm result_$i
done
Related
I am trying to get the response code of the curl command to a variable and output of the same curl command in a file. Another curl command within a different function will run and the output should append to the same output file.
response=$(curl -i -H "content-type: application/json" -w "%{http_code}" -u "$id:$cred" -H "Accept: application/json" -X POST http://sitename -d '{'<input>'}' >> out.txt
This is giving me neither the response nor the curl output in the file.
Below is working and gives me response as the desired http code though.
response=$(curl -i -H "content-type: application/json" -w "{http_code}" -u "$id:$cred" -H "Accept: application/json" -X POST http://sitename -d '{'<input>'}'
Echo $response
This suppresses the output and just outputs the status:
curl --write-out '%{http_code}' --silent --output /dev/null --noproxy '*' http://www.google.com/
Adding a header makes the entire response print:
curl --write-out '%{http_code}' --silent --output /dev/null --noproxy '*' --header ''"'"'Host:' '192.168.0.1:2345'"'" http://www.google.com/
How can I stop the output printing to stdout when injecting a header?
Let's take a closer look at your command. Below invocation will write each separate argument in a new line.
$ printf '%q\n' curl --write-out '%{http_code}' --silent --output /dev/null --noproxy '*' --header ''"'"'Host:' '192.168.0.1:2345'"'" http://www.google.com/
curl
--write-out
%\{http_code\}
--silent
--output
/dev/null
--noproxy
\*
--header
\'Host:
192.168.0.1:2345\'
http://www.google.com/
Bingo, though Host: 192.168.0.1:2345 must be a single argument, you're providing it to curl as two separate arguments, so it tries to fetch 192.168.0.1:2345' first. And since --output is applied to only one URL, the response from http://www.google.com/ is printed.
Do it like this and it'll work.
curl --write-out '%{http_code}\n' --silent --output /dev/null --noproxy '*' --header 'Host: 192.168.0.1:2345' http://www.google.com/
EDIT:
I have managed to make it work with
response=$(
curl -k -X PUT -d "$marathon_payload" --write-out %{http_code} --silent --output "$tmp"\
-H "Authorization: token=$dcos_token" -H "$header_content_type" $app_id_url
)
The single quotes were causing the problem. It took a few gyrations but all good.
MORAL: quotes inside the value don't matter if the value is properly quoted UNLESS you eval the whole thing, and I should have known that. Occam's wins again.
end edit
I am initiating Mesosphere microservice deployments with curl, but it won't succeed without using eval. Since I recently inherited this code I've been trying to scrub the eval out of it just as a matter of habit, but it's thwarting me.
The script initiates the deployment with
response=$(
eval curl -k -X PUT -d "'$marathon_payload'" --write-out %{http_code} --silent --output $tmp\
-H "'Authorization: token=$dcos_token'" -H "'$header_content_type'" $app_id_url
)
If it gets a 200 or a 201, it loops a curl to effectively screen-scrape the deployments page till the request disappears.
chkDeploy() { rm -f $tmp;
eval curl -k -X GET --silent --write-out %{http_code} --silent --output $tmp\
-H "'Authorization: token=$dcos_token'" -H "'$header_content_type'" $deployments_url
}
response=$( chkDeploy )
$dcos_token is a base64 encoded string.
It then checks the service with another curl loop to the info page so it can verify the version number. This one is working fine with no eval.
chkCode() {
curl -k -X GET --write-out %{http_code} --silent --output $tmp $info_url;
}
response=$( chkCode )
The first two return 401, authentication failure.
I'm guessing the auth token quoting is off.
There's no reason to use eval here; you just need to quote the arguments to -H properly.
response=$(
curl -k -X PUT -d "$marathon_payload" \
--write-out %{http_code} \
--silent --output "$tmp" \
-H "Authorization: token=$dcos_token" \
-H "$header_content_type" "$app_id_url"
)
i have following text file (output.txt)
TECH-746 TECH 10400
TECH-747 TECH 10400
i need to read all columns and pass it to 3 variables and then submit it to curl command. While read simple won't work with curl (don't know why) so i need to use for loop (it works with curl), can i use one for loop or need to nest multiple ones
for project in `cat output.txt`; do
echo $project
curl -D- -u user:pass -X POST --data "{\"fields\":{\"project\":{\"key\":\"TECH\"},\"parent\":{\"key\":\"$project\"},\"summary\":\"test",\"description\":\"test.\",\"issuetype\":{\"name\":\"Sub-task\"}}}" -H "Content-Type:application/json" --silent https://jira.company.com/rest/api/latest/issue/ >/dev/null
code above works, so i just want to "extend" to to include all other columns in file
while read can pull a line into distinct variables.
while read project col2 col3
do
curl -D- -u user:pass -X POST --data "{\"fields\":{\"project\":{\"key\":\"TECH\"},\"parent\":{\"key\":\"$project\"},\"summary\":\"test",\"description\":\"test.\",\"issuetype\":{\"name\":\"Sub-task\"}}}" -H "Content-Type:application/json" --silent https://jira.company.com/rest/api/latest/issue/ >/dev/null
done < sourcefile.txt
EDIT:
The curl command also misses escaping one quote, compare the two lines below, the first one is the original, the second one is the one I've corrected.
curl -D- -u user:pass -X POST --data "{\"fields\":{\"project\":{\"key\":\"TECH\"},\"parent\":{\"key\":\"$project\"},\"summary\":\"test",\"description\":\"test.\",\"issuetype\":{\"name\":\"Sub-task\"}}}" -H "Content-Type:application/json" --silent https://jira.company.com/rest/api/latest/issue/ >/dev/null
curl -D- -u user:pass -X POST --data "{\"fields\":{\"project\":{\"key\":\"TECH\"},\"parent\":{\"key\":\"$project\"},\"summary\":\"test\",\"description\":\"test.\",\"issuetype\":{\"name\":\"Sub-task\"}}}" -H "Content-Type:application/json" --silent https://jira.company.com/rest/api/latest/issue/ >/dev/null
I have a script that ran as a daemon listening to a file:
#!/bin/bash
echo '1'
while inotifywait -e close_write /home/homeassistant/.homeassistant/automations.yaml
do
echo 'automations'
curl -X POST -H "x-ha-access: pass" -H "Content-Type: application/json" http://hassbian.local:8123/api/services/automation/reload
done;
I wanted to listen to several files, and tried adding two more loops:
while inotifywait -e close_write /home/homeassistant/.homeassistant/groups.yaml
do
echo 'gropus'
curl -X POST -H "x-ha-access: pass" -H "Content-Type: application/json" http://hassbian.local:8123/api/services/group/reload
done;
while inotifywait -e close_write /home/homeassistant/.homeassistant/core.yaml
do
echo 'core'
curl -X POST -H "x-ha-access: pass" -H "Content-Type: application/json" http://hassbian.local:8123/api/services/homeassistant/reload_core_config
done;
I realized that the first loop never gets closed so the other ones never get started, but not sure how I should solve this.
You need to run the first loop in a background process so that it doesn't block your script. You may want to run each loop in the background for symmetry, then wait on them at the end of the script.
while inotifywait -e close_write /home/homeassistant/.homeassistant/groups.yaml
do
echo 'gropus'
curl -X POST -H "x-ha-access: pass" -H "Content-Type: application/json" http://hassbian.local:8123/api/services/group/reload
done &
while inotifywait -e close_write /home/homeassistant/.homeassistant/core.yaml
do
echo 'core'
curl -X POST -H "x-ha-access: pass" -H "Content-Type: application/json" http://hassbian.local:8123/api/services/homeassistant/reload_core_config
done &
wait
However, you can run inotifywait in monitor mode and monitor multiple files, piping its output into a single loop. (Caveat: like any line-oriented output format, this cannot cope with filenames containing newlines. See the --format and --csv options for dealing with filenames containing whitespace.)
files=(
/home/homeassistant/.homeassistant/groups.yaml
/home/homeassistant/.homeassistant/core.yaml
)
take_action () {
echo "$1"
curl -X POST "x-ha-access: pass" -H "Content-Type: application/json" \
http://hassbian.local:8123/api/services/"$2"
}
inotifywait -m -e close_write "${files[#]}" |
while IFS= read -r fname _; do
case $fname in
*/groups.yaml) take_action "groups" "group/reload" ;;
*/core.yaml) take_action "core" "homeassistant/reload_core_config" ;;
sac
done