convert some of the response values in curl command - shell

Question

I use the below script to get the average response time for a specific website. It works just fine. I just need to convert some of the values in the response like the time_total, i want to view it in milliseconds and the size_download in KB format. And in the end of the command where I share the average response time, i also want to print it in milliseconds. Any help is really appreciated.
for ((i=1;i<=50;i++)); do curl -w 'Return Code: %{http_code}; Bytes Received: %{size_download}; Response Time: %{time_total}\n' "https://www.google.com" -m 2 -o /dev/null -s; done |tee /dev/tty|awk '{ sum += $NF; n++ } END { If (n > 0); print "Average Response Time =",sum /n;}'

Answer 1

Three answer here...
As this question is tagged shell (not bash), here is three different way for doing this, by using awk + bash, shell + bc or bash alone.
1. Using awk to process output and compute averages
As doing forks like curl ... | awk ... repetitively, is resource killer, I prefer to run awk only once, doing the whole formatting output job under awk:
iter=10
for ((i=iter;i--;));do
curl -w '%{http_code} %{size_download} %{time_total}\n' \
"https://www.google.com" -m 2 -o /dev/null -s
sleep .02
done | awk -v iter=$iter '
BEGIN {
ttim=0;tsiz=0;
printf " %-3s %8s %11s %10s\n","Res","Size","Time","Rate"
};
{
printf " %-3s %7.2fK %9.2fms %7.2fK/s\n", \
$1, $2/1024,$3*1000,$2/$3/1024;
tsiz+=$2;ttim+=$3;
};
END {
printf "Tot %7.2fK %9.2fms\nAvg %7.2fK %9.2fms %7.2fK/s\n", \
tsiz/1024,ttim*1000, tsiz/iter/1024,ttim/iter*1000,tsiz/ttim/1024;
}'
May produce:
Res Size Time Rate
200 14.61K 128.48ms 113.71K/s
200 14.75K 131.06ms 112.52K/s
200 14.73K 131.71ms 111.85K/s
200 14.72K 130.24ms 113.05K/s
200 14.66K 134.68ms 108.86K/s
200 14.69K 131.39ms 111.79K/s
200 14.63K 131.15ms 111.53K/s
200 14.70K 126.26ms 116.42K/s
200 14.71K 129.08ms 113.98K/s
200 14.68K 131.23ms 111.86K/s
Tot 146.88K 1305.28ms
Avg 14.69K 130.53ms 112.53K/s
2. Working on averages using shell
As shell don't work with floating numbers, you have to use bc or another subprocess to resolv this operations.
As doing forks like var=$(echo '3*4'|bc) is resource killer, I prefer to run bc only once, as a background process. One advantage of this is bc can store overall variables (total download and total size here).
First part: Init some variables and run bc
Creating two fifos for backgrounded bc and one fifo for curl, making parsing of curl's output easier.
Declaring some variables, and numfmt function into bc for further use...
Note about numfmt: This function compute human readable presentation from integer value, output two values
octal character octal value of b, K, M, G, P and T and
floating number from submited value, divided by power of 1024
The character from octal could be output by printf '%b' \\$value under shell.
#!/bin/sh
target=${1:-https://www.google.com}
iter=${2:-10}
delay=${3:-.02}
tempdir=$(mktemp -d)
bcin="$tempdir/bcin" bcout="$tempdir/bcout" curlout="$tempdir/curlout"
mkfifo "$bcin" "$bcout" "$curlout"
exec 3<>"$bcin"; exec 4<>"$bcout"
cleanUp() { [ -e "$bcin" ] && rm "$bcin" "$bcout" "$curlout" && rmdir "$tempdir"
exit;}
trap cleanUp 0 1 2 3 6 15
bc -l <&3 >&4 &
mBc() { echo >&3 "$2"; read -r $1 <&4 ;}
cat >&3 <<EOInitBc
unit[0]=142;unit[1]=113;unit[2]=115;unit[3]=107;unit[4]=124;unit[5]=120
define void numfmt (s) {
if (s==0) { print "0,0\n"; return;};
p=l(s)/l(1024);
scale=0;
p=p/1;
scale=20;
print unit[p]," ",s/1024^p,"\n";
};
tsiz=0;ttim=0;
EOInitBc
# read variables from curl
checkHttpRes() {
curl -sm2 -w'%{http_code} %{size_download} %{time_total}' -o/dev/null \
"$1" >"$curlout" &
read -r cod siz tim <"$curlout"
mBc rate "tsiz+=$siz;ttim+=$tim;$siz/$tim;"
mBc mtim "$tim*1000"
mBc 'hunit hsz' "numfmt($siz)"
mBc 'hurt hrat' "numfmt($rate)"
printf ' %-3s %7.2f%b %9.2fms %7.2f%b/s\n' "$cod" "$hsz" "\\$hunit" \
"$mtim" "$hrat" "\\$hurt"
}
# Last part: main routine
printf ' %-3s %8s %11s %10s\n' Res Size Time Rate
i="$iter"
while [ "$i" -gt 0 ];do
checkHttpRes "$target"
sleep "$delay"
i=$((i-1))
done
mBc 'hutsz htsz' "numfmt(tsiz)"
mBc 'huasz hasz' "numfmt(tsiz/$iter)"
mBc ttim "1000*ttim"
mBc atim "1000*ttim/$iter"
mBc 'huart hart' "numfmt(tsiz/ttim)"
printf 'Tot %7.2f%b %9.2fms\nAvg %7.2f%b %9.2fms %7.2f%b/s\n' \
"$htsz" "\\$hutsz" "$ttim" "$hasz" "\\$huasz" "$atim" "$hart" "\\$huart"
Run sample:
$ ./curlStat.sh http://www.google.com 10 .1
Res Size Time Rate
200 14.76K 141.84ms 104.09K/s
200 14.65K 136.21ms 107.53K/s
200 14.61K 136.74ms 106.86K/s
200 14.67K 138.08ms 106.26K/s
200 14.70K 130.56ms 112.56K/s
200 14.65K 135.72ms 107.97K/s
200 14.68K 135.28ms 108.53K/s
200 14.64K 134.20ms 109.07K/s
200 14.70K 136.32ms 107.82K/s
200 14.71K 136.19ms 108.00K/s
Tot 146.77K 1361.14ms
Avg 14.68K 136.11ms 107.83K/s
3. Last: bash
This question is tagged sh, but for ((i=... syntax is a bashism. So here is a compact pure bash version of this:
#!/bin/bash
target=${1:-https://www.google.com} iter=${2:-10} delay=${3:-.02}
txsz(){ local i=$(($1>=1<<50?5:$1>=1<<40?4:$1>=1<<30?3:$1>=1<<20?2:$1>1023?1:0
)) a=(b K M G T P);((i>4?i+=-2:0))&&a=(${a[#]:2})&&set -- $(($1>>20)) $2;local\
r=00$((1000*$1/(1024**i)));printf -v $2 %.2f%s ${r::-3}.${r: -3} ${a[i]};}
declare -i ttim=0 tsiz=0
checkHttpRes() { local code size time ustim hsz hrat
read -r code size time < <(
curl -sm2 -w'%{http_code} %{size_download} %{time_total}' -o/dev/null "$1"
)
printf -v ustim '%.6f' "$time"
ustim=$((10#${ustim/.}))
tsz "$size" hsz
tsz $(( size*10**7/ustim/10 )) hrat
ttim+=ustim tsiz+=size ustim=00$ustim
printf ' %-3s %8s %9.2fms %8s/s\n' "$code" "$hsz" \
"${ustim::-3}.${ustim: -3}" "$hrat"
}
printf ' %-3s %8s %11s %10s\n' Res Size Time Rate
for ((i=iter;i--;)) ;do
checkHttpRes "$target"
sleep "$delay"
done
tsz $tsiz htsz
ustim=00$ttim uatim=00$((ttim/iter))
tsz $((tsiz/iter)) hasz
tsz $(( tsiz*10**7/ttim/10 )) hart
printf 'Tot %8s %9.2fms\nAvg %8s %9.2fms %8s/s\n' "$htsz" \
"${ustim::-3}.${ustim: -3}" "$hasz" "${uatim::-3}.${uatim: -3}" "$hart"
Without any fork to bc nor awk, all operation are done in pseudo float using shifted integers.
This will produce same result:
Res Size Time Rate
200 14.68K 132.79ms 110.55K/s
200 14.68K 135.59ms 108.24K/s
200 14.68K 132.31ms 110.99K/s
200 14.75K 141.66ms 104.15K/s
200 14.66K 139.90ms 104.79K/s
200 14.71K 140.07ms 105.00K/s
200 14.68K 142.74ms 102.86K/s
200 14.64K 133.42ms 109.71K/s
200 14.72K 135.62ms 108.56K/s
200 14.71K 139.16ms 105.72K/s
Tot 146.92K 1373.25ms
Avg 14.69K 137.32ms 106.98K/s

Answer 2

You can just pipe the curl output through awk and format it as you want like this:
curl -w 'Return Code: %{http_code}; Bytes Received: %{size_download}; Response Time: %{time_total}\n' "https://www.google.com" -m 2 -o /dev/null -s | awk '{printf "Return Code: %d; KiB Received: %f; Response Time(ms): %f\n", $3, $6/1024, $9*1000}'
So the oneliner is the following:
for ((i=1;i<=50;i++)); do curl -w 'Return Code: %{http_code}; Bytes Received: %{size_download}; Response Time: %{time_total}\n' "https://www.google.com" -m 2 -o /dev/null -s | awk '{printf "Return Code: %d; KiB Received: %f; Response Time(ms): %f\n", $3, $6/1024, $9*1000}'; done | tee /dev/tty |awk '{ sum += $NF; n++ } END { If (n > 0); print "Average Response Time =",sum /n;}'
You can also format the numbers as you want by puttingg, for example %.2f for 2 decimal precision or %d for integer...