For loop and confition including Curl command - bash

I am trying to write a script where I enumerate users by checking the HTTP reponse length. I want to get output "good" when response is not equal 23, however I get these errors now:
for ((i=113;i<=115;i++)); do
if [[curl -i -s -k -X 'GET' "http://myurl/some.asp?q=$i" |
grep Content-Length | cut -d' ' -f2 != 23]]
then
echo "good"
fi
done
Output:
bash: [[curl: command not found
cut: !=: No such file or directory
cut: 23]]: No such file or directory
cut: !=: No such file or directory
cut: 23]]: No such file or directory
bash: [[curl: command not found
cut: !=: No such file or directory
cut: 23]]: No such file or directory
bash: [[curl: command not found
If I simply make a script without if condition, then it works well:
for ((i=113;i<=115;i++)); do
curl -i -s -k -X 'GET' "http://myurl/some.asp?q=$i" |
grep Content-Length
done
I checked many examples but can't seem to figure out where I am doing wrong.

After updating your initial error, you may have a syntax like (suggestion: put some effort on format, so that it is more clear what you have and what may be wrong):
for ((i=113;i<=115;i++))
do
if [[ curl -i -s -k -X 'GET' "http://myurl/some.asp?q=$i" | grep Content-Length | cut -d' ' -f2 != 23 ]]
then
echo "good"
fi
done
This is returning you the error:
bash: conditional binary operator expected bash: syntax error near
-i'`
Which is normal, because you are basically saying:
if [[ command ]]; then ...
Where command is a set of multiple piped commands. However, in [[ you just can add expressions on the form "$var" -eq 23 or "$(command)" -ne 23.
So use $( ) to execute the command: if [[ "$(command)" -ne 23 ]]:
if [[ "$(curl -i -s -k -X 'GET' "http://myurl/some.asp?q=$i" | grep Content-Length | cut -d' ' -f2)" -ne 23 ]]
Note I am using -ne to perform an integer comparison that means "not equal to".
Finally, notice that awk alone can do what grep and cut do in two steps:
... | grep "Content-Length" | cut -d' ' -f2
This means: check the line containing "Content-Length" and print its second field. awk makes it simple by saying:
... | awk '/Content-Length/ {print $2}'
And last, but not least, your expression for ((i=113;i<=115;i++)) can be also written as for i in {113..115} using brace expansion.

If you want to test a result of the command execution you should put it into $() . So the resulting script should look as follows:
for i in {113..115}; do if [[ $(curl -i -s -k -X 'GET' "http://myurl/some.asp?q=$i" | grep Content-Length | cut -d' ' -f2) != 23 ]]; then echo "good" ; fi; done
Also I've changed a way you iterate the values. {a..b} in bash provides a sequence from 'a' to 'b'.

Related

Bash, loop unexpected stop

I'm having problems with this last part of my bash script. It receives input from 500 web addresses and is supposed to fetch the server information from each. It works for a bit but then just stops at like the 45 element. Any thoughts with my loop at the end?
#initializing variables
timeout=5
headerFile="lab06.output"
dataFile="fortune500.tsv"
dataURL="http://www.tech.mtu.edu/~toarney/sat3310/lab09/"
dataPath="/home/pjvaglic/Documents/labs/lab06/data/"
curlOptions="--fail --connect-timeout $timeout"
#creating the array
declare -a myWebsitearray
#obtaining the data file
wget $dataURL$dataFile -O $dataPath$dataFile
#getting rid of the crap from dos
sed -n "s/^m//" $dataPath$dataFile
readarray -t myWebsitesarray < <(cut -f3 -d$'\t' $dataPath$dataFile)
myWebsitesarray=("${myWebsitesarray[#]:1}")
websitesCount=${#myWebsitesarray[*]}
echo "There are $websitesCount websites in $dataPath$dataFile"
#echo -e ${myWebsitesarray[200]}
#printing each line in the array
for line in ${myWebsitesarray[*]}
do
echo "$line"
done
#run each website URL and gather header information
for line in "${myWebsitearray[#]}"
do
((count++))
echo -e "\\rPlease wait... $count of $websitesCount"
curl --head "$curlOptions" "$line" | awk '/Server: / {print $2 }' >> $dataPath$headerFile
done
#display results
echo "Results: "
sort $dataPath$headerFile | uniq -c | sort -n
It would certainly help if you actually passed the --connect-timeout option to curl. As written, you are currently passing the single argument --fail --connect-timeout $timeout rather than 3 distinct arguments --fail, --connect-timeout, and $timeout. This is one instance where you should not quote the variable. IOW, use:
curl --head $curlOptions "$line"

curl in bash script vs curl one liner

This code ouputs a http status of 000 - which seems to indicate something didn't connect properly but when I do this curl outside of the bash script it works fine and produces a 200 so something with this code is off... any guidance?
#!/bin/bash
URLs=$(< test.txt | grep Url | awk -F\ ' { print $2 } ')
# printf "Preparing to check $URLs \n"
for line in $URLs
do curl -L -s -w "%{http_code} %{url_effective}\\n" $line
done
http://beerpla.net/2010/06/10/how-to-display-just-the-http-response-code-in-cli-curl/
your script works on my vt.
I added in a couple of debugging lines, this may help you to see where any metacharacters are getting in, as I would have to agree with the posted coments.
I've output lines in the for to a file which is then printed out with od.
I have amended the curl line to grab the last line, just to get the response code.
#!/bin/bash
echo -n > $HOME/Desktop/urltstfile # truncate urltstfile
URLs=$(cat testurl.txt | grep Url | awk -F\ ' { print $2 } ')
# printf "Preparing to check $URLs \n"
for line in $URLs
do echo $line >> $HOME/Desktop/urltstfile;
echo line:$line:
curl -IL -s -w "%{http_code}\n" $line | tail -1
done
od -c $HOME/Desktop/urltstfile
#do curl -L -s -w "%{http_code} %{url_effective}\\n" "$line\n"

How to redirect grep to a while loop

Hi I have the following bash script code
group2=0
while read -r line
do
popAll=$line | cut -d "{" -f2 | cut -d "}" -f1 | tr -cd "," | wc -c
if [[ $popAll = 0 ]]; then
group2 = $((group2+2));
else
group2 = $((group2+popAll+1));
fi
done << (grep -w "token" "$file")
and I get the following error:
./parsingTrace: line 153: syntax error near unexpected token `('
./parsingTrace: line 153: `done << (grep -w "pop" "$file")'
I do not want to pipe grep to the while, because I want variable inside the loop to be visible outside
The problem is in this line:
done << (grep -w "token" "$file")
# ^^
You need to say < and then <(). The first one is to indicate the input for the while loop and the second one for the process substitution:
done < <(grep -w "token" "$file")
# ^ ^
Note however that there are many others things you want to check. See the comments for a discussion and paste the code in ShellCheck for more details. Also, by indicating some sample input and desired output I am sure we can find a better way to do this.

[: : bad number on the bash script

This is my bash script:
#!/usr/local/bin/bash -x
touch /usr/local/p
touch /usr/local/rec
DATA_FULL=`date +%Y.%m.%d.%H`
CHECK=`netstat -an | grep ESTAB | egrep '(13001|13002|13003|13004|13061|13099|16001|16002|16003|16004|16061|16099|18001|18002|18003|18004|18061|18099|20001|20002|20003|20004|20061|20099|13000|16000|18000|20000)' | awk '{ print $5 }' | sort -u | wc -l`
netstat -an | grep ESTAB | egrep '(13001|13002|13003|13004|13061|13099|16001|16002|16003|16004|16061|16099|18001|18002|18003|18004|18061|18099|20001|20002|20003|20004|20061|20099|13000|16000|18000|20000)' | awk '{ print $5 }' | sort -u | wc -l > /usr/local/www/p
STAT=`cat /usr/local/www/rec`
if [ "$CHECK" -gt "$STAT" ]; then
echo $CHECK"\n"$DATA_FULL > /usr/local/p
fi
Ofcourse I've runned chmod +x script.sh and then sh script.sh, then I receive the following message: [: : bad number.
Why does it happends?
Run your script using
sh -x script.sh
It'll print every line it executes and the variable output.
Run the netstat command and stat command outside and check.
If these are integer for sure, use this syntax,
if [ "0$(echo $CHECK|tr -d ' ')" -gt "0$(echo $STAT|tr -d ' ')" ];
A simple hack. Only works if $STAT is always either empty or positive number.
Are you sure that both STAT and CHECK are numbers that can be compared with -gt?
probably your /usr/local/www/rec is empty. Try
STAT=`cat /usr/local/www/rec 2>/dev/null || echo 0`
maybe.

AWK: execute CURL on each line and parse result

given an input stream with following lines:
123
456
789
098
...
I would like to call
curl -s http://foo.bar/some.php?id=xxx
with xxx being the number for each line, and everytime let an awk script fetch some information from the curl output which is written to the output stream. I am wondering if this is possible without using the awk "system()" call in following way:
cat lines | grep "^[0-9]*$" | awk '
{
system("curl -s " $0 \
" | awk \'{ #parsing; print }\'")
}'
You can use bash and avoid awk system call:
grep "^[0-9]*$" lines | while read line; do
curl -s "http://foo.bar/some.php?id=$line" | awk 'do your parsing ...'
done
A shell loop would achieve a similar result, as follows:
#!/bin/bash
for f in $(cat lines|grep "^[0-9]*$"); do
curl -s "http://foo.bar/some.php?id=$f" | awk '{....}'
done
Alternative methods for doing similar tasks include using Perl or Python with an HTTP client.
If your file gets dynamically appended the id's, you can daemonize a small while loop to keep checking for more data in the file, like this:
while IFS= read -d $'\n' -r a || sleep 1; do [[ -n "$a" ]] && curl -s "http://foo.bar/some.php?id=${a}"; done < lines.txt
Otherwise if it's static, you can change the sleep 1 to break and it will read the file and quit when there is no data left, pretty useful to know how to do.

Resources