I am trying to write a script to test if my firewall is blocking multiple pages. When I try to read one or multiple lines from command line I get a -ne: unitary operator expected error.
This is my attempt:
sh test www.3232.com.pe
www.3232.com.pe
test: line 7: [: -ne: unary operator expected
$ cat test
#!/bin/sh
for var in "$#"
do
echo $var
res=`curl -s -I $var | grep HTTP/1.1 | awk {'print $2'}`
if [ $res -ne 200 ]
then
echo "Error on $var"
fi
done
Consider what happens if curl does not successfully connect:
res=`curl -s -I $var | grep HTTP/1.1 | awk {'print $2'}`
if [ $res -ne 200 ]
If curl does not get any headers, or does not get a HTTP/1.1 header, then the pipline in the first command outputs nothing and res is assigned to nothing. In that case, the test in the second line will fail with an unexpected operator error.
You need to first test that res is nonempty. And, if it is empty, take an appropriate action.
Related
I am trying to write a script where I enumerate users by checking the HTTP reponse length. I want to get output "good" when response is not equal 23, however I get these errors now:
for ((i=113;i<=115;i++)); do
if [[curl -i -s -k -X 'GET' "http://myurl/some.asp?q=$i" |
grep Content-Length | cut -d' ' -f2 != 23]]
then
echo "good"
fi
done
Output:
bash: [[curl: command not found
cut: !=: No such file or directory
cut: 23]]: No such file or directory
cut: !=: No such file or directory
cut: 23]]: No such file or directory
bash: [[curl: command not found
cut: !=: No such file or directory
cut: 23]]: No such file or directory
bash: [[curl: command not found
If I simply make a script without if condition, then it works well:
for ((i=113;i<=115;i++)); do
curl -i -s -k -X 'GET' "http://myurl/some.asp?q=$i" |
grep Content-Length
done
I checked many examples but can't seem to figure out where I am doing wrong.
After updating your initial error, you may have a syntax like (suggestion: put some effort on format, so that it is more clear what you have and what may be wrong):
for ((i=113;i<=115;i++))
do
if [[ curl -i -s -k -X 'GET' "http://myurl/some.asp?q=$i" | grep Content-Length | cut -d' ' -f2 != 23 ]]
then
echo "good"
fi
done
This is returning you the error:
bash: conditional binary operator expected bash: syntax error near
-i'`
Which is normal, because you are basically saying:
if [[ command ]]; then ...
Where command is a set of multiple piped commands. However, in [[ you just can add expressions on the form "$var" -eq 23 or "$(command)" -ne 23.
So use $( ) to execute the command: if [[ "$(command)" -ne 23 ]]:
if [[ "$(curl -i -s -k -X 'GET' "http://myurl/some.asp?q=$i" | grep Content-Length | cut -d' ' -f2)" -ne 23 ]]
Note I am using -ne to perform an integer comparison that means "not equal to".
Finally, notice that awk alone can do what grep and cut do in two steps:
... | grep "Content-Length" | cut -d' ' -f2
This means: check the line containing "Content-Length" and print its second field. awk makes it simple by saying:
... | awk '/Content-Length/ {print $2}'
And last, but not least, your expression for ((i=113;i<=115;i++)) can be also written as for i in {113..115} using brace expansion.
If you want to test a result of the command execution you should put it into $() . So the resulting script should look as follows:
for i in {113..115}; do if [[ $(curl -i -s -k -X 'GET' "http://myurl/some.asp?q=$i" | grep Content-Length | cut -d' ' -f2) != 23 ]]; then echo "good" ; fi; done
Also I've changed a way you iterate the values. {a..b} in bash provides a sequence from 'a' to 'b'.
So I'm writing a script that will take the output of a grep as an array and then iterate a filter over it to output to a file. I'm testing it on my own site and the wget works as expected and generates a list of URLS in the spider.queue. The grep command will also work in terms of filtering by a keyword, but when I add it into a while loop and use an if statement to check if it already exists I'll get the error;
./spider.sh: 19: ./spider.sh: Syntax error: "(" unexpected (expecting "done")
Which would lead me to believe it's a syntax problem with one of the loops.
#!/bin/sh
# Usage - ./spider.sh searchterm www.website.com
## Parameters
search=$1
URL=$2
## Spider WGET
wget -r -e robots=off --header="Accept: text/html" --user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:21.0) Gecko/20100101 Firefox/21.0" http://$URL 2>&1 | grep '^--' 2>&1 | awk '{ print $3 }' | grep -v '\.\(css\|js\|png\|gif\|jpg\|JPG\)$' >> spider.queue
## Keyword filter with grep
while true
do
PROFILES=($(grep -l -r "$search" $URL))
for x in ${PROFILES[*]}
do
if grep -q $x crawler.queue; then
echo "Already Exists"
else
$x >> crawler.queue
fi
done
done
/bin/sh does not support arrays, so the syntax error is caused by PROFILES=(....
Switch to #!/bin/bash.
I have this bash script:
How I can call it the grep's output in the if else statement? If I suppress the output of the grep command with -q, also will work?
#!/usr/bin/env bash
DOMAINS=( '.com' '.biz' )
while read input; do
for (( i=0;i<${#DOMAINS[#]};i++)); do
jwhois --force-lookup --disable-cache --no-redirect -c jwhois.conf "$input${DOMAINS[$i]}" | MATCH="$(grep -oPa '^.*\b(clientTransferProhibited)\b.*$')"
if [ $? -eq 0 ]; then
echo -e "$input${DOMAINS[$i]}\tregistered\t" $(date +%y/%m/%d_%H:%M:%S) "\t" "$MATCH" |& tee --append output/registered.txt
else
echo -e "$input${DOMAINS[$i]}\tavailable\t" $(date +%y/%m/%d_%H:%M:%S) "\t" "$MATCH" |& tee --append output/available.txt
fi
done
done < "$1"
So MATCH="$(grep -oPa '^.*\b(clientTransferProhibited) used by "$MATCH" in the if else statement not outputting anything, but if I just use grep -oPa '^.*\b(clientTransferProhibited), it's printing the line without problem. The reason why I want to use as a variable, because I want to put in specific places in the if else statements.
Actual output:
$ domain1.com available 15/11/16_14:13:05
$ domain1.biz available 15/11/16_14:13:05
$ domain2.com registered 15/11/16_14:13:05
$ domain2.biz registered 15/11/16_14:13:05
Output that I want:
$ domain1.com available 15/11/16_14:13:05
$ domain1.biz available 15/11/16_14:13:05
$ domain2.com registered 15/11/16_14:13:05 Status: clientTransferProhibited http://www.icann.org/epp#clientTransferProhibited
$ domain2.biz registered 15/11/16_14:13:05 Status: clientTransferProhibited http://www.icann.org/epp#clientTransferProhibited
So the result of this line is what you want stored in a variable...
jwhois --force-lookup --disable-cache --no-redirect -c jwhois.conf "$input${DOMAINS[$i]}" | grep -oPa '^.*\b(clientTransferProhibited)\b.*$'
In that case, you want to evaluate that line and set the entire thing to your desired variable. The type of assignment you attempt in the middle of a pipeline will not be valid.
MATCH=$(jwhois --force-lookup --disable-cache --no-redirect -c jwhois.conf "$input${DOMAINS[$i]}" | grep -oPa '^.*\b(clientTransferProhibited)\b.*$')
Those are two different MATCH variables, because pipes execute subshells:
Bash subshell/pipelines - which parts are executing in subshells?
Try it this way instead:
MATCH="$(jwhois --force-lookup --disable-cache --no-redirect -c jwhois.conf "$input${DOMAINS[$i]}" | grep -oPa '^.*\b(clientTransferProhibited)\b.*$')"
response=$(curl -sL -w \\n%{http_code} "http://<ip_addr>/api/1/app" -X DELETE)
echo response
if [ "$response" -eq 200 ]
then
echo "Got 200 OK"
else
echo "not getting the result"
fi
What i'm trying to do is to get the http response code.
I'm positive that the response should be 200 OK
When I run the script I'm getting
{
"result":true
}
200
tst.sh: line 302: [: {
200: integer expression expected
I even don't want to display
{
"result":true
}
I just want to print 200 and compare 200.
Just with curl command:
curl -sL -w '%{http_code}' "http://<ip_addr>/api/1/app" -X DELETE -o /dev/null
Get the last line of the output.
response=$(curl -sL -w \\n%{http_code} "http://<ip_addr>/api/1/app" -X DELETE | tail -1)
Besides tail -n you can also use:
awk 'END { print }'
sed -n '$p'
Another way if you're using bash is to remove everything before the last line:
shopt -s extglob
response=${response##*[[:space:]]}
In your above, you are missing a $ in your first echo, but no matter. You can strip the {"result":true} from the $response string
before the if statement:
response=${response#*\}\ }
I have a simple BASH shell script which checks the HTTP response code of a curl command.
The logic is fine, but I am stuck on "simply" printing out the "output".
I am using GNU bash, version 3.2.25(1)-release (x86_64-redhat-linux-gnu)
I would like to output the URL with a tab - then the 404|200|501|502 response. For example:
http://www.google.co.uk<tab>200
I am also getting a strange error where the "http" part of a URL is being overwritten with the 200|404|501|502. Is there a basic BASH shell scripting (feature) which I am not using?
thanks
Miles.
#!/bin/bash
NAMES=`cat $1`
for i in $NAMES
do
URL=$i
statuscode=`curl -s -I -L $i |grep 'HTTP' | awk '{print $2}'`
case $statuscode in
200)
echo -ne $URL\t$statuscode;;
301)
echo -ne "\t $statuscode";;
302)
echo -ne "\t $statuscode";;
404)
echo -ne "\t $statuscode";;
esac
done
From this answer you can use the code
response=$(curl --write-out %{http_code} --silent --output /dev/null servername)
Substituted into your loop this would be
#!/bin/bash
NAMES=`cat $1`
for i in $NAMES
do
URL=$i
statuscode=$(curl --write-out %{http_code} --silent --output /dev/null $i)
case $statuscode in
200)
echo -e "$URL\t$statuscode" ;;
301)
echo -e "$URL\t$statuscode" ;;
302)
echo -e "$URL\t$statuscode" ;;
404)
echo -e "$URL\t$statuscode" ;;
* )
;;
esac
done
I've cleaned up the echo statements too so for each URL there is a new line.
try
200)
echo -ne "$URL\t$statuscode" ;;
I'm taking a stab here, but I think what's confusing you is the fact that curl is sometimes returning more than one header info (hence more than one status code) when the initial request gets redirected.
For example:
[me#hoe]$ curl -sIL www.google.com | awk '/HTTP/{print $2}'
302
200
When you're printing that in a loop, it would appear that the second status code has become part of the next URL.
If this is indeed your problem, then there are several ways to solve this depending on what you're trying to achieve.
If you don't want to follow redirections, simple leave out the -L option in curl
statuscode=$(curl -sI $i | awk '/HTTP/{print $2}')
To take only the last status code, simply pipe the whole command to tail -n1 to take only the last one.
statuscode=$(curl -sI $i | awk '/HTTP/{print $2}' | tail -n1)
To show all codes in the order, replace all linebreaks with spaces
statuscode=$(curl -sI $i | awk '/HTTP/{print $2}' | tr "\n" " ")
For example, using the 3rd scenario:
[me#home]$ cat script.sh
#!/bin/bash
for URL in www.stackoverflow.com stackoverflow.com stackoverflow.com/xxx
do
statuscode=$(curl -siL $i | awk '/^HTTP/{print $2}' | tr '\n' ' ')
echo -e "${URL}\t${statuscode}"
done
[me#home]$ ./script.sh
www.stackoverflow.com 301 200
stackoverflow.com 200
stackoverflow.com/xxx 404