I am having an issue that i cant seem to figure out after a couple hours of tinkering with this. I cant seem this script to return anything to the final column.
#!/bin/bash
file="/Users/USER12/Desktop/url-list.txt"
log="/Users/USER12/Desktop/url-results.txt"
fmt="%-25s%-12s%-16s%-20s\n"
printf "$fmt" DOMAIN_NAME HTTP_CODE RESPONSE_TIME CONTENT_CHECK > "$log"
while read line
do
read code time < <(curl -o /dev/null --silent --head --write-out '%{http_code} %{time_total}' "$line")
curl "$line" 2>/dev/null > /Users/USER12/Desktop/domainQueryString_output.txt
ifStatementConditional=`grep "THE CONTENT I'M LOOKING TO VERIFY" /Users/USER12/Desktop/domainQueryString_output.txt | wc -l`
if [ $ifStatementConditional -eq 1 ] ; then
second_check="online"
else
second_check="DOMAIN IS OFFLINE"
fi
printf "$fmt" "$line" "$code" "$time" "$second_chance" >> "$log"
done <"$file"
It returns the following but nothing to the final column....
DOMAIN_NAME HTTP_CODE RESPONSE_TIME CONTENT_CHECK
google.com 301 1.177
Thanks for the help guys. The Help is Much Appreciated.
You have "$second_chance" where you should have had "$second_check".
Other then that, the following is a better way to do your if check:
if grep "THE CONTENT I'M LOOKING TO VERIFY" $yourfile -q
then
...
else
...
fi
Related
Looking to change these url's: Looking at cut -d'/' -f1,2,3,.. test Not sure if this the best method?
https://www.example.com/images/fdgdfg4wretrgretg/cad0be71.jpg
https://www.example.com/images/fdgdfg4w/cad0be71/32234/234234.jpg
https://www.example.com/images//77777/dfgdfgdfg/cad0be71.jpg
Connvert into:
https://www.example.com/images/fdgdfg4wretrgretg/
https://www.example.com/images/fdgdfg4w/cad0be71/
https://www.example.com/images//77777/
Why not using bash directly?
declare regexp='^(https?://[^/]+/[^/]+)(/.*)?$'
if [[ "$url" =~ $regexp ]]; then
url="${BASH_REMATCH[1]}"
fi
Since it supports regexp, this should work for most case.
And in a loop:
declare regexp='^(https?://[^/]+/[^/]+)(/.*)?$'
while read -r url; do
if [[ "$url" =~ $regexp ]]; then
url="${BASH_REMATCH[1]}"
echo "url: $url"
else
echo "error: mismatch $url"
fi
done < domains.txt
Or use sed directly:
sed -E -e 's#^(https?://[^/]+/[^/]+)(/.*)?$#\1#g' domains.txt
And if you need to do something more in bash (you may consider reading the link from Ed Morton comment):
while read -r url; do
echo "url: $url"
done < <(sed -E -e 's#^(https?://[^/]+/[^/]+)(/.*)?$#\1#g' domains.txt)
I'm trying to create a bash script that check url from list status code and echo server name from header. I'm actually new.
#!/bin/bash
while read LINE; do
curl -o /dev/null --silent --head --write-out '%{http_code}' "$LINE"
echo " $LINE" &
curl -I /dev/null --silent --head | grep -Fi Server "$SERVER"
echo " $SERVER"
done < dominios-https
I get the following output
301 http://example.com
grep: : No such file or directory
1) while read LINE can not use last line if text file not ended with new line.
2) You don't set "$SERVER" anywhere, and grep say it
3) Not all servers return "Server:" in headers
try it:
scriptDir=$( dirname -- "$0" )
for siteUrl in $( < "$scriptDir/myUrl.txt" )
do
if [[ -z "$siteUrl" ]]; then break; fi # break line if him empty
httpCode=$( curl -I -o /dev/null --silent --head --write-out '%{http_code}' "$siteUrl" )
echo "HTTP_CODE = $httpCode"
headServer=$( curl -I --silent --head "$siteUrl" | grep "Server" | awk '{print $2}' )
echo "Server header = $headServer"
done
so the error I'm getting is syntax error near unexpected token `fi' on the second last 'fi'. Been scratching my head for a while about this. Any help is greatly appreciated! Thanks!
#!/bin/bash
TFILE=/tmp/scripts/pdsh_demo.tmp
if [ -f $TFILE ]; then
rm $TFILE
fi
/usr/bin/pdsh -R ssh -w host[0001-0200] 'command | grep -v "something"' >> $TFILE
if [ ! -s $TFILE ]; then
exit
fi
if [ -f $TFILE ]; then
if grep -q "something" $TFILE ; then
grep -i "something" $TFILE | mailx -r "test.server" -s "Critical: something" -a $TFILE "test#test.com"
fi
fi
you should make this:
if grep -q "something" $TFILE ; then
into this:
if [ $(grep -q "something" $TFILE) ] ; then
add brackets around your args
space before & after brackets
put then on a new line
in
if grep -q "something" $TFILE ; then
should be
if [ $(grep -q "something" $TFILE) ]; then
First time posting. HELLO WORLD. Working on my first script that just simply checks if a list of my websites are online and then returns the HTTP code and the amount of time it took to return that to another file on my desktop.
-- THIS SCRIPT WILL BE RUNNING ON MAC OSX --
I would like to amend my script so that it formats its output into 3 neat columns.
currently
#!/bin/bash
file="/Users/USER12/Desktop/url-list.txt"
printf "" > /Users/USER12/Desktop/url-results.txt
while read line
do
printf "$line" >> /Users/USER12/Desktop/url-results.txt
printf "\t\t\t\t" >> /Users/USER12/Desktop/url-results.txt
curl -o /dev/null --silent --head --write-out '%{http_code} %{time_total}' "$line" >> /Users/USER12/Desktop/url-results.txt
printf "\n" >> /Users/USER12/Desktop/url-results.txt
done <"$file"
which outputs in the following format
google.com 200 0.389
facebook.com 200 0.511
abnormallyLongDomain.com 200 0.786
but i would like to format into neat aligned columns for easy reading
DOMAIN_NAME HTTP_CODE RESPONSE_TIME
google.com 200 0.389
facebook.com 200 0.511
abnormallyLongDomain.com 200 0.486
Thanks for the help everyone!!
column is very nice. You are, however, already using printf which gives you fine control over the output format. Using printf's features also allows the code to be somewhat simplified:
#!/bin/bash
file="/Users/USER12/Desktop/url-list.txt"
log="/Users/USER12/Desktop/url-results.txt"
fmt="%-25s%-12s%-12s\n"
printf "$fmt" DOMAIN_NAME HTTP_CODE RESPONSE_TIME > "$log"
while read line
do
read code time < <(curl -o /dev/null --silent --head --write-out '%{http_code} %{time_total}' "$line")
printf "$fmt" "$line" "$code" "$time" >> "$log"
done <"$file"
With the above defined format, the output looks like:
DOMAIN_NAME HTTP_CODE RESPONSE_TIME
google.com 301 0.305
facebook.com 301 0.415
abnormallyLongDomain.com 000 0.000
You can fine-tune the output format, such as spacing or alignment, by changing the fmt variable in the script.
Further Refinements
The above code opens and closes the log file with each loop. This can be avoided as Charles Duffy suggests, simply by using exec to redirect stdout to the log file before the first printf statement:
#!/bin/bash
file="/Users/USER12/Desktop/url-list.txt"
exec >"/Users/USER12/Desktop/url-results.txt"
fmt="%-25s%-12s%-12s\n"
printf "$fmt" DOMAIN_NAME HTTP_CODE RESPONSE_TIME
while read line
do
read code time < <(curl -o /dev/null --silent --head --write-out '%{http_code} %{time_total}' "$line")
printf "$fmt" "$line" "$code" "$time"
done <"$file"
Alternatively, as Chepner suggests, the print statements can be grouped:
#!/bin/bash
file="/Users/USER12/Desktop/url-list.txt"
fmt="%-25s%-12s%-12s\n"
{
printf "$fmt" DOMAIN_NAME HTTP_CODE RESPONSE_TIME
while read line
do
read code time < <(curl -o /dev/null --silent --head --write-out '%{http_code} %{time_total}' "$line")
printf "$fmt" "$line" "$code" "$time"
done <"$file"
} >"/Users/USER12/Desktop/url-results.txt"
An advantage of grouping is that, after the group, stdout is automatically restored to its normal value.
Shortened a bit
#!/bin/bash
file="./url.txt"
fmt="%s\t%s\t%s\n"
( printf "$fmt" "DOMAIN_NAME" "HTTP_CODE" "RESPONSE_TIME"
while read -r line
do
printf "$fmt" "$line" $(curl -o /dev/null --silent --head --write-out '%{http_code} %{time_total}' "$line")
done <"$file" ) | column -t > ./out.txt
Don't need redirect every printf but you can enclose the part of your script into (...) and run it in an subshell a redirect it's output. Print every field separated with one tab and use the column command to format it nicely.
Anyway, usually is better don't put filenames (nor headers) into the script and reduce it to
#!/bin/bash
while read -r line
do
printf "%s\t%s\t%s\n" "$line" $(curl -o /dev/null --silent --head --write-out '%{http_code} %{time_total}' "$line")
done | column -t
and use it like:
myscript.sh < url-list.txt >result.txt
this allows you use your script in pipes, like:
something_produces_urls | myscript.sh | grep 200 > somewhere.txt
I have a simple BASH shell script which checks the HTTP response code of a curl command.
The logic is fine, but I am stuck on "simply" printing out the "output".
I am using GNU bash, version 3.2.25(1)-release (x86_64-redhat-linux-gnu)
I would like to output the URL with a tab - then the 404|200|501|502 response. For example:
http://www.google.co.uk<tab>200
I am also getting a strange error where the "http" part of a URL is being overwritten with the 200|404|501|502. Is there a basic BASH shell scripting (feature) which I am not using?
thanks
Miles.
#!/bin/bash
NAMES=`cat $1`
for i in $NAMES
do
URL=$i
statuscode=`curl -s -I -L $i |grep 'HTTP' | awk '{print $2}'`
case $statuscode in
200)
echo -ne $URL\t$statuscode;;
301)
echo -ne "\t $statuscode";;
302)
echo -ne "\t $statuscode";;
404)
echo -ne "\t $statuscode";;
esac
done
From this answer you can use the code
response=$(curl --write-out %{http_code} --silent --output /dev/null servername)
Substituted into your loop this would be
#!/bin/bash
NAMES=`cat $1`
for i in $NAMES
do
URL=$i
statuscode=$(curl --write-out %{http_code} --silent --output /dev/null $i)
case $statuscode in
200)
echo -e "$URL\t$statuscode" ;;
301)
echo -e "$URL\t$statuscode" ;;
302)
echo -e "$URL\t$statuscode" ;;
404)
echo -e "$URL\t$statuscode" ;;
* )
;;
esac
done
I've cleaned up the echo statements too so for each URL there is a new line.
try
200)
echo -ne "$URL\t$statuscode" ;;
I'm taking a stab here, but I think what's confusing you is the fact that curl is sometimes returning more than one header info (hence more than one status code) when the initial request gets redirected.
For example:
[me#hoe]$ curl -sIL www.google.com | awk '/HTTP/{print $2}'
302
200
When you're printing that in a loop, it would appear that the second status code has become part of the next URL.
If this is indeed your problem, then there are several ways to solve this depending on what you're trying to achieve.
If you don't want to follow redirections, simple leave out the -L option in curl
statuscode=$(curl -sI $i | awk '/HTTP/{print $2}')
To take only the last status code, simply pipe the whole command to tail -n1 to take only the last one.
statuscode=$(curl -sI $i | awk '/HTTP/{print $2}' | tail -n1)
To show all codes in the order, replace all linebreaks with spaces
statuscode=$(curl -sI $i | awk '/HTTP/{print $2}' | tr "\n" " ")
For example, using the 3rd scenario:
[me#home]$ cat script.sh
#!/bin/bash
for URL in www.stackoverflow.com stackoverflow.com stackoverflow.com/xxx
do
statuscode=$(curl -siL $i | awk '/^HTTP/{print $2}' | tr '\n' ' ')
echo -e "${URL}\t${statuscode}"
done
[me#home]$ ./script.sh
www.stackoverflow.com 301 200
stackoverflow.com 200
stackoverflow.com/xxx 404