I have a simple BASH shell script which checks the HTTP response code of a curl command.
The logic is fine, but I am stuck on "simply" printing out the "output".
I am using GNU bash, version 3.2.25(1)-release (x86_64-redhat-linux-gnu)
I would like to output the URL with a tab - then the 404|200|501|502 response. For example:
http://www.google.co.uk<tab>200
I am also getting a strange error where the "http" part of a URL is being overwritten with the 200|404|501|502. Is there a basic BASH shell scripting (feature) which I am not using?
thanks
Miles.
#!/bin/bash
NAMES=`cat $1`
for i in $NAMES
do
URL=$i
statuscode=`curl -s -I -L $i |grep 'HTTP' | awk '{print $2}'`
case $statuscode in
200)
echo -ne $URL\t$statuscode;;
301)
echo -ne "\t $statuscode";;
302)
echo -ne "\t $statuscode";;
404)
echo -ne "\t $statuscode";;
esac
done
From this answer you can use the code
response=$(curl --write-out %{http_code} --silent --output /dev/null servername)
Substituted into your loop this would be
#!/bin/bash
NAMES=`cat $1`
for i in $NAMES
do
URL=$i
statuscode=$(curl --write-out %{http_code} --silent --output /dev/null $i)
case $statuscode in
200)
echo -e "$URL\t$statuscode" ;;
301)
echo -e "$URL\t$statuscode" ;;
302)
echo -e "$URL\t$statuscode" ;;
404)
echo -e "$URL\t$statuscode" ;;
* )
;;
esac
done
I've cleaned up the echo statements too so for each URL there is a new line.
try
200)
echo -ne "$URL\t$statuscode" ;;
I'm taking a stab here, but I think what's confusing you is the fact that curl is sometimes returning more than one header info (hence more than one status code) when the initial request gets redirected.
For example:
[me#hoe]$ curl -sIL www.google.com | awk '/HTTP/{print $2}'
302
200
When you're printing that in a loop, it would appear that the second status code has become part of the next URL.
If this is indeed your problem, then there are several ways to solve this depending on what you're trying to achieve.
If you don't want to follow redirections, simple leave out the -L option in curl
statuscode=$(curl -sI $i | awk '/HTTP/{print $2}')
To take only the last status code, simply pipe the whole command to tail -n1 to take only the last one.
statuscode=$(curl -sI $i | awk '/HTTP/{print $2}' | tail -n1)
To show all codes in the order, replace all linebreaks with spaces
statuscode=$(curl -sI $i | awk '/HTTP/{print $2}' | tr "\n" " ")
For example, using the 3rd scenario:
[me#home]$ cat script.sh
#!/bin/bash
for URL in www.stackoverflow.com stackoverflow.com stackoverflow.com/xxx
do
statuscode=$(curl -siL $i | awk '/^HTTP/{print $2}' | tr '\n' ' ')
echo -e "${URL}\t${statuscode}"
done
[me#home]$ ./script.sh
www.stackoverflow.com 301 200
stackoverflow.com 200
stackoverflow.com/xxx 404
Related
I'm having problems with this last part of my bash script. It receives input from 500 web addresses and is supposed to fetch the server information from each. It works for a bit but then just stops at like the 45 element. Any thoughts with my loop at the end?
#initializing variables
timeout=5
headerFile="lab06.output"
dataFile="fortune500.tsv"
dataURL="http://www.tech.mtu.edu/~toarney/sat3310/lab09/"
dataPath="/home/pjvaglic/Documents/labs/lab06/data/"
curlOptions="--fail --connect-timeout $timeout"
#creating the array
declare -a myWebsitearray
#obtaining the data file
wget $dataURL$dataFile -O $dataPath$dataFile
#getting rid of the crap from dos
sed -n "s/^m//" $dataPath$dataFile
readarray -t myWebsitesarray < <(cut -f3 -d$'\t' $dataPath$dataFile)
myWebsitesarray=("${myWebsitesarray[#]:1}")
websitesCount=${#myWebsitesarray[*]}
echo "There are $websitesCount websites in $dataPath$dataFile"
#echo -e ${myWebsitesarray[200]}
#printing each line in the array
for line in ${myWebsitesarray[*]}
do
echo "$line"
done
#run each website URL and gather header information
for line in "${myWebsitearray[#]}"
do
((count++))
echo -e "\\rPlease wait... $count of $websitesCount"
curl --head "$curlOptions" "$line" | awk '/Server: / {print $2 }' >> $dataPath$headerFile
done
#display results
echo "Results: "
sort $dataPath$headerFile | uniq -c | sort -n
It would certainly help if you actually passed the --connect-timeout option to curl. As written, you are currently passing the single argument --fail --connect-timeout $timeout rather than 3 distinct arguments --fail, --connect-timeout, and $timeout. This is one instance where you should not quote the variable. IOW, use:
curl --head $curlOptions "$line"
When running an ldapsearch we get a return code indicating success or failure. This way we can use an if statement to check success.
On failure when using debug it prints if the cert validation failed. How can I capture the output of the command while checking the sucess or failure of ldapsearch?
ldapIP=`nslookup corpadssl.glb.intel.com | awk '/^Address: / { print $2 }' | cut -d' ' -f2`
server=`nslookup $ldapIP | awk -F"= " '/name/{print $2}'`
ldap='ldapsearch -x -d8 -H "ldaps://$ldapIP" -b "dc=corp,dc=xxxxx,dc=com" -D "name#am.corp.com" -w "366676" (mailNickname=sdent)"'
while true; do
if [[ $ldap ]] <-- capture text output here ??
then
:
else
echo $server $ldapIP `date` >> fail.txt
fi
sleep 5
done
As #codeforester suggested, you can use $? to check the return code of the last command.
ldapIP=`nslookup corpadssl.glb.intel.com | awk '/^Address: / { print $2 }' | cut -d' ' -f2`
server=`nslookup $ldapIP | awk -F"= " '/name/{print $2}'`
while true; do
captured=$(ldapsearch -x -d8 -H "ldaps://$ldapIP" -b "dc=corp,dc=xxxxx,dc=com" -D "name#am.corp.com" -w "366676" "(mailNickname=sdent)")
if [ $? -eq 0 ]
then
echo "${captured}"
else
echo "$server $ldapIP `date`" >> fail.txt
fi
sleep 5
done
EDIT: at #rici suggestion (and because I forgot to do it)... ldap needs to be run before the if.
EDIT2: at #Charles Duffy suggestion (we will get there), we don't need to store the command in a variable.
I want to check if my VPN is connected to a specific country. The VPN client has a status option but sometimes it doesn't return the correct country, so I wrote a script to check if I'm for instance connected to Sweden. My script looks like this:
#!/bin/bash
country=Sweden
service=expressvpn
while true; do
if ((curl -s https://www.iplocation.net/find-ip-address | grep $country | grep -v "grep" | wc -l) > 0 )
then
echo "$service connected!!!"
else
echo "$service not connected!"
$service connect $country
fi;
sleep 5;
done
The problem is, it always says "service connected", even when it isn't. When I enter the curl command manually, wc -l returns 0 if it didn't find Sweden and 1 when it does. What's wrong with the if statement?
Thank you
Peter
(( )) enters a math context -- anything inside it is interpreted as a mathematical expression. (You want your code to be interpreted as a math expression -- otherwise, > 0 would be creating a file named 0 and storing wc -l's output in that file, not comparing the output of wc -l to 0).
Since you aren't using )) on the closing side, this is presumably exactly what's happening: You're storing the output of wc -l in a file named 0, and then using its exit status (successful, since it didn't fail) to decide to follow the truthy branch of the if statement. [Just adding more parens on the closing side won't fix this, either, since curl -s ... isn't valid math syntax].
Now, if you want to go the math approach, what you can do is run a command substitution, which replaces the command with its output; that is a math expression:
# smallest possible change that works -- but don't do this; see other sections
if (( $(curl -s https://www.iplocation.net/find-ip-address | grep $country | grep -v "grep" | wc -l) > 0 )); then
...if your curl | grep | grep | wc becomes 5, then after the command substitution this looks like:
if (( 5 > 0 )); then
...and that does what you'd expect.
That said, this is silly. You want to know if your target country is in curl's output? Just check for that directly with shell builtins alone:
if [[ $(curl -s https://www.iplocation.net/find-ip-address) = *"$country"* ]]; then
echo "Found $country in output of curl" >&2
fi
...or, if you really want to use grep, use grep -q (which suppresses output), and check its exit status (which is zero, and thus truthy, if and only if it successfully found a match):
if curl -s https://www.iplocation.net/find-ip-address | grep -q -e "$country"; then
echo "Found $country in output of curl with grep" >&2
fi
This is more efficient in part because grep -q can stop as soon as it finds a match -- it doesn't need to keep reading more content -- so if your file is 16KB long and the country name is in the first 1KB of output, then grep can stop reading from curl (and curl can stop downloading) as soon as that first match 1KB in is seen.
The result of the curl -s https://www.iplocation.net/find-ip-address | grep $country | grep -v "grep" | wc -l statement is text. You compare text and number, that is why your if statement does not work.
This might solve your problem;
if [ $(curl -s https://www.iplocation.net/find-ip-address | grep $country | grep -v "grep" | wc -l) == "0" ] then ...
That worked, thank you for your help, this is what my script looks now:
#!/bin/bash
country=Switzerland
service=expressvpn
while true; do
if curl -s https://www.iplocation.net/find-ip-address | grep -q -e "$country"; then
echo "Found $country in output of curl with grep" >&2
echo "$service not connected!!!"
$service connect Russia
else
echo "$service connected!"
fi;
sleep 5;
done
This code ouputs a http status of 000 - which seems to indicate something didn't connect properly but when I do this curl outside of the bash script it works fine and produces a 200 so something with this code is off... any guidance?
#!/bin/bash
URLs=$(< test.txt | grep Url | awk -F\ ' { print $2 } ')
# printf "Preparing to check $URLs \n"
for line in $URLs
do curl -L -s -w "%{http_code} %{url_effective}\\n" $line
done
http://beerpla.net/2010/06/10/how-to-display-just-the-http-response-code-in-cli-curl/
your script works on my vt.
I added in a couple of debugging lines, this may help you to see where any metacharacters are getting in, as I would have to agree with the posted coments.
I've output lines in the for to a file which is then printed out with od.
I have amended the curl line to grab the last line, just to get the response code.
#!/bin/bash
echo -n > $HOME/Desktop/urltstfile # truncate urltstfile
URLs=$(cat testurl.txt | grep Url | awk -F\ ' { print $2 } ')
# printf "Preparing to check $URLs \n"
for line in $URLs
do echo $line >> $HOME/Desktop/urltstfile;
echo line:$line:
curl -IL -s -w "%{http_code}\n" $line | tail -1
done
od -c $HOME/Desktop/urltstfile
#do curl -L -s -w "%{http_code} %{url_effective}\\n" "$line\n"
I've written the bash script (searchuser) which should display all the users who are executing a specific program or a script (at least a bash script). But when searching for scripts fails because the command the SO is executing is something like bash scriptname.
This script acts parsing the ps command output, it search for all the occurrences of the specified program name, extracts the user and the program name, verifies if the program name is that we're searching for and if it's it displays the relevant information (in this case the user name and the program name, might be better to output also the PID, but that is quite simple). The verification is accomplished to reject all lines containing program names which contain the name of the program but they're not the program we are searching for; if we're searching gedit we don't desire to find sgedit or gedits.
Other issues I've are:
I would like to avoid the use of a tmp file.
I would like to be not tied to GNU extensions.
The script has to be executed as:
root# searchuser programname <invio>
The script searchuser is the following:
#!/bin/bash
i=0
search=$1
tmp=`mktemp`
ps -aux | tr -s ' ' | grep "$search" > $tmp
while read fileline
do
user=`echo "$fileline" | cut -f1 -d' '`
prg=`echo "$fileline" | cut -f11 -d' '`
prg=`basename "$prg"`
if [ "$prg" = "$search" ]; then
echo "$user - $prg"
i=`expr $i + 1`
fi
done < $tmp
if [ $i = 0 ]; then
echo "No users are executing $search"
fi
rm $tmp
exit $i
Have you suggestion about to solve these issues?
One approach might looks like such:
IFS=$'\n' read -r -d '' -a pids < <(pgrep -x -- "$1"; printf '\0')
if (( ! ${#pids[#]} )); then
echo "No users are executing $1"
fi
for pid in "${pids[#]}"; do
# build a more accurate command line than the one ps emits
args=( )
while IFS= read -r -d '' arg; do
args+=( "$arg" )
done </proc/"$pid"/cmdline
(( ${#args[#]} )) || continue # exited while we were running
printf -v cmdline_str '%q ' "${args[#]}"
user=$(stat --format=%U /proc/"$pid") || continue # exited while we were running
printf '%q - %s\n' "$user" "${cmdline_str% }"
done
Unlike the output from ps, which doesn't distinguish between ./command "some argument" and ./command "some" "argument", this will emit output which correctly shows the arguments run by each user, with quoting which will re-run the given command correctly.
What about:
ps -e -o user,comm | egrep "^[^ ]+ +$1$" | cut -d' ' -f1 | sort -u
* Addendum *
This statement:
ps -e -o user,pid,comm | egrep "^\s*\S+\s+\S+\s*$1$" | while read a b; do echo $a; done | sort | uniq -c
or this one:
ps -e -o user,pid,comm | egrep "^\s*\S+\s+\S+\s*sleep$" | xargs -L1 echo | cut -d ' ' -f1 | sort | uniq -c
shows the number of process instances by user.