Check website is online status Using JSON response [closed] - bash

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 4 years ago.
Improve this question
I want to check is my website online or not using BASH script.
Currently i am trying to check if curl response error then restart APACHE but its not working because i am using cloud flare. I am not a BASH script developer. I need help to make it possible to check JSON response from my URL if its true then OKAY otherwise restart Apache.
URL="http://apental-calc.com/online.php"
for i in {1..5};do
curl $URL >/dev/null 2>&1
if [[ $? -ne 0 ]];then
echo "Restarting Apache2"
service apache2 restart
sleep 5
continue
else
echo "Working"
exit 0
fi
done

I guess what you are trying to recover is a HTTP Code stating that the request was successful, in that case you can try this:
HTTP_RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" $URL)
This will print the HTTP status of the request done, for instance:
HTTP_RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" http://www.google.com)
echo $HTTP_RESPONSE
301
200 and 30X codes indicate a working server. If you don´t get result, then it is time to check CURL exit code:
if [ $? -eq 7 ] then ...
Being 7 the code returned when CURL can not connect to the host:
Failed to connect to host. curl managed to get an IP address to the machine and it tried to setup a TCP connection to the host but failed. This can be because you have specified the wrong port number, entered the wrong host name, the wrong protocol or perhaps because there is a firewall or another network equipment in between that blocks the traffic from getting through
See all codes:
https://ec.haxx.se/usingcurl-returns.html
UPDATE: As discussed in the comments, the question is not clear on whether the check is to be done in the HTTP Status code or in a JSON payload (which is missing). In case you want to check the payload, this can be used:
URL Serving a dummy JSON:
https://jsonplaceholder.typicode.com/todos/2
{
"userId": 1,
"id": 2,
"title": "quis ut nam facilis et officia qui",
"completed": false
}
To parse this, this can be used (let´s pretend we want to check the "title"):
$ curl -s https://jsonplaceholder.typicode.com/todos/2 | grep -Po '"title": ".*' | cut -f 4 -d '"'
This matches only the specified pattern and extracts the value.

Related

Bash script to test status of site

I have a script for testing the status of a site, to then run a command if it is offline. However, I've since realised because the site is proxied through Cloudflare, it always shows the 200 status, even if the site is offline. So I need to come up with another approach. I tried testing the site using curl and HEAD. Both get wrong response (from Cloudflare).
What I have found is that HTTPie command gets the response I need. Although only when I use the -h option (I have no idea why that makes a difference, since visually the output looks identical to when I don't use -h).
Assuming this is an okay way to go about reaching my aim ... I'd like to know how I can test if a certain string appears more than 0 times.
The string is location: https:/// (with three forward slashes).
The command I use to get the header info from the actual site (and not simply from what Cloudflare is dishing up) is, http -h https://example.com/.
I am able to test for the string using, http -h https://example.com | grep -c 'location: https:///'. This will output 1 when the string exists.
What I now want to do is run a command if the output is 1. But this is where I need help. My bash skills are minimal, and I am going about it the wrong way. What I came up with (which doesn't work) is:
#!/bin/bash
STR=$(http -h https://example.com/)
if (( $(grep -c 'location: https:///' $STR) != 1 )); then
echo "Site is UP"
exit
else
echo "Site is DOWN"
sudo wo clean --all && sudo wo stack reload --all
fi
Please explain to me why it's not working, and how to do this correctly.
Thank you.
ADDITIONS:
What the script is testing for is an odd situation in which the site suddenly starts redirecting to, literally, https:///. This obviously causes the site to be down. Safari, for instance, takes this as a redirection to localhost. Chrome simply spits the dummy with a redirect error, ERR_INVALID_REDIRECT.
When this is occurring, the headers from the site are:
HTTP/2 301
server: nginx
date: Thu, 12 May 2022 10:19:58 GMT
content-type: text/html; charset=UTF-8
content-length: 0
location: https:///
x-redirect-by: WordPress
x-powered-by: WordOps
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
referrer-policy: no-referrer, strict-origin-when-cross-origin
x-download-options: noopen
x-srcache-fetch-status: HIT
x-srcache-store-status: BYPASS
I choose to test for the string location: https:/// since that's the most specific (and unique) to this issue. Could also test for HTTP/2 301.
The intention of the script is to remedy the problem when it occurs, as a temporary solution whilst I figure out what's causing Wordpress to generate such an odd redirect. Also in case it happens whilst I am not at work, or sleeping. :-) I will have a cron job running the script every 5 mins, so at least the site is never down for longer than that.
grep reads a file, not a string. Also, you need to quote strings, especially if they might contain whitespace or shell metacharacters.
More tantentially, grep -q is the usual way to check if a string exists at least once. Perhaps see also Why is testing “$?” to see if a command succeeded or not, an anti-pattern?
I can see no reason to save the string in a variable which you only examine once; though if you want to (for debugging reasons etc) probably avoid upper case variables. See also Correct Bash and shell script variable capitalization
The parts which should happen unconditionally should be outside the condition, rather than repeated in both branches.
Nothing here is Bash-specific, so I changed the shebang to use sh instead, which is more portable and sometimes faster. Perhaps see also Difference between sh and bash
#!/bin/sh
if http -h https://example.com/ | grep -q 'location: https:///'
then
echo "Site is UP"
else
echo "Site is DOWN"
fi
sudo wo clean --all && sudo wo stack reload --all
For basic diagnostics, probably try http://shellcheck.net/ before asking for human assistance.

wget resolves to a different IP than host

I have a shell script in which I use host to get the IP of the target site to update ufw and allow outbound traffic to that IP. However, when I make the subsequent wget call to the same base URL, it resolves to a different IP, and thus is blocked by ufw. Just to test, I tried pinging the URL, and it returned a different third IP.
We're blocking all outbound traffic by default in ufw, and only enable what we need to go out, so I need the script to update the correct IP so I can wget the content. The IP in each instance (host vs wget) is consistently the same, but they return different values with respect to each other, so I don't think it's simply a DNS issue. How do I get a consistent IP to update the firewall with, so that the subsequent wget request performs successfully? I disabled the firewall as a test, and was able to download from the URL successfully, so the issue is definitely in getting a consistent IP to point to.
HOSTNAME=<name of site to resolve>
LOGFILE=<logfile path>
Current_IP=$(host $HOSTNAME | head -n 1 | cut -d " " -f 4)
#this echoes the correct value
echo $Current_IP
if [ ! -f $LOGFILE ]; then
/usr/sbin/ufw allow out from any to $Current_IP
echo $Current_IP > $LOGFILE
echo New IP address found and logged >> ./download.log
else
Old_IP=$(cat $LOGFILE)
if [ "$Current_IP" = "$Old_IP" ] ; then
echo IP address has not changed >> ./download.log
else
/usr/sbin/ufw delete allow out from any to $Old_IP
/usr/sbin/ufw allow out from any to $Current_IP
echo $Current_IP > $LOGFILE
echo IP Address was updated in ufw >> ./download.log
fi
fi
After that updates the firewall, a subsequent wget to HOSTNAME attempts to go out to a different IP than was just updated.
Turns out the difference was "www.". When I was resolving host I was not using www, and when I was using wget I was using www, and thus they resolved to different IPs for this particular site.

Bash How to kill wget process after a given timeout? [duplicate]

This question already has answers here:
Timeout command on Mac OS X?
(6 answers)
Closed 1 year ago.
UPDATE:
This question has been closed. I asked in another question.
Bash how to run multiple wget with output by sequence (NOT Parallel) and delete the wget file for speedtest purpose
I used timeout solution
#!/bin/bash
function speedtest() {
local key=$1
local url=$2
timeout 10 wget $url
echo -e "\033[40;32;1m$key is completed.\033[0m"
}
speedtest "Lisbon" "https://lg-lis.fdcservers.net/100MBtest.zip"
speedtest "London" "https://lg-lon.fdcservers.net/100MBtest.zip"
speedtest "Madrid" "https://lg-mad.fdcservers.net/100MBtest.zip"
speedtest "Paris" "https://lg-par2.fdcservers.net/100MBtest.zip"
When I run the bash script, this is the output.
» ./wget_speedtest.sh [2021/05/8 |15:42:49]
Redirecting output to ‘wget-log’.
Lisbon is completed.
Redirecting output to ‘wget-log.1’.
London is completed.
Redirecting output to ‘wget-log.2’.
Madrid is completed.
Redirecting output to ‘wget-log.3’.
Paris is completed.
I am expected to see the kb/s for running after 10 seconds for each wget.
I have a list of wget of file to download and want to see the download speed. I just want to run about 10 seconds, then print out the result what is the download speed.
I have 20 different server file to test out. My goal is to see the how kb/s download for that 10 seconds.
e.g
> wget https://lg-lis.fdcservers.net/100MBtest.zip
--2021-05-08 13:37:37-- https://lg-lis.fdcservers.net/100MBtest.zip
Resolving lg-lis.fdcservers.net (lg-lis.fdcservers.net)... 50.7.43.4
Connecting to lg-lis.fdcservers.net (lg-lis.fdcservers.net)|50.7.43.4|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [application/zip]
Saving to: ‘100MBtest.zip’
100MBtest.zip 0%[ ] 679.66K 174KB/s eta 9m 26s ^C
This is my bash file
#!/bin/bash
function speedtest() {
local key=$1
local url=$2
( cmdpid=$$;
(sleep 10; kill $cmdpid; rm -f 100M) \
& while ! wget "$url"
do
echo -e "\033[40;32;1m$key for 10 seconds done.\033[0m"
done )
}
speedtest "Lisbon" "https://lg-lis.fdcservers.net/100MBtest.zip"
speedtest "London" "https://lg-lon.fdcservers.net/100MBtest.zip"
speedtest "Madrid" "https://lg-mad.fdcservers.net/100MBtest.zip"
speedtest "Paris" "https://lg-par2.fdcservers.net/100MBtest.zip"
However, the above code does not work, it still download at background and redirect to wget-log
> ./wget_speedtest.sh
--2021-05-08 13:41:56-- https://lg-lis.fdcservers.net/100MBtest.zip
Resolving lg-lis.fdcservers.net (lg-lis.fdcservers.net)... 50.7.43.4
Connecting to lg-lis.fdcservers.net (lg-lis.fdcservers.net)|50.7.43.4|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [application/octet-stream]
Saving to: ‘100M’
100M 5%[==> ] 5.84M 1.02MB/s eta 79s [1] 21251 terminated ./wget_speedtest.sh
Redirecting output to ‘wget-log’.
Use the timeout command (part of GNU coreutils):
$ timeout 10 wget https://lg-lis.fdcservers.net
/100MBtest.zip
--2021-05-07 22:53:20-- https://lg-lis.fdcservers.net/100MBtest.zip
Resolving lg-lis.fdcservers.net (lg-lis.fdcservers.net)... 50.7.43.4
Connecting to lg-lis.fdcservers.net (lg-lis.fdcservers.net)|50.7.43.4|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [application/zip]
Saving to: ‘100MBtest.zip’
100MBtest.zip 0%[ ] 23.66K 10.8KB/s
$
If the specified time is exceeded, timeout exits with a status of 124.
timeout --help or info coreutils timeout for more information.
If you're on MacOS, see Timeout command on Mac OS X? for some suggested alternatives.

curl error 18 transfer closed with outstanding read data remaining

Setup
I'm Using curl in the following bash script to push a JSON file to a REST API running in tomcat sitting behind nginx.
while IFS= read -d '' -r file; do
base=$(basename "$file")
datetime=$(find $file -maxdepth 0 -printf "%TY/%Tm/%Td %TH:%TM:%.2TS")
curl -vX POST -H "Content-Type: application/json" -H "Cache-Control: no-cache" \
-d #"$file" -u vangeeij:eian12 \
"http://192.168.105.10/homeaccess/services/aCStats/uploadData?username=vangeeij&filename=$base&datetime=$datetime"
#sudo mv "$file" /home/vangeeij/acserver/resultsOld
done < <(sudo find . -type f -print0)
Problem
When running this script I get a http 400 response with curl error:
curl: (18) transfer closed with outstanding read data remaining
What I have tried
I have found 2 things. First running the same URL and body through Postman yields a successful POST.
I found that this error goes away when the last parameter is removed from the URL &datetime=$datetime
I have also found a few connections between this error and setting a curl option something like
curl_setopt($curl, CURLOPT_HTTPHEADER, array('Expect:'));
But I'm not sure where/how to set this exactly when using curl in a simple bash script
Question
What do I need to change in my curl command to get rid of the error and still be able to use all parameters?
UPDATE
Starting a new question, as further investigation has lead me to a better understanding of the problem.
New Question Link
The error has to do with the fact that the parameter datetime= ends up with text in it that needs to be URL encoded.
This was confirmed by replacing the variable with 2017%2F03%2F01%2008%3A50%3A56
and it worked.
So now the problem is, that I can't get --data-urlencode datetime=$datetime to work. It seems this just gets appended to the JSON data or something.
This error is being generated by the fact that the datetime= paramater is being passed in with non encoded non URL friendly characters... (eg. space).
The fix to this would be to find a way to convert the $datetime to a URLEncoded String.
eg. convert:
2017/03/01 08:50:56
TO
2017%2F03%2F01%2008%3A50%3A56
See the following discussion for one method to accomplish this.
Post JSON data to Rest with URLEncoded query paramaters

BASH- trouble pinging from text file lines

Have a text file w/ around 3 million URL's of sites I want to block.
Trying to ping them one by one (yes, I know it is going to take some time).
Have a script (yes, I am a bit slow in BASH) which reads the lines one at a time from text file.
Obviously cannot print text file here. Text file was created >> w/ Python some time ago.
Problem is that ping returns "unknown host" w/ every entry. If I make a smaller file by hand using the same entries the script works. I thought it may be a white space or end of line issue so tried addressing that in script. What could the issue possibly be?
#!/bin/bash
while read line
do
li=$(echo $line|tr -d '\n')
li2=$(echo $li|tr -d ' ')
if [ ${#line} -lt 2 ]
then
continue
fi
ping -c 2 -- $li2>>/dev/null
if [ $? -gt 0 ]
then
echo 'bad'
else
echo 'good'
fi
done<'temp_file.txt'
Does the file contains URLs or hostnames ?
If it contains URLs you must extract the hostname from URLs before pinging:
hostname=$(echo "$li2"|cut -d/ -f3);
ping -c 2 -- "$hostname"
Ping is used to ping hosts. If you have URLs of websites, then it will not work. Check that you have hosts in your file , example www.google.com or an IP address and not actual full website urls. If you want to check actual URLs, use a tool like wget and another tool like grep/awk to grab for errors like 404 or others. Last but not least, people who are security conscious will sometimes block pinging from the outside, so take note.
C heck if the file contains windows-style \r\n line endings: head file | od -c
If so, to fix it: dos2unix filename filename
I wouldn't use ping for this. It can easily be blocked, and it's not the best way to check for either ip addresses or if a server presents web pages.
If you just want to find the corresponding IP, use host:
$ host www.google.com
www.google.com is an alias for www.l.google.com.
www.l.google.com has address 209.85.149.106
www.l.google.com has address 209.85.149.147
www.l.google.com has address 209.85.149.99
www.l.google.com has address 209.85.149.103
www.l.google.com has address 209.85.149.104
www.l.google.com has address 209.85.149.105
As you see, you get all the IPs registered to a host. (Note that this requires you to parse the hostname from your urls!)
If you want to see if a URL points at a web server, use wget:
wget --spider $url
The --spider flag makes wget not save the page, just check that it exists. You could look at the return code, or add the -S flag (which prints the HTTP headers returned)

Resources