I'm trying to check if a list of urls are valid or not. I'm trying this Bash but I'm no getting the correct syntax.
for url in $(cat domains.txt)
do curl --output /dev/null --silent --head --fail "$url"; then
echo "URL exists: $url"
else
echo "URL does not exist: $url"
done
Can anyone give me a hand?
you're missing an if (and corresponding fi) to make your code work.
#!/bin/bash
for url in $(cat domains.txt)
do
if curl --output /dev/null --silent --head --fail "$url"; then
echo "URL exists: $url"
else
echo "URL does not exist: $url"
fi
done
You can also can make that the script check each url and if it's a redirection , give you the redirected url:
#!/bin/bash
for URL in $(cat urls.txt) ; do
res="$(curl --silent --head --location --output /dev/null --write-out '%{http_code}' $URL | grep '^2')"
if [[ $res == 200 ]]; then
curl -s -L -I -o /dev/null -w $URL' -> %{url_effective}\n' $URL;
else
echo "$URL -> "does not exist""
fi
done
`
Related
I am writing a script to save a website locally. Specifically, I am trying to download articles from a website while logged in. However, when I run the following command, it saves the website without logging in. cookies.txt has the proper information, probably "JSON=$(curl -b "cookies.txt" -c "cookies.txt" -s "${API}&page =$i")" seems to be the problem, does anyone know why this happens?
#!/bin/bash
curl -X POST -H "Content-Type: application/json" -c "cookies.txt" -d '{"login":"user_name","password":"password"}' https://note.com/api/v1/sessions/sign_in --cookie-jar cookies.txt
#info
NAME=somebody
API="https://note.com/api/v2/creators/${NAME}/contents?kind=note"
TOTAL_COUNT=$(curl -s $API | jq ".data.totalCount")
END=$(( $TOTAL_COUNT / 6 + 1 ))
echo "we will get ${NAME}'s articles"
for i in $(seq 1 $END); do
JSON=$(curl -b "cookies.txt" -c "cookies.txt" -s "${API}&page=$i")
IDS=($(echo $JSON | jq '.data.contents[].id'))
URLS=($(echo $JSON | jq -r '.data.contents[].noteUrl'))
IDS_COUNT=$(( ${#IDS[#]} - 1 ))
for j in $(seq 0 $IDS_COUNT); do
ID=${IDS[$j]}
URL=${URLS[$j]}
mkdir -p "note/${NAME}"
echo "ID: ${ID}"
monolith.exe $URL -s -o "note/${NAME}/${ID}.html"
done
done
echo "end"
I'm using the following script to purge cache from cdn,
#!/bin/bash
## API keys ##
zone_id=""
api_key=""
login_id=""
akamai_crd=""
## URL ##
urls="$1"
[ "$urls" == "" ] && { echo "Usage: $0 url"; exit 1; }
echo "Purging $urls..."
curl -X DELETE "https://api.cloudflare.com/client/v4/zones/${zone_id}/purge_cache" \
-H "X-Auth-Email: ${login_id}" \
-H "X-Auth-Key: ${api_key}" \
-H "Content-Type: application/json" \
--data "{\"files\":[\"${urls}\"]}"
#echo "CF is done now purging from Akamai ..."
echo "..."
curl -v -s https://api.ccu.akamai.com/ccu/v2/queues/default -H "Content-Type:application/json" -d '{"objects":["$urls"]}' -u $akamai_crd
The 1st part for cloudflare is working fine the 2nd part when I pass it to Akamai
["$urls"]
I keep getting an error and it's passing the url as an argument it returns the variable itself ($urls) not the arg value.
I ran the script as following:
sh +x script.sh url
Any advise here?
First i would change this:
SCRIPTNAME=$(basename "$0")
...
if [ $# != 1 ] then
echo "Usage: $SCRIPTNAME url"
exit
fi
$urls="$1"
Change your second curl command as the following (you need to escape the quotes):
--data "{\"files\":[\"${urls}\"]}"
Avoid creating JSON by hand like this; you can't guarantee that the resulting JSON is properly escaped. Use a tool like jq instead.
#!/bin/bash
## API keys ##
zone_id=""
api_key=""
login_id=""
akamai_crd=""
## URL ##
url=${1:?Usage: $0 url}
headers=(
-H "X-Auth-Email: $login_id"
-H "X-Auth-Key: $api_key"
-H "Content-Type: application/json"
)
purge_endpoint="https://api.cloudflare.com/client/v4/zones/${zone_id}/purge_cache"
echo "Purging $url..."
jq -n --arg url "$url" '{files: [$url]}' |
curl -X DELETE "$purge_endpoing" "${headers[#]}" --data #-
#echo "CF is done now purging from Akamai ..."
echo "..."
jq -n --arg url "$url" '{objects: [$url]}' |
curl -v -s -H "Content-Type:application/json" -d #- -u "$akamai_crd"
I'm trying to create a bash script that check url from list status code and echo server name from header. I'm actually new.
#!/bin/bash
while read LINE; do
curl -o /dev/null --silent --head --write-out '%{http_code}' "$LINE"
echo " $LINE" &
curl -I /dev/null --silent --head | grep -Fi Server "$SERVER"
echo " $SERVER"
done < dominios-https
I get the following output
301 http://example.com
grep: : No such file or directory
1) while read LINE can not use last line if text file not ended with new line.
2) You don't set "$SERVER" anywhere, and grep say it
3) Not all servers return "Server:" in headers
try it:
scriptDir=$( dirname -- "$0" )
for siteUrl in $( < "$scriptDir/myUrl.txt" )
do
if [[ -z "$siteUrl" ]]; then break; fi # break line if him empty
httpCode=$( curl -I -o /dev/null --silent --head --write-out '%{http_code}' "$siteUrl" )
echo "HTTP_CODE = $httpCode"
headServer=$( curl -I --silent --head "$siteUrl" | grep "Server" | awk '{print $2}' )
echo "Server header = $headServer"
done
Here is my shell script. I want to put a condition to my curl commands based on the status i get. I need help in grepping "Http/1.1 401 Unauthorized" from the first curl command. After that i need to put it in a condition if status is 401, execute 2nd and 3rd curl command. Pls help
STATUS="HTTP/1.1 401 Unauthorized"
read $STATUS
for ((i=1; i<=2000; i++)); do
curl -i -vs -X POST -D post.txt -H "$SESSION_TOKEN" -H "$AUTH_TOKEN" -H "Accept:$ACCEPT_HEADER" -H "Content-Type:text/plain" "http://$BASE_URI/api/$PLATFORM//$CHANNEL_ID/subscription" | grep -e "*Unauth*" >> post.txt
if [$STATUS]
then
curl -i -vs -X POST -D tmp.txt -H "Content-Type:text/plain" --data "$SECRET" -H "Accept:$ACCEPT_HEADER" -H "Connection:close" http://$BASE_URI/api/${PLATFORM}/authenticate >> tmp1.txt
SESSION_TOKEN=`grep SessionToken tmp.txt`
curl -i -vs -X POST -D tmp2.txt -H "$SESSION_TOKEN" -H "Content-Type:text/plain" -H "Content-Type:application/json" --data "{ \"username\":\"$USER_NAME\",\"password\":\"$PASSWORD\", \"rememberMe\":\"false\"}" http://$BASE_URI/api/web/users/authenticate >> data.json
AUTH_TOKEN=`grep Authorization tmp2.txt`
continue
fi
done
The proper solution :
res=$(curl -s -o /dev/null -w '%{http_code}\n' http://google.fr)
if ((res == 404)); then
echo "404 spotted"
fi
or
res=$(curl -s -o /dev/null -w '%{http_code}\n' http://google.fr)
if [[ $res == 404 ]]; then
echo "404 spotted"
fi
Check
man curl | less +/'^ *-w'
You have to capture the curl output and examine the status line:
output=$( curl -i ... )
if [[ $output == "$STATUS"* ]]; then
# I was unauthorized
else
# ...
fi
Be aware that [ (and [[ I use in my answer)
is not mere syntax, it is a command, and as such it requires a space between it and its arguments
I am writing a shell script to monitor website live or not and send an email alert below is my code
#!/bin/bash
if [[ curl -s --head --request GET http://opx.com/opx/version | grep "200 OK" > /dev/null] && [ curl -s --head --request GET http://oss.com/version | grep "200 OK" > /dev/null ]]
then echo "The HTTP server on opx.com and oss.com is up!" #> /dev/null
else
msg="The HTTP server opx.com Or oss.com is down "
email="opx-noc#opx.com"
curl --data "body=$msg &to=$email &subject=$msg" https://opx.com/email/send
fi;
if i run this code i got
./Monitoring_Opx_Oss: line 2: conditional binary operator expected
./Monitoring_Opx_Oss: line 2: syntax error near `-s'
./Monitoring_Opx_Oss: line 2: `if [[ curl -s --head --request GET http://opx.com/opx/version | grep "200 OK" > /dev/null] && [ curl -s --head --request GET http://oss.com/version | grep "200 OK" > /dev/null ]] '
Please correct me...
Change it do this:
if [ $(curl -s --head --request GET http://opx.opera.com/opx/version | grep "200 OK" > /dev/null) ] && [ $(curl -s --head --request GET http://oss.opera.com/version | grep "200 OK" > /dev/null) ]
To check the status of a command inside an if, you have to do it like
if [ $(command) ]
while you were using
if [ command]
note also the need of spaces around [ ]: if [_space_ command _space_ ]
Update
Based on Ansgar Wiechers's comment, you can also use the following:
if curl -s --head --request GET http://opx.com/opx/version | grep "200 OK" > /dev/null && curl -s --head --request GET http://oss.com/version | grep "200 OK" > /dev/null;
That is,
if command && command