i need check status of some link listen in a file called paths.txt, i have write a simple for loop but not work
for LINK in $(cat paths.txt)
do
CURL="$(curl -I $LINK 2>/dev/null | head -n 1 | cut -d$' ' -f2)"
if [ "${CURL}" = "200" ]
then
echo ${LINK} >> true.txt
else
echo ${LINK} >> false.txt
fi
done
If i launch the curl command from terminal i read the right status, instead if i make an echo of CURL variable i not have any output
Run the script like this: bash -x myscript and watch the executed commands.
don't use cat in bash scripts, because is bad practice and can be problems with big files.
SCRIPTDIR=`dirname -- "$0"`
for siteUrl in $( < "$SCRIPTDIR/paths.txt")
do
if [[ -z "$siteUrl" ]]; then break; fi
if [[ "$(curl --write-out "%{http_code}\n" --silent --output /dev/null ${siteUrl} )" = "200" ]]; then
echo ${siteUrl} >> true.txt
else
echo ${siteUrl} >> false.txt
fi
done
Related
I got a bash script wich reads urls from diffrent .txt files (ven Array defines the txt files) and I curl them and if the Response Code is not 200 it writes an error in error.txt
Sadly it always does it at the end, even if there is no error in any of the links, any idea why?
for i in "${ven[#]}"; do
while IFS='' read -r line || [[ -n "$line" ]]; do
IP=$(curl --write-out '%{http_code}' --silent --output /dev/null $line?upstream=$1)
if [ $IP != 200 ]
then
counter=$((counter + 1))
echo $(date +"%d.%m.%y %T") : $line >> error.txt
fi
done < $i
done
Your script has some quoting issues, although the main issue is the inconsistent use of test expression brackets (double vs. single) and how you are checking for the number.
if [[ "$IP" -ne 200 ]]
-ne means "not equal", and since you've already used double brackets stay consistent.
The other stuff is more preventative in the way you quote the variables:
for i in "${ven[#]}"; do
while IFS='' read -r line || [[ -n "$line" ]]; do
IP=$(curl --write-out '%{http_code}' --silent --output /dev/null "$line?upstream=$1")
if [[ "$IP" -ne 200 ]]; then
counter=$((counter + 1))
echo "$(date +'%d.%m.%y %T')" : "$line" >> error.txt
fi
done < "$i"
done
NOTE: If a site is redirecting ( 301 ) then it will show an error — something to maybe consider.
For some reason commands are not running after the first if statement.
"/usr/local//backup.sh: sleep: not found" for example..
The first if statement goes like:
if [ "$command" == "start -c" ]; then
echo -e "${ORANGE}Starting website backup...${E}"
sleep 1
DT=`date +%Y-%m-%d`
echo -e "${ORANGE}Started backup at $DT ${E}"
sleep 1
tar -zcvf web-$DT.tar.gz /usr/local/www/nginx
echo -e "${ORANGE}Compatced files, moving to backup directory${E}"
sleep 1
mv web-$DT.tar.gz $PATH/web-$DT.tar.gz
echo -e "${CYAN}Moved to backups ($PATH/web-$DT.tar.gz)\nDo you want to upload to ftp?${E}\n${CYAN} 'y' or 'n'?${E}"
read answer
if [ "$answer" == "y" ]; then
echo -e "${ORANGE}Uploading to ftp server...${E}"
sleep 1
if ftp -in -u ftp://$USER:$PASS#$HOST/Backups/f $PATH/web-$DT.tar.gz; then
echo -e "${GREEN}Uploaded! Bye!${E}"
else
echo -e "${ORANGE}Couldn't upload with ftp command, trying with curl...${E}"
sleep 1
curl -T "$PATH/web-$DT.tar.gz" -u $USER:$PASS ftp://$HOST/Backups/f/
echo -e "${GREEN}Uploaded! Bye!${E}"
fi
else
echo -e "${GREEN}Bye!${E}"
fi
Everything works fine inside this statement, however, after the elif compassion it just doesn't.
Here's the problematic code:
elif [ "$command" == "start -d" ]; then
echo -e "${ORANGE}Starting database backup...${E}"
sleep 1
DT=`date +%Y-%m-%d`
echo -e "${ORANGE}Started backup at $DT ${E}"
sleep 1
umask 177
echo -e "${ORANGE}Dumping database 'account'\n${E}"
mysqldump --user=$DBUSER --password=$DBPASS --host=$HOST account > $PATH/account-$DT.sql
sleep 1
echo -e "${ORANGE}Dumping database 'game'\n${E}"
mysqldump --user=$DBUSER --password=$DBPASS --host=$HOST game > $PATH/game-$DT.sql
sleep 1
echo -e "${ORANGE}Dumping database 'forum'\n${E}"
mysqldump --user=$DBUSER --password=$DBPASS --host=$HOST forum > $PATH/forum-$DT.sql
sleep 1
echo -e "${CYAN}Moved to backups ($PATH/<database-date>.sql)\nDo you want to upload to ftp?${E}\n${CYAN} 'y' or 'n'?${E}"
read answer
if [ "$answer" == "y" ]; then
echo -e "${ORANGE}Uploading to ftp server...${E}"
sleep 1
curl -T "$PATH/account-$DT.sql" -u $USER:$PASS ftp://$HOST/Backups/f/Databases/
curl -T "$PATH/game-$DT.sql" -u $USER:$PASS ftp://$HOST/Backups/f/Databases/
curl -T "$PATH/forum-$DT.sql" -u $USER:$PASS ftp://$HOST/Backups/f/Databases/
sleep 1
echo -e "${GREEN}Uploaded! Bye!${E}"
else
echo -e "${GREEN}Bye!${E}"
fi
and then I just ended it..
else
echo -e "${RED}Sorry! Not found${E}"
fi
As you can see there's spaces in the if statements and all of that, so what's wrong with this??
I've the #!/bin/bash up top as well, all the variables exist too.
You seem to have overwritten the value of the PATH variable at some point in your code. The shell needs that variable in order to know where to find programs like sleep (Hence the "not found" error). Name your path variable something else.
I have a script that runs flawlessly on many of the servers required. But recently it's failed on servers with old ksh versions.
Can you help me fix the offending line:
#!/bin/ksh
SUCCESS=0
FAILURE=0
while read IP
do
CURL=$(curl -s -m 2 -x http://$IP -L http://icanhazip.com)
if [[ "${IP%%:*}" == "${CURL%%:*}" ]] ; then
SUCCESS=$[SUCCESS+1]
echo "$IP ✓"
else
FAILURE=$[FAILURE+1]
echo "$IP X"
fi
done < <(curl -sL vpn-proxy-list.txt);
echo "✓: $SUCCESS X: $FAILURE"
The final line returns:
line 3: syntax error at line 14: `<(' unexpected
Unfortunately I'm unable to update ksh.
Can you help me make the done < <(curl -sL vpn-proxy-list.txt); portion simply work in bash? Or compatible with older versions (1993) of ksh?
You don't appear to be doing anything in the while body that cause trouble if it was run in a subshell, so I'd just stick with a plain pipline:
#!/bin/ksh
curl -sL vpn-proxy-list.txt | while read -r ip; do
output=$(curl -s -m 2 -x "http://$ip" -L http://icanhazip.com)
if [[ "${ip%%:*}" == "${output%%:*}" ]]; then
echo "$ip Y"
else
echo "$ip X"
fi
done
Now you're asking something that breaks because you're making variable changes in a subshell, and those variables disappear when the subshell exits. A workaround: use grouping braces
curl -sL vpn-proxy-list.txt | {
success=0
failure=0
while read -r ip; do
output=$(curl -s -m 2 -x "http://$ip" -L http://icanhazip.com)
if [[ "${ip%%:*}" == "${output%%:*}" ]]; then
echo "$ip Y"
let success+=1
else
echo "$ip X"
let failure+=1
fi
done
echo there were $success successes
echo there were $failure failures
}
# variables "success" and "failure" don't exist here.
You could make use of named pipes.
mkfifo foobar
curl -sL vpn-proxy-list.txt > foobar &
# Maybe sleep for a while here
while read -r IP; do
# do something here
done < foobar
rm foobar
An array holds the files accessed, and the archive files are split into smaller sizes in preparation for online backup. I am attempting to retrieve the exit code for each iteration through the loop of the split command. However, it is returning Exit Code 1, yet it says that the operation was successful. Why?
#!/bin/bash
declare -a SplitDirs
declare -a CFiles
CDIR=/mnt/Net_Pics/Working/Compressed/
SDIR=/mnt/Net_Pics/Working/Split/
Err=/mnt/Net_Pics/Working
SplitDirs=(`ls -l "$CDIR" --time-style="long-iso" | egrep '^d' | awk '{print $8}'`)
for dir in "${SplitDirs[#]}"
do
if [ ! -d "$SDIR""$dir" ]; then
mkdir "$SDIR""$dir"
else continue
fi
CFiles=(`ls -l "$CDIR$dir" --time-style="long-iso" | awk '{print $8}'`)
for f in "${CFiles[#]}"
do
if [ ! -e "$SDIR""$dir"/"$f" ]; then
split -d -a 4 -b 1992295 "$CDIR""$dir"/"$f" "$SDIR""$dir"/"$f" --verbose
if [[ "$?" == 1 ]]
then
rm -rf "$SDIR""$dir" && echo "$SDIR""$dir" "Removed due to Error code" "$?""." "Testing Archives and Retrying..." 2>&1 | tee "$Err"/Split_Err.log
7z t "$CDIR""$dir"/"$f" >> tee stdout.log 2>> "$Err"/"$dir"/7z_Err.log >&2
mkdir "$SDIR""$dir" && split -d -a 4 -b 1992295 "$CDIR""$dir"/"$f" "$SDIR""$dir"/"$f" --verbose
if [[ "$?" == 1 ]]
then
rm -rf "$SDIR""$dir" && echo "$SDIR""$dir" "Removed a second time due to Error code "$?". Skipping..." 2>&1 | tee "$Err"/Split_Err.log
continue
else
echo "Split Success:" "$SDIR""$dir"/"$f" "ended with Exit status" "$?" && continue
fi
else
echo "Split Success:" "$SDIR""$dir" "ended with Exit status" "$?" && continue
fi
else
echo "$SDIR""$dir"/"$f" "Exists... Skipping Operation" 2>&1 | tee "$Err"/"$dir"/Split_Err.log
continue
fi
done
(The echo piping in a previous revision of the question was misplaced code, and thank you for pointing that out. The exit code remains the same, though. Overall,the script does what I want it to except for the exit code portion.)
Remove | echo $?. you are processing the return code of echo command(last command).
I've got a little puzzler here for the bash scripting experts... I have a bash script that needs to create a small (80 byte) binary file when it runs. The contents of the file need to be contained inside the script itself (i.e. I don't want to just package the file along with the script).
My script currently does it like this:
echo 'begin-base64 644 dummy.wav' > /tmp/dummy.uu
echo 'UklGRkgAAABXQVZFZm10IBAAAAADAAEAAHcBAADcBQAEACAAZmFjdAQAAAAAAAAAUEVBSxAAAAAB' >> /tmp/dummy.uu
echo 'AAAAQDYlTAAAAAAAAAAAZGF0YQAAAAA=' >> /tmp/dummy.uu
echo '====' >> /tmp/dummy.uu
uudecode -o /tmp/dummy.wav /tmp/dummy.uu
rm /tmp/dummy.uu
... after the above runs, I have my file /tmp/dummy.wav. But I just found out that the computer this script is to run on does not have uudecode installed (and I'm not allowed to install it), so I need to find some other way to create this file. Any ideas?
Some installers do something similar to this:
#!/bin/bash
tail -n +4 $0 | tar xvzf -
exit
<tgz file appended here><newline>
If the target computer has perl available:
perl -ne 'print unpack("u",$_)' > dummy.wav <<EOD
M4DE&1D#```!7059%9FUT(!`````#``$``'<!``#<!0`$`"``9F%C=`0`````
C````4$5!2Q`````!````0#8E3```````````9&%T80``````
EOD
That's using the non-base64 format you get from just doing uuencode dummy.wav < dummy.wav on your original computer.
Failing that, you can always do this:
echo -ne '\x52\x49\x46\x46\x48\x00\x00\x00' > dummy.wav
echo -ne '\x57\x41\x56\x45\x66\x6d\x74\x20' >> dummy.wav
echo -ne '\x10\x00\x00\x00\x03\x00\x01\x00' >> dummy.wav
echo -ne '\x00\x77\x01\x00\x00\xdc\x05\x00' >> dummy.wav
echo -ne '\x04\x00\x20\x00\x66\x61\x63\x74' >> dummy.wav
echo -ne '\x04\x00\x00\x00\x00\x00\x00\x00' >> dummy.wav
echo -ne '\x50\x45\x41\x4b\x10\x00\x00\x00' >> dummy.wav
echo -ne '\x01\x00\x00\x00\x40\x36\x25\x4c' >> dummy.wav
echo -ne '\x00\x00\x00\x00\x00\x00\x00\x00' >> dummy.wav
echo -ne '\x64\x61\x74\x61\x00\x00\x00\x00' >> dummy.wav
This little bit of bash was generated with:
$ hexdump -e '"echo -ne '\''" 8/1 "x%02x" "'\'' >> dummy.wav\n"' dummy.wav | sed 's;x;\\x;g;1s/>/ /'
Edited to add:
As pointed out in a reply here, something like this is also a possibility:
xargs -d'\n' -n1 echo -ne > dummy.wav <<EOD
\x52\x49\x46\x46\x48\x00\x00\x00\x57\x41\x56\x45\x66\x6d\x74\x20
\x10\x00\x00\x00\x03\x00\x01\x00\x00\x77\x01\x00\x00\xdc\x05\x00
\x04\x00\x20\x00\x66\x61\x63\x74\x04\x00\x00\x00\x00\x00\x00\x00
\x50\x45\x41\x4b\x10\x00\x00\x00\x01\x00\x00\x00\x40\x36\x25\x4c
\x00\x00\x00\x00\x00\x00\x00\x00\x64\x61\x74\x61\x00\x00\x00\x00
EOD
(the -d argument is important to turn off xargs's own backslash processing)
You can also turn the 8/1 in my hexdump command into 80/1 and have a single long echo line.
Put an exit at the end of your script, append the file to the end of the script, and use tail -c 80 to get at the contents. This will work as long as you don't need to worry about newline conversion issues.
This is another example to decode radix 64 formatted data, it runs slow, but it is functional.
#!/bin/bash
exec<$0
while read line ; do if [ "$line" = "#payload" ] ; then break; fi; done
r64='ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'
i=0; while [ $i -lt 256 ] ; do tab[$i]=-1 ; let i=$i+1 ;done
i=0; while [ $i -lt 64 ] ; do tab[`printf "%d" "'${r64:$i:1}"`]=$i ; let i=$i+1; done
bi=0
while read -n 1 x
do
in=${tab[`printf "%d" "'$x"`]}
if [ $in -ge 0 ]; then case $bi in
0 ) out=$(($in<<2)); bi=6 ;;
2 ) out=$(($out|$in)); printf \\$(printf '%03o' $(($out&255)) ); bi=0 ;;
4 ) out=$(($out+($in>>2))); printf \\$(printf '%03o' $(($out&255)) );
bi=0; out=$(($in<<6)); bi=2 ;;
* ) out=$(($out+($in>>4))); printf \\$(printf '%03o' $(($out&255)) );
bi=0; out=$(($in<<4)); bi=4 ;;
esac fi
done
exit
#payload
dnZ2dnZ2dnZ2dnZ2dnZ2dnZ2XmAgICAte3Z2dnZ2dnZ2dnZ2dnZ2dgp2dnZ2dnZ2dnZ2dnZ2dnZ2PiAg
ICAgIC4gLXZ2dnZ2dnZ2dnZ2dnZ2CnZ2dnZ2dnZ2dnZ2dnZ2dn0gICAgICAgPT4gLXZ2dnZ2dnZ2dnZ2
dnYKdnZ2dnZJdnZJdnZJdnZJOyAgICAgICAtICAgPXZJdkl2dkl2dkl2dgp2dnZ2SXZ2dnZ2dnZ2dnZg
ICAgICAgICAgICAgbnZ2dnZJdnZ2dnZ2CnZ2dnZ2dnZ2SXZJdnZJdiAgIC4gICAgICwgICA8dnZ2SXZ2
dkl2dkkKdnZ2SXZ2SXZ2dnZ2SXZJIF9zOyAgX3VvLyAgID12dnZ2dnZ2dnZJdgp2dnZ2dkl2dnZJdnZ2
dnYgdyRtICBtQCRtICAgPXZ2dnZJdnZJdnZ2CnZ2dnZJdnZ2dnZ2dkl2SSBmPTQuO1cgYFE7ICA9dnZ2
dnZ2dnZ2dnYKdnZ2SXZ2dnZJdnZJdnZ2IHQtM3MlJiAgbWAgID12dnZ2SXZJdnZJdgp2dnZ2dnZ2SXZ2
dnZ2dnYgXWlvWjZYYXVQICAgPXZ2dnZ2dnZ2SXZ2CnZ2dkl2dkl2dnZJdnZJdi4pbVojWlojWlMgICAu
dnZ2SXZJdnZ2dnYKdnZ2dnZ2dnZ2dnZ2SXZ2OjNYWlpaI1pTWCAgICB7dnZ2dnZ2dkl2dgp2dnZ2SXZ2
SXZ2SXZ2dnY7PFNYWlhTZFhuIC5pLj12dnZJdnZJdnZ2CnZ2dkl2dnZ2dkl2dnZ2dmBdJVhYWlhubW0+
IC1gIHZ2dnZ2dnZ2dnYKdnZ2dnZ2SXZ2dnZ2SXYlIGptdklud1FXUW0gICAgPHZ2SXZ2SXZ2SQp2dnZJ
dnZ2dkl2dkl2dmAuUVFvWG1tUVFRUWMgICAge0l2dnZ2dnZ2CnZ2dnZ2dkl2dnZ2dnYrIGpRV1FtV1FR
UVFRayAgICAtdnZ2dkl2dkkKdnZ2dkl2dnZ2SXZJPiBfUVFRUVFRUVFRUVFRLiAgICA9dkl2dnZ2dgp2
dnZJdnZ2SXZ2dmwgIF1RUVFRV1FRUVdXOCRMICAgICA8dnZ2SXZ2CnZ2dnZ2dnZ2dnZ2OyAgbm1RUVFt
UVFRbXdvb20gLiAgIC1JdnZ2dnYKdnZ2SXZ2SXZ2SX0gID1RV1FRUVFRUVFRUVFtMlsgLSAgID12dkl2
dgp2dnZ2dnZ2dkl2Oy4gZFFRUVFRUVFRUVFRUVFRcSAgLiAgIEl2dnZ2CnZ2dnZJdkl2dnZgLjxRUVFR
UVFRUVFRUVFRUVdRKC4uLiAgPEl2dnYKdnZ2SXZ2dnZ2PiAgZFFRUVFRUVFRUVFRUVFRUVFbICAuICAg
dnZ2SQp2dnZ2dnZ2dnYnIC5RUVFRUVFRUVFRUVFRUVFRUWsgIC4gICB7dnZ2CnZ2dkl2dkl2PiAuXVFR
UVFRV1dXUVFRUVFRUVFRbSAgICAgIClsdnYKdnZ2dnZ2dnZgIDpqUVFRUVEjUVdRUVFRUVFRUVFXICAu
ICAgOnZ2SQp2dnZ2SXZ2bCAgOmpXUVFRUUVXV1FRUVFRV1FRUVcgIGAgICA6dnZ2CnZ2dkl2dnZJLl86
alFRUVFRRVdRUVFRUVFRUVFRVyAuIC4uID12dnYKdnZ2dnZ2dnZkIzYvUVdRUVFFUVFRUVFRUVFRV1dM
LiAgIDogKXZ2dgp2dnZJdnZJMyNaWkwtJFFRUVFRV1FRUVFRUVFCWiNgICAgLmRvdnZ2CnZ2dnZ2SXZa
IyMjWj4tNFFRUVdRUVFRUVFRUUVaay4gICBqWlh2dnYKdnZ2dndvbVgjWiNaIy4gNFFRUVFRUVFRUVdX
MVpYc189dVhaaHZ2dgp2dnZaWiNaI1VVWiNaTCAgXVFRUVFRUVFRUVdlWFpYcVhtWiNVenZ2CnZ2SVgj
I1ojWiMjWiNaLyAuUVFRUVFRUVFRVzEzI1paWlojWiMjb3YKdnZ2ZFVaIyNVWiMjVVVoX2FRUVFRUVFR
UVFQOlhaIyNVI1ojVVojaAp2dklkIyNaI1ojI1ojWlpaV1FRUVFRUVFXUCA9ZFojWiNaIyNaIyNaCnZ2
dlojWiMjVVVaI1ojWlpKUVFRUVFXUF4gIClYIyNaI1VVWiNVWjEKdnZ7WlojWlVaIyNaIyNaVXMtIT8i
fiAgICAgdlgjWiMjWiNaWF5sdgp2bCBZWFhYWFpaVVUjWlpaMS4gICAgICAgICB2WFojWiNaWCIgIDx2
CnZzICAtfiJJMVhYWFpaWm8xICAgICAgICAgIEluWFhaU31gICAgPHYKdnY7ICAgICAtLTwxMjIxbGAg
ICAgICAgICAgPElubjF9ICAgICB2SQp2dmwsICAgICAgICB+Kz5gICAgICAgICAgICAgfnwrfiAgICAu
JUl2CnZ2dnZpLiAgICAgICAgICAgIF9pc2ksICAgICAgICAgICAgX3ZJdnYKdnZ2dnZ2c19fXy4uLi5f
XyV2dnZ2SXZpLCxfLiAuLl9fPnZ2dnZ2dgp2dnZ2SXZ2dm52dnZ2dnZudnZ2dnZ2dnZubnZ2dnZ2dnZ2
dnZ2dnZ2Cg==
#!/bin/bash
# Define usage help
usage () {
echo -e "USAGE:\n\t$0 <file to create> <dir to tar> <name of script or command to run>\n"
exit 0
}
# check commandline arguments
if [ "$1" = "-h" ]; then usage; fi
if [ -z $1 ]; then usage; fi
if [ -z $2 ]; then usage; fi
if [ -z $3 ]; then usage; fi
# test for the directory and if it exists, create the bin file using tar
if [ -d "$2" ]; then
cat >$1<<EOF
#!/bin/sh -e
sed -e '1,/^exit$/d' "\$0" | tar xzf - && "./$2/$3"
exit
EOF
tar czf - $2 >> $1
else
echo "$2 does not exist, aborting!"
exit 1
fi
# make the new file executable and exit
chmod +x $1
exit 0
I would use base64 encoding, as that seems to be the general replacement for uu encoding, and operates on very similar principles.
From my point of view uuencode and uudecode are essential, but
that's only my opinion.
Without creating temporary files you could also do something
like this (uudecode.sh):
#!/bin/bash
# --
# -- Uudecoding without using a regular temporary file
# --
# -- Create a named pipe:
mknod /tmp/.dummypipe p
# -- Starting uudecoding on that pipe in the background:
uudecode -o dummy.txt /tmp/.dummypipe &
# -- Push base64-uuencoded content into the named pipe:
cat <<END_DUMMY > /tmp/.dummypipe
begin-base64 644 dummy.txt
VGhpcyBpcyB0aGUgdXVkZWNvZGVkIHRleHQuCg==
====
END_DUMMY
# -- Remove the named pipe
rm /tmp/.dummypipe
Just encode the binary data in base64 and do something like this:
#! /bin/bash
cat <<EOF | base64 -d > wherever-it-goes.bin
UtEOtUaZcUCrEJtPJrA34BH8Wdpxb1/DtfMo5syiE/h+moNyApEkg2ZwA2/jTDdfl4WijnNbMzvZ
RrF3i7X353AjmTjLBz1NcOOJJhRPYLJ4WQYONyYj/fAhnXQd+s4SHaNponOWKj1AAzdlJY1VLWaX
P8QBJQcn2FTL4pJ3N04=
EOF