It seems whenever I test this on the shell it works but when I'm running it from a script it won't.
Shell:
fips=55555
echo https://www2.census.gov/geo/tiger/TIGER2016/ROADS/tl_2016_$fips'_'roads.zip
>>https://www2.census.gov/geo/tiger/TIGER2016/ROADS/tl_2016_55555_roads.zip
I'm trying to loop through some data in a text file like this:
And parse it/input into a curl url:
But the output has the beginning of the string overwritten like this:
./readFipsData.txt
>>_roads.zipw2.census.gov/geo/tiger/TIGER2016/ROADS/tl_2016_48001
Note: I am using Git Bash for Windows if that makes any difference.
cat ./testFipsData.txt |
while read line
do
counter=0
for col in $line
do
if [ $counter == 0 ]
then
state=$col
elif [ $counter == 1 ]
then
county=$col
elif [ $counter == 2 ]
then
fips=$col
fi
((counter++))
done
#curl -k -o ./$county.zip "https://www2.census.gov/geo/tiger/TIGER2016/ROADS/tl_2016_$fips_roads.zip"
echo https://www2.census.gov/geo/tiger/TIGER2016/ROADS/tl_2016_$fips'_'roads.zip
done
Thanks to all of your suggestions here's what worked:
cat ./testFipsData.txt |
while read state county fips
do
fips=${fips%$'\r'}
curl -k -o ./$county.zip https://www2.census.gov/geo/tiger/TIGER2016/ROADS/tl_2016_$fips'_'roads.zip
done
Related
So I have a shell script that contains a big if/fi block, and it was working fine until I decided to place an else case for this big if/fi block. Now I am getting this error:
/root/VVPN/Scripts/scriptPrincipal.sh: line 201: syntax error near unexpected token `else'
/root/VVPN/Scripts/scriptPrincipal.sh: line 201: `else'
I went through 8~10 stackoverflow posts where people had exactly the same error, except that all of them were simple syntax errors like a missing space after the [ of the if statement, or a : instead of a ; before the then keyword, or an else intended for an if that was already closed with a fi, etc... (you get the idea :p).
However I've checked my code for all these errors over and over and everything seems to be correct when it comes to if/else/fi syntax. I even showed the code to some colleagues and they too couldn't find the reason for this error.
Here's the code:
if [ ${CP} != "continue" ]
then
echo 'Downloading the necessary files from the server...'
# If folder F*** doesn't exist in /root/VVPN/Numbers
if [ ! -d ${N} ]
then
# Create folders F***, and F***/Results
mkdir ${N}
mkdir ${N}/Results
cd ${N}
# Get real number to factorize from server (e.g: wget server.com:8000/route/to/F1067/F1067)
ROUTE="VVPN/Numbers/${N}/${N}"
REAL_NUM_FILENAME=${N}
while true
do
wget --retry-connrefused --tries=inf -q ${SERVER}/${ROUTE} -O ${REAL_NUM_FILENAME} --continue
if [ $? -eq 0 ]; then
break
fi
sleep 1
done
# Get ECM-Program tuned for this number from server (e.g: wget server.com:8000/route/to/F1067/ecm)
ROUTE="VVPN/Numbers/${N}/ecm"
PROGRAM_NAME='ecm'
while true
do
wget --retry-connrefused --tries=inf -q ${SERVER}/${ROUTE} -O ${PROGRAM_NAME} --continue
if [ $? = 0 ]; then
break
fi
sleep 1
done
else
# The folder already exists, now we have to check whether the number has ",c" label or not
cd ${N}
fi
# Give the permission to execute program
chmod +x ecm
# Make 6 directories, one for each SPE
for i in {1..6}
do
mkdir "SPE${i}"
cp ecm "SPE${i}/"
done
# if there is no checkpoints:
if [ $CP = $N ]
then
# Get currentSigma for this number from server (e.g : wget server.com:8000/rout/to/F1067/currentSigma.txt)
ROUTE="VVPN/Numbers/${N}/currentSigma"
REAL_FILE_NAME='sigma'
while true
do
wget --retry-connrefused --tries=inf -q ${SERVER}/${ROUTE} -O ${REAL_FILE_NAME} --continue
if [ $? = 0 ]; then
break
fi
sleep 1
done
else
#The number has a ",c" label (= w/ checkpoint)
# Get i (server sigma) and C (current job counter) from the server (e.g: wget server.com:8000/route/to/F1067/icy)
ROUTE="icy?number=${N}"
SIGMA_C_Y='icy'
while true
do
wget --retry-connrefused --tries=inf -q ${SERVER}/${ROUTE} -O ${SIGMA_C_Y} --continue
if [ $? = 0 ]; then
break
fi
sleep 1
done
i=$(cat ${SIGMA_C_Y} | cut -d "," -f1)
C=$(cat ${SIGMA_C_Y} | cut -d "," -f2)
Y=$(cat ${SIGMA_C_Y} | cut -d "," -f3)
echo $i > sigma
echo $C > /root/VVPN/Scripts/C
# Get the checkpoints from the server
ROUTE="VVPN/Numbers/${N}/CP/${i},${C},${Y}"
for speNum in {1..6}
do
SPE="SPE${speNum}"
touch $SPE/again
for bigX in {0..3}
do
CHECKPOINT="pointsCurve${bigX}.${Y}"
while true
do
wget --retry-connrefused --tries=inf -q ${SERVER}/${ROUTE}/${SPE}/${CHECKPOINT} -O ${CHECKPOINT} --continue
if [ $? = 0 ]; then
mv $CHECKPOINT $SPE
break
fi
sleep 1
done
done
done
fi
cd ..
else
echo "Found ${N}'s folder on this PS3"
fi
So the else mentioned in the error (at line 201) is actually the last else in the code. The script works fine without this else and the echo that comes right after.
Any help would be much appreciated :)
Could you change if [ $CP = $N ] as below;
if [ "$CP" == "$N" ]
This is not if else problem; For example; if you run the following code, output is same. So You should focus other commands inside if statement.
CP="continue1"
if [ ${CP} != "continue" ]
then
while #this is wrong
echo ok
echo ok
echo ok
else
fi
./test.sh: line 8: syntax error near unexpected token `else'
./test.sh: line 8: `else'
Some weeks ago I found in this site a very useful bash script that downloads images from google image results (download images from google with command line)
Although the script is quite complicate for me, I did some simple modifications so as not to rename the results so as to keep the original names.
However, since the last week, the script stopped working... probably Google updated the code or something, and the regexes of the script don't parse the results any more. I don't know enough about google's codes, web programing or regexing to see what is wrong, although I did some educated guesses, but still didn't work.
My (unworking) tweaked script is this
#! /bin/bash
# function to create all dirs til file can be made
function mkdirs {
file="$1"
dir="/"
# convert to full path
if [ "${file##/*}" ]; then
file="${PWD}/${file}"
fi
# dir name of following dir
next="${file#/}"
# while not filename
while [ "${next//[^\/]/}" ]; do
# create dir if doesn't exist
[ -d "${dir}" ] || mkdir "${dir}"
dir="${dir}/${next%%/*}"
next="${next#*/}"
done
# last directory to make
[ -d "${dir}" ] || mkdir "${dir}"
}
# get optional 'o' flag, this will open the image after download
getopts 'o' option
[[ $option = 'o' ]] && shift
# parse arguments
count=${1}
shift
query="$#"
[ -z "$query" ] && exit 1 # insufficient arguments
# set user agent, customize this by visiting http://whatsmyuseragent.com/
useragent='Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:31.0) Gecko/20100101 Firefox/31.0'
# construct google link
link="www.google.cz/search?q=${query}\&tbm=isch"
# fetch link for download
imagelink=$(wget -e robots=off --user-agent "$useragent" -qO - "$link" | sed 's/</\n</g' | grep '<a href.*\(png\|jpg\|jpeg\)' | sed 's/.*imgurl=\([^&]*\)\&.*/\1/' | head -n $count | tail -n1)
imagelink="${imagelink%\%*}"
# get file extention (.png, .jpg, .jpeg)
ext=$(echo $imagelink | sed "s/.*\(\.[^\.]*\)$/\1/")
# set default save location and file name change this!!
dir="$PWD"
file="google image"
# get optional second argument, which defines the file name or dir
if [[ $# -eq 2 ]]; then
if [ -d "$2" ]; then
dir="$2"
else
file="${2}"
mkdirs "${dir}"
dir=""
fi
fi
# construct image link: add 'echo "${google_image}"'
# after this line for debug output
google_image="${dir}/${file}"
# construct name, append number if file exists
if [[ -e "${google_image}${ext}" ]] ; then
i=0
while [[ -e "${google_image}(${i})${ext}" ]] ; do
((i++))
done
google_image="${google_image}(${i})${ext}"
else
google_image="${google_image}${ext}"
fi
# get actual picture and store in google_image.$ext
wget --max-redirect 0 -q "${imagelink}"
# if 'o' flag supplied: open image
[[ $option = "o" ]] && gnome-open "${google_image}"
# successful execution, exit code 0
exit 0
one way to invetigate : provide -x option to bash so to have the trace of your script; that is change /bin/bash to /bin/bash -x in your script -or- simply invoke your script with
bash -x <yourscript>
You can also annotate your script with echo commands to track some variables.
I am trying to do the following in bash:
get my external IP
read first line of a file
compare both values
if it is not the same, delete the file and recreate it with the current address
I really don't know why this fails, all my script does is to output my current address and the first line of the file (which by the way is simply "asd" for testing)
#!/bin/bash
IP= curl http://ipecho.net/plain
OLD= head -n 1 /Users/emse/Downloads/IP/IP.txt
if [ "$IP" = "$OLD" ]; then
exit
else
rm /Users/emse/Downloads/IP/IP.txt
$IP> /Users/emse/Downloads/IP/IP.txt
exit
fi
Some obvious problems in your script:
Don't put spaces on either side of equal sign if you want to do assignment
You want the output of curl, head so wrap them in backticks (`)
You want to write $IP into the file, not to execute the content of it as a command, so echo it
The script becomes:
#!/bin/bash
IP=`curl http://ipecho.net/plain`
OLD=`head -n 1 /Users/emse/Downloads/IP/IP.txt`
if [ "$IP" = "$OLD" ]; then
exit
else
rm /Users/emse/Downloads/IP/IP.txt
echo $IP > /Users/emse/Downloads/IP/IP.txt
exit
fi
Excellent answer qingbo, just a tad bit of refinement:
#!/bin/bash
IP=`curl http://ipecho.net/plain`
OLD=`head -n 1 /Users/emse/Downloads/IP/IP.txt`
if [ "$IP" != "$OLD" ]; then
echo $IP > /Users/emse/Downloads/IP/IP.txt # > creates/truncates/replaces IP.txt
fi
So I have been struggling with this task for eternity and still don't get what went wrong. This program doesn't seem to download ANY pdfs. At the same time I checked the file that stores final links - everything stored correctly. The $PDFURL also checked, stores correct values. Any bash fans ready to help?
#!/bin/sh
#create a temporary directory where all the work will be conducted
TMPDIR=`mktemp -d /tmp/chiheisen.XXXXXXXXXX`
echo $TMPDIR
#no arguments given - error
if [ "$#" == "0" ]; then
exit 1
fi
# argument given, but wrong format
URL="$1"
#URL regex
URL_REG='(https?|ftp|file)://[-A-Za-z0-9\+&##/%?=~_|!:,.;]*[-A-Za-z0-9\+&##/%=~_|]'
if [[ ! $URL =~ $URL_REG ]]; then
exit 1
fi
# go to directory created
cd $TMPDIR
#download the html page
curl -s "$1" > htmlfile.html
#grep only links into temp.txt
cat htmlfile.html | grep -o -E 'href="([^"#]+)\.pdf"' | cut -d'"' -f2 > temp.txt
# iterate through lines in the file and try to download
# the pdf files that are there
cat temp.txt | while read PDFURL; do
#if this is an absolute URL, download the file directly
if [[ $PDFURL == *http* ]]
then
curl -s -f -O $PDFURL
err="$?"
if [ "$err" -ne 0 ]
then
echo ERROR "$(basename $PDFURL)">&2
else
echo "$(basename $PDFURL)"
fi
else
#update url - it is always relative to the first parameter in script
PDFURLU="$1""/""$(basename $PDFURL)"
curl -s -f -O $PDFURLU
err="$?"
if [ "$err" -ne 0 ]
then
echo ERROR "$(basename $PDFURLU)">&2
else
echo "$(basename $PDFURLU)"
fi
fi
done
#delete the files
rm htmlfile.html
rm temp.txt
P.S. Another minor problem I have just spotted. Maybe the problem is with the if in regex? I pretty much would like to see something like that there:
if [[ $PDFURL =~ (https?|ftp|file):// ]]
but this doesn't work. I don't have unwanted parentheses there, so why?
P.P.S. I also ran this script on URLs beginning with http, and the program gave the desired output. However, it still doesn't pass the test.
I have a problem, and I am pretty new to writing bash. I am parsing a csv file, checking for a few things. If a check is true, change the variable which will later be written to a file. I am reading an input file and outputting to a file as well, and If a certain argument checks True, then I want to prompt the user and pause the script until the user verifies the information matches (manual verification).
I have my most recent attempt which is not prompting. It just continues to read and write to the output. I am pretty sure because the output is going directly to my output file, but I do not know a way to direct the prompt to the terminal window which is where I am stuck.
INPUT=$TMPSAVE
IFS=,
[ ! -f $INPUT ] && { echo "$INPUT file not found"; exit 99; }
while read cdwon cdwod manu_date hp_sn manu_sn wiped_by wiped_date checked_by disposition readonly
do
for i in ${!allassigned[#]}
do
if [[ -n $manu_sn ]]
then
if echo ${allassigned[i]} | grep -q $manu_sn
then
physicaldrive=${allassigned[i-1]}
disk=$(hpacucli ctrl slot=${SLOT} show config detail | grep -B 4 ${physicaldrive} | head -1 | awk '{print $NF}');
if [[ -n $disk ]]; then #proceed to wipe drive
mount ${disk}${PRIMARY} ${MOUNT}
if [ -e $DIR ]; then
####### the file exists, now what to do with it? Automatcially prompt user?
cat $DIR > /dev/tty
echo "Does the drive serial number (${allassigned[i]}) match what was provided from the database ($manu_sn)? (y/n)" > /dev/tty
read
if [ "$REPLY" == "Y" ] || [ "$REPLY" == "y" ] || [ "$REPLY" == "YES" ] || [ "$REPLY" == "yes" ]; then
checked_by=$username
checked_bydate=`date`
fi
fi
fi
fi
fi
done
echo "$cdwon,$cdwod,$manu_date,$hp_sn,$manu_sn,$wiped_by,$wiped_date,$checked_by,$disposition,$readonly";
continue;
done < $INPUT > $OUTPUT
I solved my own issue here. I found out that read by default is reading from the stdin. When I was trying to prompt for input it was using stdin, so the lines I was reading were technically the stdin input. If you want to read a file in a while loop with the method I have done you have to change the fd like so:
while read -u 6 column1 column2
do
.....body
done 6< $INPUTFILE
Key being the "6" which could be any arbitrary number.