I'm running in to a situation where the curl seeks that I do to a server to fetch some information intermittently breaks and instead of populating the resultant file with actual information it populates it with error it got and then move to next datapoint(in for loop), so I decided to catch and retry.(Using bash). But for some reason it's not doing what it should be, the generated file turns out to be empty. Below is what I have written, if you have a better/easier way to approach this or see a problem in below please go ahead and share. Thanks in advance.
for i in `cat master_db_list`
do
curl -sN --negotiate -u foo:bar "${URL}/$i/table" > $table_info/${i}
#Seek error catch
result=`cat $table_info/${i} | grep -i "Error"`
#lowercase comparison, since error can be in any case
echo -e "result is: $result \n" >> $job_log 2>&1
while [[ "${result,,}" =~ error ]]
do
echo "Retrying table list generation for ${i}" >> $job_log 2>&1
curl -sN --negotiate -u foo:bar "${URL}/$i/table" > $table_info/${i}
result=`cat $table_info/${i} | grep -i "Error"`
done
done
turns out , nothing wrong with above, some of the curls calls returned empty values, decreased seek frequency with sleeps in between and is working all ok now.
Related
What I want to do: I want to find all the products(URLs) which are not redirected.
To get the final URL after redirection I'm using curl command as follows:
curl -Ls -o /dev/null -w %{url_effective} "$URL"
This is working fine. Now I want to iterate over URLs to get which are the URLs that are not redirected and display them as output of program. I've the following code:
result=""
productIds=$1
for productId in $(echo $productIds | sed "s/,/ /g")
do
echo "Checking product: $productId";
URL="http://localhost/?go=$productId";
updatedURL=`curl -Ls -o /dev/null -w %{url_effective} "$URL"`
echo "URL : $URL, UpdatedUrl: $updatedURL";
if [ "$URL" == "$updatedURL" ]
then
result="$result$productId,";
fi
done
The curl command works only for the first product. But from 2nd to last product, all the URL and updatedURL are same. I can't understand the reason why? The productId is changing in every iteration, so I think it cannot be something related to caching.
Also, I've tried the following variant of curl also:
updatedURL=$(curl -Ls -o /dev/null -w %{url_effective} "$URL")
updatedURL="$(curl -Ls -o /dev/null -w %{url_effective} "$URL")"
Edit: After trying with debug mode and lot of different ways. I noticed a pattern i.e. If I manually hit the following on terminal:
curl -Ls -o /dev/null -w %{url_effective} "http://localhost/?go=32123"
Then in shell script these urls will work fine. But if I don't hit manually then curl will also not work for those products via shell script.
Just add #!/bin/bash to be the first line of shell. It does produce required output. Now invocation should be like this bash file.sh 123,456,789,221
Invocation via Bourne shell sh file.sh 123,456,789,221 does requires code changes.Do let me know if you require that too :)
I would suggest changing your loop to something like this:
IFS=, read -ra productIds <<<"$1"
for productId in "${productIds[#]}"; do
url=http://localhost/?go=$productId
num_redirects=$(curl -Ls -o /dev/null -w %{num_redirects} "$url")
if [ "$num_redirects" -eq 0 ]; then
printf 'productId %s has no redirects\n' "$productId"
fi
done
This splits the first argument passed to the script into an array, using a comma as the delimiter. The number of redirects is stored to a variable. When the number is zero, the message is printed.
I have to admit that I can't see anything inherently broken with your original approach so it's possible that there is something extra going on that we're not aware of. If you could provide a reproducible test case then we would be able to help you more effectively.
Im working on a bash script to check an ftp site to see if some files exist on it. It works by looping through an array of files using curl to check if the file exists on the ftp server. If it doesnt then curl should output an error which im looking for in my if loop and echoing a result based on it. Code below:
#!/bin/bash
# script variables
ftpaddress=ftpaddress
ftplocation=folderlocation
ftpusername=ftpusername
ftppassword=ftppassword
err="curl: (19) Given file does not exist"
# Initialise array of files
files=( "file1.xml" "file2.xml" "zy.xml" )
for f in "${files[#]}"; do
result=(curl ftp://$ftpaddress$ftplocation$f --user $ftpusername:$ftppassword --head)
if [[ "$result" == "$err" ]]; then
echo $f " Does not exist!!!!"
else
echo $f "Exists"
fi
done
Something about the if loop is failing, ive tried lots of variations on it but have been unable to find one that works. At the moment the result is never matching the error. When I run the curl just from the command line it outputs the error i have set to $err if the file doesnt exist but when running the script the else branch is being selected everytime saying the file does exist. I've even tried setting the error to be*"19"*and it still doesnt match. I've spent a lot of time looking it up and testing it but have had no luck so would appreciate any help that can be given.
Thanks
( ) will get you the return value $( ) will get you stdout
result=$(curl ftp://$ftpaddress$ftplocation$f --user $ftpusername:$ftppassword --head 2>&1)
I'm looking to get a simple listing of all the objects in a public S3 bucket.
I'm aware how to get a listing with curl for upto 1000 results, though I do not understand how to paginate the results, in order to get a full listing. I think marker is a clue.
I do not want to use a SDK / library or authenticate. I'm looking for a couple of lines of shell to do this.
#!/bin/sh
# setting max-keys higher than 1000 is not effective
s3url=http://mr2011.s3-ap-southeast-1.amazonaws.com?max-keys=1000
s3ns=http://s3.amazonaws.com/doc/2006-03-01/
i=0
s3get=$s3url
while :; do
curl -s $s3get > "listing$i.xml"
nextkey=$(xml sel -T -N "w=$s3ns" -t \
--if '/w:ListBucketResult/w:IsTruncated="true"' \
-v 'str:encode-uri(/w:ListBucketResult/w:Contents[last()]/w:Key, true())' \
-b -n "listing$i.xml")
# -b -n adds a newline to the result unconditionally,
# this avoids the "no XPaths matched" message; $() drops newlines.
if [ -n "$nextkey" ] ; then
s3get=$s3url"&marker=$nextkey"
i=$((i+1))
else
break
fi
done
I need to use 'last' to search through a list of users who logged into a system, i.e.
last -f /var/log/wtmp <username>
Considering the number of bzipped archive files in that directory, and considering I am on a shared system, I am trying to include an inline bzcat, but nothing seems to work. I have tried the following combinations with no success:
last -f <"$(bzcat /var/log/wtmp-*)"
last -f <$(bzcat /var/log/wtmp-*)
bzcat /var/log/wtmp-* | last -f -
Driving me bonkers. Any input would be great!
last (assuming the Linux version) can't read from a pipe. You'll need to temporarily bunzip2 the files to read them.
tempfile=`mktemp` || exit 1
for wtmp in /var/log/wtmp-*; do
bzcat "$wtmp" > "$tempfile"
last -f "$tempfile"
done
rm -f "$tempfile"
You can only use < I/O redirection on one file at a time.
If anything is going to work, then the last line of your examples is it, but does last recognize - as meaning standard input? (Comments in another answer indicate "No, last does not recognize -". Now you see why it is important to follow all the conventions - it makes life difficult when you don't.) Failing that, you'll have to do it the classic way with a shell loop.
for file in /var/log/wtmp-*
do
last -f <(bzcat "$file")
done
Well, using process substitution like that is pure Bash...the classic way would be more like:
tmp=/tmp/xx.$$ # Or use mktemp
trap "rm -f $tmp; exit 1" 0 1 2 3 13 15
for file in /var/log/wtmp-*
do
bzcat $file > $tmp
last -f $tmp
done
rm -f $tmp
trap 0
I want to create a shellscript that reads files from a .diz file, where information about various source files are stored, that are needed to compile a certain piece of software (imagemagick in this case). i am using Mac OSX Leopard 10.5 for this examples.
Basically i want to have an easy way to maintain these .diz files that hold the information for up-to-date source packages. i would just need to update these .diz files with urls, version information and file checksums.
Example line:
libpng:1.2.42:libpng-1.2.42.tar.bz2?use_mirror=biznetnetworks:http://downloads.sourceforge.net/project/libpng/00-libpng-stable/1.2.42/libpng-1.2.42.tar.bz2?use_mirror=biznetnetworks:9a5cbe9798927fdf528f3186a8840ebe
script part:
while IFS=: read app version file url md5
do
echo "Downloading $app Version: $version"
curl -L -v -O $url 2>> logfile.txt
$calculated_md5=`/sbin/md5 $file | /usr/bin/cut -f 2 -d "="`
echo $calculated_md5
done < "files.diz"
Actually I have more than just one question concerning this.
how to calculate and compare the checksums the best? i wanted to store md5 checksums in the .diz file and compare it with string comparison with "cut"ting out the string
is there a way to tell curl another filename to save to? (in my case the filename gets ugly libpng-1.2.42.tar.bz2?use_mirror=biznetnetworks)
i seem to have issues with the backticks that should direct the output of the piped md5 and cut into the variable $calculated_md5. is the syntax wrong?
Thanks!
The following is a practical one-liner:
curl -s -L <url> | tee <destination-file> |
sha256sum -c <(echo "a748a107dd0c6146e7f8a40f9d0fde29e19b3e8234d2de7e522a1fea15048e70 -") ||
rm -f <destination-file>
wrapping it up in a function taking 3 arguments:
- the url
- the destination
- the sha256
download() {
curl -s -L $1 | tee $2 | sha256sum -c <(echo "$3 -") || rm -f $2
}
while IFS=: read app version file url md5
do
echo "Downloading $app Version: $version"
#use -o for output file. define $outputfile yourself
curl -L -v $url -o $outputfile 2>> logfile.txt
# use $(..) instead of backticks.
calculated_md5=$(/sbin/md5 "$file" | /usr/bin/cut -f 2 -d "=")
# compare md5
case "$calculated_md5" in
"$md5" )
echo "md5 ok"
echo "do something else here";;
esac
done < "files.diz"
My curl has a -o (--output) option to specify an output file. There's also a problem with your assignment to $calculated_md5. It shouldn't have the dollar sign at the front when you assign to it. I don't have /sbin/md5 here so I can't comment on that. What I do have is md5sum. If you have it too, you might consider it as an alternative. In particular, it has a --check option that works from a file listing of md5sums that might be handy for your situation. HTH.