I have written a shell script to download and install the tomcat server v (8.5.31). wget http://www.us.apache.org/dist/tomcat/tomcat-8/v8.5.31/bin/apache-tomcat-8.5.31.tar.gz It was working fine, but as soon the version got changed to 9.0.10, it started giving error as 404 not found.
So what should I do to get the latest version always.
TL;DR
TOMCAT_VER=`curl --silent http://mirror.vorboss.net/apache/tomcat/tomcat-8/ | grep v8 | awk '{split($5,c,">v") ; split(c[2],d,"/") ; print d[1]}'`
wget -N http://mirror.vorboss.net/apache/tomcat/tomcat-8/v${TOMCAT_VER}/bin/apache-tomcat-${TOMCAT_VER}.tar.gz
I encountered the same challenge.
However for my solution I require the latest 8.5.x Tomcat version which keeps changing.
Since the URL to download Tomcat remains the same, with only the version changing, I found the following solution works for me:
TOMCAT_VER=`curl --silent http://mirror.vorboss.net/apache/tomcat/tomcat-8/ | grep v8 | awk '{split($5,c,">v") ; split(c[2],d,"/") ; print d[1]}'`
echo Tomcat version: $TOMCAT_VER
Tomcat version: 8.5.40
grep v8 - returns the line with the desired version:
<img src="/icons/folder.gif" alt="[DIR]"> v8.5.40/ 2019-04-12 13:16 -
awk '{split($5,c,">v") ; split(c[2],d,"/") ; print d[1]}' - Extracts the version we want:
8.5.40
I then proceed to download Tomcat using the extracted version:
wget -N http://mirror.vorboss.net/apache/tomcat/tomcat-8/v${TOMCAT_VER}/bin/apache-tomcat-${TOMCAT_VER}.tar.gz
This is the complete curl response from which the version is extracted using curl, grep and awk:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>
<head>
<title>Index of /apache/tomcat/tomcat-8</title>
</head>
<body>
<h1>Index of /apache/tomcat/tomcat-8</h1>
<pre><img src="/icons/blank.gif" alt="Icon "> Name Last modified Size Description<hr><img src="/icons/back.gif" alt="[PARENTDIR]"> Parent Directory -
<img src="/icons/folder.gif" alt="[DIR]"> v8.5.40/ 2019-04-12 13:16 -
<hr></pre>
<address>Apache/2.4.25 (Debian) Server at mirror.vorboss.net Port 80</address>
</body></html>
I've found a way using the official github mirror.
Basically, one has to query the github api for all available tags.
Afterwards, for each tag, the date has to be determined.
Finally, the tag with the latest date is the latest tag!
Try this script - let's call it latest-tag. It's dependent on jq. It takes a short while to execute, but should print the URL of the tarball of the lastest tag (currently: https://api.github.com/repos/apache/tomcat/tarball/TOMCAT_9_0_10)
#!/bin/bash
# Prints the url to the latest tag of given github repo
# $1: repo (e.g.: apache/tomcat )
# $2: optional github credentials. Credentials are needed if running into the api rate limit (e.g.: <user>|<user>:<authkey>)
repo=${1:?Missing parameter: repo (e.g.: apache/tomcat )}
[ -n "$2" ] && credentials="-u $2"
declare -a commits
declare -a tarball_urls
while IFS=, read commit_url tarball_url
do
date=$(curl $credentials --silent "$commit_url" | jq -r ".commit.author.date")
if [[ "$date" > ${latest_date:- } ]]
then
latest_date=$date
latest_tarball_url=$tarball_url
fi
done < <( curl $credentials --silent "https://api.github.com/repos/$repo/tags" | jq -r ".[] | [.commit.url, .tarball_url] | #csv" | tr -d \")
echo $latest_tarball_url
Usage:
./latest-tag apache/tomcat
You might get hindered by the rate limit of the github api.
Therefore, you might want to supply github credentials to the script:
./latest-tag apache/tomcat <username>
This will ask you for your github password. In order to run it interactively, you can supply the script with a personal github api token:
./latest-tag apache/tomcat <username>:<api token>
Disclaimer - this solution uses screen scraping
Find and download latest version of Apache Tomcat 9 for Linux or Windows-x64.
Uses Python 3.7.3
import os
import urllib.request
url_ends_with = ".tar.gz\"" # Use line for Non-Windows
url_ends_with = "windows-x64.zip\"" # Use line for Windows-x64
url_starts_with = "\"http"
dir_to_contain_download = "tmp/"
tomcat_apache_org_frontpage_html = "tomcat.apache.org.frontpage.html"
download_page = "https://tomcat.apache.org/download-90.cgi"
try:
if not os.path.exists(dir_to_contain_download):
os.makedirs(dir_to_contain_download, exist_ok=True)
htmlfile = urllib.request.urlretrieve(download_page, dir_to_contain_download + tomcat_apache_org_frontpage_html)
fp = open(dir_to_contain_download + tomcat_apache_org_frontpage_html)
line = fp.readline()
cnt = 1
while line:
line = fp.readline()
cnt += 1
if url_ends_with in line and url_starts_with in line:
tomcat_url_index = line.find(url_ends_with)
tomcat_url = line[line.find(url_starts_with) + 1 : tomcat_url_index + len(url_ends_with) - 1]
print ("Downloading: " + tomcat_url)
print ("To file: " + dir_to_contain_download + tomcat_url[tomcat_url.rfind("/")+1:])
zipfile = urllib.request.urlretrieve(tomcat_url, dir_to_contain_download + tomcat_url[tomcat_url.rfind("/")+1:])
break
finally:
fp.close()
os.remove(dir_to_contain_download + "/" + tomcat_apache_org_frontpage_html)
As I don't have enough reputation to answer to Jonathan or edit his post, here is my solution (tested with versions 8-10):
#!/bin/bash
wantedVer=9
TOMCAT_VER=`curl --silent http://mirror.vorboss.net/apache/tomcat/tomcat-${wantedVer}/|grep -oP "(?<=\"v)${wantedVer}(?:\.\d+){2}\b"|sort -V|tail -n 1`
wget -N https://mirror.vorboss.net/apache/tomcat/tomcat-${TOMCAT_VER%.*.*}/v${TOMCAT_VER}/bin/apache-tomcat-${TOMCAT_VER}.tar.gz
# Apache download link: wget -N https://dlcdn.apache.org/tomcat/tomcat-${TOMCAT_VER%.*.*}/v${TOMCAT_VER}/bin/apache-tomcat-${TOMCAT_VER}.tar.gz
I had trouble with Jonathans code, because there were different versions downloadable at the same time which broke the composed download-link. In this solution, only the newest one is regarded.
Altering the first line is enough to distinguish between different Main Versions.
Code Explained:
curl grabs an apache-directory-listing for the wanted tomcat-Version.
Grep then extracts all different Versions with a positive lookbehind, using the (-P) Perl regex pattern and keeping only the matching part (-o). Result: line(s) each containing one version Number in no particular order.
These lines get sorted by Version (sort -V) and only the last line (tail -n 1), in which is the greatest of all Versions is located, is assigned to the variable TOMCAT_VER.
At last, the download link is created with the gathered version information and downloaded via wget , but only if it is a newer version than present (-N).
I wrote this code:
TOMCAT_URL=$(curl -sS https://tomcat.apache.org/download-90.cgi | grep \
'>tar.gz</a>' | head -1 | grep -E -o 'https://[a-z0-9:./-]+.tar.gz')
TOMCAT_NAME=$(echo $TOMCAT_URL | grep -E -o 'apache-tomcat-[0-9.]+[0-9]')
It's not the most efficient way possible but it's very easy to understand how it works and it does work. Update the download-XX.cgi link to 10 if you want that version.
Then you can do:
curl -sS $TOMCAT_URL | tar xfz -
ln -s $TOMCAT_NAME apache-tomcat
and you will have the current version of Tomcat at apache-tomcat. When a new version comes out you can use this to do an easy update while keeping the old version there.
Related
I'm writing a simple script to notify me whenever a new firmware update is available in my Mikrotik router. The RouterOS command for this purpose is:
ssh myrouter system package update print
And when its run normally it works fine, showing something like this:
channel: stable
installed-version: 6.48
latest-version: 6.48.1
status: New version is available
The full command I am using to only retrieve the relevant info so a notification is triggered is:
ssh -q myrouter system package update print |tail -2 |head -1 |sed "s/: /\n/" |tail -1
Which results in this when it is run "as is" (not inside a script nor anything else):
New version is available
However, I cannot seem to be able to parse this text inside a simple if statement, no matter if I manipulate it directly or first redirecting it to a local file:
UPDATES_AVAILABLE=$(ssh myrouter system package update print |tail -2 |head -1 |cut -d: -f2)
if [[ "$(echo ${UPDATES_AVAILABLE})" == "New version is available" ]]; then
echo "INFO: New firmware version is available"
fi
This last one always returns no content, wether is run inside a script or directly in the shell.
What am I missing?
Oh nevermind, I just fixed it.
After redirecting the output to a /tmp/mikrotik.tmp file I just thought about running:
file /tmp/mikrotik.tmp
And then I saw that it returned this:
mikrotik: ASCII text, with CRLF line terminators
So as soon as I added | dos2unix inside my string filter command it just started working as expected, and now this:
UPDATES_AVAILABLE=$(ssh myrouter system package update print |dos2unix |tail -2 |head -1 |cut -d: -f2)
if [[ "$(echo ${UPDATES_AVAILABLE})" == "New version is available" ]]; then
echo "INFO: New firmware version is available"
fi
Is returning correctly:
INFO: New firmware version is available
I have the following requirement here: Fetch all the commits from our SVN from the last two years and list the title of all the JIRA issues that had code committed. Our commit rules are pretty strict, so a commit must start with the JIRA code, like: COR-3123 Fixed the bug, introduced a new one
So, I wrote the following shell script to get this working:
svn log -r{2012-04-01}:{2014-04-01} | grep "COR-" | cut -f1 -d" " | sort -u
This gets me all the JIRA codes.
But now I want to use these in the following command:
wget --quiet --load-cookies cookies.txt -O - http://jira.example.com/browse/{HERE} | sed -n -e 's!.*<title>\(.*\)</title>.*!\1!p'
Ie: get the JIRA page via wget and parse out the title... (I have already cached my login credentials to use with wget in cookies.txt)
and obviously to the location {HERE} I want to insert the code obtained from the first list. Doing this via a two step (step 1: get list, step 2 iterate via list) script (python, perl, ... ) is not a problem, but I'd like to know if it's possible to do it in ONE step, using bash :)
(Yes, I know there is JIRA rest API)
You can use xargs to pass the parameter to wget:
xargs -I {} wget http://jira.example.com/browse/{}
I am using the following cron task:
wget -O ~/files/csv_backups/products_csv/products-`date +%Y-%m-%d-%H%M%S`.csv http://[site_URL]/product-report/csv
to grab a daily CSV of product sales. It is currently placing it in a files directory on the server and names it with a date stamp.
What I am looking to do is for it to also send an email with either an attachment to the file or a link to the file in the body, to a specific email address. I can't seem to find a way to do this. Any thoughts?
UPDATE:
I've tried with the code suggested but it just outputted the trailing path to the file, not a proper download link. So I tried the following
echo "Download File" | mail -s "Daily Products CSV Report" username#mail.com
But I get the echo printed in plain text. Closer..
UPDATE:
1) OK... So my first problem is that the timestamp I am trying to rename the file to doesn't work and the download fails. So I removed the timestamp (even though I would like it so it could make backups...) for now. The following code is my new line I am using to download the file. It should constantly download a new copy regardless of server timestamp. This is because the file is not actually generated at all until you visit that URL so there is no file to compare times with before wget tries to download it.
wget --timestamping -r -O files/csv_backups/products_csv/products.csv http://www.example.com/reports/product-report/csv
2) For the email, I removed the html and just left the following because for some reason the echo was rendering the html in plaintext which doesn't work. Why ever that is happening may be also why I couldn't use the $(ls -1t * | head -1) line at the end of the URL to grad the most recent file...
echo "http://www.example.com/files/csv_backups/products_csv/products.csv" | mail...
If you don't have mutt installed then you can do something like:
echo "$(ls -1t ~/files/csv_backups/products_csv/* | head -1)" |
mail -s "subject line" email#address.com
This will give path to the file in subject body
The whole echo $(ls -1t ~/files/csv_backups/products_csv/* | head -1) is because you will not know what the file name is since you are adding timestamp not datestamp. The %H%M%S will add hour minutes and seconds to the filename as well.
With the above command we assume the latest file in that directory will be the one downloaded recently.
Alternatively you can use uuencode which is part of GNU sharutils.
uuencode $(ls -1t ~/files/csv_backups/products_csv/* | head -1) $(ls -1t ~/files/csv_backups/products_csv/* | head -1) |
mail -s "subject line" email#address.com
Heiner Steven wrote a good article about [writing scripts for sending files via email](http://www.shelldorado.com/articles/mailattachments.html. He also wrote sendfile which uses the metamail package (available at the link above).
The other approach is to use perl ;-) (man perlfaq9 covers e-mail) or to dust off of uuencode.
uuencode Sales.cvs Sales.cvs | mail -s "Kaching" listofmanagers#widgetsinc.com
NB: I think all versions of uuencode require that you give the name of the attachment twice: once for the file to encode; and once for the name of the file for extraction from the message. uuencode may also appear as b64encode on some BSD systems. You can Base64 encode with uuencode using the -m switch.
I would like to implement a version checking in a bash script, which would verify a file on my website, and then if the version of the script does not match the last version, it opens a web page. I heard I would use cURL, but didnt find any tuto about my case.
I'm just gonna write what I would do:
#configuration
VERSION=1.0.0
URL='http://example.com/version'
DL_URL='http://example.com/download'
#later on whenever you want
# Assume the url only displays the current version number
CHECK_VERSION=$(wget -O- "$URL")
# If you remove the periods, it works as a way to
# lexicorigraphically compare the version numbers as numbers
CURRENT_NUMBER=$(echo "$VERSION" | tr -d '.')
NEW_NUMBER=$(echo "$CHECK_VERSION" | tr -d '.')
test "$CURRENT_NUMBER" -gt "$NEW_NUMBER" || x-www-browser "$DL_URL"
I want to write a script to find the latest version of rpm of a given package available from a mirror for eg: http://mirror.centos.org/centos/5/updates/x86_64/RPMS/
The script should be able to run on majority of linux flavors (eg centos, redhat, ubuntu). So yum based solution is not an option. Is there any existing script that does this? Or can someone give me a general idea on how to go about this?
Thx to levislevis85 for the wget cli. Try this:
ARCH="i386"
PKG="pidgin-devel"
URL=http://mirror.centos.org/centos/5/updates/x86_64/RPMS
DL=`wget -O- -q $URL | sed -n 's/.*rpm.>\('$PKG'.*'$ARCH'.rpm\).*/\1/p' | sort | tail -1`
wget $URL/$DL
I Will put my comment here, otherwise the code will not be readable.
Try this:
ARCH="i386"
PKG="pidgin-devel"
URL=http://mirror.centos.org/centos/5/updates/x86_64/RPMS
DL=`wget -O- -q $URL | sed -n 's/.*rpm.>\('$PKG'.*'$ARCH'.rpm\).*<td align="right">\(.*\)-\(.*\)-\(.*\) \(..\):\(..\) <\/td><td.*/\4 \3 \2 \5 \6 \1/p' | sort -k1n -k2M -k3n -k4n -k5n | cut -d ' ' -f 6 | tail -1`
wget $URL/$DL
What it does is:
wget - get the index file
sed - cut out some parts and put it together in different order. Should result in Year Month Day Hour Minute and Package, like:
2009 Oct 27 01 14 pidgin-devel-2.6.2-2.el5.i386.rpm
2009 Oct 30 10 49 pidgin-devel-2.6.3-2.el5.i386.rpm
sort - order the columns n stays for numerical and M for month
cut - cut out the filed 6
tail - show only last entry
the problem with this could be, if some older package release comes after a newer then this script will also fail. If the output of the site changes, the script will fail. There are always a lot of points where a script could fail.
using wget and gawk
#!/bin/bash
pkg="kernel-headers"
wget -O- -q http://mirror.centos.org/centos/5/updates/x86_64/RPMS | awk -vpkg="$pkg" 'BEGIN{
RS="\n";FS="</a>"
z=split("Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec",D,"|")
for(i=1;i<=z;i++){
date[D[i]]=sprintf("%02d",i)
}
temp=0
}
$1~pkg{
p=$1
t=$2
gsub(/.*href=\042/,"",p)
gsub(/\042>.*/,"",p)
m=split(t,timestamp," ")
n=split(timestamp[1],d,"-")
q=split(timestamp[2],hm,":")
datetime=d[3]date[d[2]]d[1]hm[1]hm[2]
if ( datetime >= temp ){
temp=datetime
filepkg = p
}
}
END{
print "Latest package: "filepkg", date: ",temp
}'
an example run of the above:
linux$ ./findlatest.sh
Latest package: kernel-headers-2.6.18-164.6.1.el5.x86_64.rpm, date: 200911041457
Try this (which requires lynx):
lynx -dump -listonly -nonumbers http://mirror.centos.org/centos/5/updates/x86_64/RPMS/ |
grep -E '^.*xen-libs.*i386.rpm$' |
sort --version-sort |
tail -n 1
If your sort doesn't have --version-sort, then you'll have to parse the version out of the filename or hope that a regular sort will do the right thing.
You may be able to do something similar with wget or curl or even a Bash script using redirections with /dev/tcp/HOST/PORT. The problem with these is that you would then have to parse HTML.