I'm trying to create a cron job that downloads the latest version of WhatsApp's APK from their website using a bash script and make it available through my site.
So far, I'm able to obtain the version number from the site using the following (user-agent part omitted):
wget -q -O - "$#" whatsapp.com/android | grep -oP '(?<=Version )([\d.]+)'
And I can download the APK using the following command:
wget http://www.whatsapp.com/android/current/WhatsApp.apk
That part is fine. What I can't figure out is how to download the APK only if it's newer than the existing APK on the server. How should the script be?
Since I'm not a command-line pro, I guess there's a better way to achieve this than my current approach, so if you have any suggestions, I'd appreciate it very much.
Seems like you need to manage the version yourself.
I would store the apk files with a version number in the filename, e.g WhatsApp_<version-number>_.apk. So the script that downloads the newer file can be as following:
# Get the local version
oldVer=$(ls -v1 | grep -v latest | tail -n 1 | awk -F "_" '{print $2}')
# Get the server version
newVer=$(wget -q -O - "$#" whatsapp.com/android | grep -oP '(?<=Version )([\d.]+)')
# Check if the server version is newer
newestVer=$(echo -e "$oldVer\n$newVer" | sort -n | tail -n 1)
#Download the newer versino
[ "$newVer" = "$newestVer" ] && [ "$oldVer" != "$newVer" ] && wget -O WhatsApp_${newVer}_.apk http://www.whatsapp.com/android/current/WhatsApp.apk || echo "The newest version already downloaded"
#Delete all files that not is a new version
find ! -name "*$newVer*" ! -type d -exec rm -f {} \;
# set the link to the latest
ln -sf $(ls -v1 | grep -v latest| tail -n1) latest
Related
I'm trying to create a code which will fetch the latest version of anaconda and install it.
Currently we can do this to install the latest version:
mkdir tmp
cd tmp
wget https://repo.anaconda.com/archive/Anaconda3-2020.11-Linux-x86_64.sh
bash Anaconda3-2020.11-Linux-x86_64.sh
I want the script to be more generalized such that the code would automatically find the latest version of anaconda, download the shell script file and install it.
You can use this to get the latest version:
wget https://repo.anaconda.com/archive/ -q -O- |\
grep 'Anaconda3'| \
sed -n 's|.*>Anaconda3-\([0-9]\{4\}\.[0-9]\{2\}\)-.*|\1|p'
uniq |\
sort -r |\
head -1
This solution works only for those versions that use the year format (e.g. 2020-07), but since the latest version will presumably be of that format that should be fine.
Some explanation:
wget to fetch the contents of the archive page, which gives us the HTML content containing all the download URLs. -q quiets the output, -O- prints to stdout. Alternatively, you can use curl -s to the same effect.
grep 'Anaconda3' gives us the lines containing Anaconda, which contain the download links.
Use sed to select the version strings from the download links, e.g. 2020-11. That gives you a list of all versions (of the format YYYY-MM).
Sort that lists and select the first entry, which is the latest version.
Use the version in the rest of your script and you are done. A complete solution would be:
version=$(wget https://repo.anaconda.com/archive/ -q -O- |\
grep 'Anaconda3'|\
sed -n 's|.*>Anaconda3-\([0-9]\{4\}\.[0-9]\{2\}\)-.*|\1|p' |\
uniq |\
sort -r |\
head -1)
wget "https://repo.anaconda.com/archive/Anaconda3-$version-Linux-x86_64.sh"
I'm sure fetching the latest version could be made more efficient, but this should be sufficient for your use case.
I use the following to install the most recent docker compose for my ubuntu server:
curl -L https://github.com/docker/compose/releases/download/1.21.2/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
How to do I make this more version agnostic. For instance, so that I do not have to go in and keep changing the version -which in this case is 1.21.2. How do I change the command so it gets the most latest stable release?
How do I change the command so it gets the most latest stable release?
You could try following:
curl -L https://github.com/docker/compose/releases/download/`curl -Ls -o /dev/null -w %{url_effective} https://github.com/docker/compose/releases/latest | awk -F / '{print $NF}'`/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
This is same as your script only replacing actual version (1.21.2 in your case) with latest tag over several steps:
First we get redirection url for latest stable:
curl -Ls -o /dev/null -w %{url_effective} https://github.com/docker/compose/releases/latest
currently it resolves to https://github.com/docker/compose/releases/tag/1.21.2
Then we get version tag out of redirection url:
| awk -F / '{print $NF}'
currently resolving to 1.21.2
Finally we execute it in place of version number using your original curl statement. Note that this can break if latest tag is not properly redirected and ads some extra complexity, but automates version pulling as requested.
Accepted answer isn't the latest stable version according to https://docs.docker.com/compose/release-notes/ (returns v2 instead of the latest v1 which I was looking for)
This is the monstrosity I went with
rm -Rf /usr/local/bin/docker-compose && version=$(curl -s https://docs.docker.com/compose/release-notes/ | grep "Docker Compose release notes" | grep "Estimated reading time" | sed 's/.*id=//g' | sed 's/<.*$//g' | sed 's/.*>//g') && curl -L https://github.com/docker/compose/releases/download/${version}/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose && chmod +x /usr/local/bin/docker-compose
I am trying to use wget in a bash and display a custom download percentage per file so that the user knows the process is running and what has been downloaded. The below seems to download the files but there is no percentage or filename being downloaded displayed. I am not sure why and cannot seem to figure it out. Thank you :).
list
xxxx://www.xxx.com/xxx/xxxx/xxx/FilterDuplicates.html
xxxx://www.xxx.com/xxx/xxxx/xxx/file1.bam
xxxx://www.xxx.com/xxx/xxxx/xxx/file2.bam
xxxx://www.xxx.com/xxx/xxxx/xxx/file1.vcf.gz
xxxx://www.xxx.com/xxx/xxxx/xxx/file2.vcf.gz
bash that uses list to download all fiiles
# download all from list
download() {
local url=$1
echo -n " "
wget --progress=dot $url 2>&1 | grep --line-buffered "%" | sed -u -e "s,\.,,g" | awk '{printf("\b\b\b\b%4s", $2)}'
echo -ne "\b\b\b\b"
echo " starting download"
}
cd "/home/user/Desktop/folder/subfolder"
wget -i /home/cmccabe/list --user=xxx--password=xxx --xxx \
xxxx://www.xxx.com/xxx/xxxx/xxx/ 2>&1 -o wget.log | download
Does anyone know an easy way to set up OSX dev machines to install xcode and set up a clean dev environment every morning at 5am?
I was looking at using something like boxen, but the problem I was facing is they all need Xcode installed to work.
You can actually automate a lot of the pre-requites for Boxen with scripts, such as automating the Xcode installation step with a script something like this:
#!/bin/sh
# Get and install Xcode CLI tools
OSX_VERS=$(sw_vers -productVersion | awk -F "." '{print $2}')
# on 10.9+, we can leverage SUS to get the latest CLI tools
if [ "$OSX_VERS" -ge 9 ]; then
# create the placeholder file that's checked by CLI updates' .dist code
# in Apple's SUS catalog
touch /tmp/.com.apple.dt.CommandLineTools.installondemand.in-progress
# find the CLI Tools update
PROD=$(softwareupdate -l | grep "\*.*Command Line" | head -n 1 | awk -F"*" '{print $2}' | sed -e 's/^ *//' | tr -d '\n')
# install it
softwareupdate -i "$PROD" -v
# on 10.7/10.8, we instead download from public download URLs, which can be found in
# the dvtdownloadableindex:
# https://devimages.apple.com.edgekey.net/downloads/xcode/simulators/index-3905972D-B609-49CE-8D06-51ADC78E07BC.dvtdownloadableindex
else
[ "$OSX_VERS" -eq 7 ] && DMGURL=http://devimages.apple.com/downloads/xcode/command_line_tools_for_xcode_os_x_lion_april_2013.dmg
[ "$OSX_VERS" -eq 8 ] && DMGURL=http://devimages.apple.com/downloads/xcode/command_line_tools_for_osx_mountain_lion_april_2014.dmg
TOOLS=clitools.dmg
curl "$DMGURL" -o "$TOOLS"
TMPMOUNT=`/usr/bin/mktemp -d /tmp/clitools.XXXX`
hdiutil attach "$TOOLS" -mountpoint "$TMPMOUNT"
installer -pkg "$(find $TMPMOUNT -name '*.mpkg')" -target /
hdiutil detach "$TMPMOUNT"
rm -rf "$TMPMOUNT"
rm "$TOOLS"
exit
fi
I have a small script that i'd like to use with cron.
Purpose: Get webpage with links, extract dates from link and download files.
Script below is not working 100% and i can't see the problem.
#!/bin/bash
for i in $(curl http://107.155.72.213/anarirecap.php 2>&1 | grep -o -E 'href="([^"#]+)"' | cut -d'"' -f2 | grep '_whole_1_3000.mp4'); do
GAMEDAY=$(echo "$i" | grep -Eo '[[:digit:]]{4}/[[:digit:]]{2}/[[:digit:]]{2}')
wget "$i" --output-document="$GAMEDAY.mp4"
done
It get's the webpage "curl http://...etc" - works
$DAY - extracts the date - works.
wget part not working when i add $DAY. I'm i blind ... what am i missing.
Look at your output format here:
wget "$i" -O 2015/05/12.mp4
This is looking for a directory named 2015 with a subdirectory named 05 in which to place the file 12.mp4. Those directories don't exist, so you get 2015/05/12.mp4: No such file or directory.
If you want to replace the /s with underscores:
wget -O "${GAMEDAY//\//_}" "$i"
Alternately, if you want to create the directories if they don't exist:
mkdir -p -- "(dirname "$GAMEDAY")"
wget -O "$GAMEDAY" "$i"