Bash script to download latest release from GitHub - bash

Looking for a simple way to download a .zip from a latest GitHub release.
There are other similar questions, but I havent been able to get them to work. :(
Trying to pull latest release from https://github.com/CTCaer/hekate
Currently ive got:
#!/bin/bash
curl -s https://api.github.com/repos/CTCaer/hekate/releases/latest | jq -r ".assets[] | select(.name | test(\"hekate_ctcaer\")) | .browser_download_url"
trying to fetch the url of the latest .zip and only grab the "hekate_ctcaer_X.X.X_Nyx_X.X.X.zip"
I saw someone trying to achieve this with 'Xidel', so im open to trying that if someone knows the syntax to grab a specific file from the GitHub api.
As I understand it (?), the Github API spits out an array for the release 'assets', so im trying to specify an item in this array that matches "hekate_ctcaer", and download the specified file.

Github is also a compatible git repo. I provide a new train of thought.
use git ls-remote to fetch last release tag.
git -c 'versionsort.suffix=-' ls-remote --tags --sort='v:refname' http://github.com/CTCaer/hekate.git
| tail --lines=1
| cut --delimiter='/' --fields=3
Here this examples outputs v5.8.0
then clone remote repo
git clone --branch v5.8.0 http://github.com/CTCaer/hekate.git
zip repos to zipped file.
zip hekate.zip -r hekate/

This will print out the url to the zip file of the latest release:
curl -sL https://api.github.com/repos/CTCaer/hekate/tags \
| jq -r '.[0].zipball_url' \
| xargs -I {} curl -sL {} -o latest.zip

I saw someone trying to achieve this with 'Xidel'
I assume you're referring to my answer here. That answer is tagged batch-file, so you first of all have to swop the quotes for bash ("function('string')" --> 'function("string")'). And secondly, you're right. You have to select the appropriate object in the "assets"-array.
$ xidel -s "https://api.github.com/repos/CTCaer/hekate/releases/latest" \
-f '$json/(assets)()[starts-with(name,"hekate_ctcaer")]/browser_download_url' \
--download '{substring-after($headers[starts-with(.,"Content-Disposition")],"filename=")}'
This downloads 'hekate_ctcaer_5.8.0_Nyx_1.3.0.zip' in the current dir.
With r8389 or newer you can just use --download ..
also how would I modify this for the following: github.com/Atmosphere-NX/Atmosphere/releases/tag/1.3.2 the .zip AND the .bin
Strictly speaking you'd have to raise a new question for this, but ok.
It appears that (at the moment) v1.3.2 is also the latest release for this repo, so you can use...
$ xidel -s "https://api.github.com/repos/Atmosphere-NX/Atmosphere/releases/latest" \
-e '$json'
or alternatively...
$ xidel -s "https://api.github.com/repos/Atmosphere-NX/Atmosphere/releases" \
-e '$json()[tag_name="1.3.2"]'
The "assets"-array here has just 2 objects; one with the zip-file and one with the bin-file, so just "follow" (--follow / -f) the 2 "browser_download_url"-keys to download:
$ xidel -s "https://api.github.com/repos/Atmosphere-NX/Atmosphere/releases" \
-f '$json()[tag_name="1.3.2"]//browser_download_url' \
--download .

Related

How to download the latest binary release from github?

I want to download the two (.bin and .zip) binaries from the latest releases.
I tried using the following command
curl -s https://github.com/Atmosphere-NX/Atmosphere/releases/latest | grep "browser_download_url.*zip" | cut -d : -f 2,3 | tr -d "\" | wget -qi -
but nothing happens, output being SYSTEM_WGETRC = c:/progra~1/wget/etc/wgetrc
I'm open to using any other (wget, ecurl etc) commands.
Is it trying to extract the download link from the HTML page? That's error prone and may break any time.
For such operations, check if they offer an API first.
They do: https://docs.github.com/en/rest/reference/releases#get-the-latest-release
You could write something like (pseudo code):
curl \
-H "Accept: application/vnd.github.v3+json" \
https://api.github.com/repos/Atmosphere-NX/Atmosphere/releases/latest \
| jq .assets[0].browser_download_url \
| xargs wget -qi -
Like suggested in the comments, test each command (pipe separated) individually.
You can use the GitHub CLI, specifically the release download command:
gh release download --repo Atmosphere-NX/Atmosphere --pattern '*.bin'
gh release download --repo Atmosphere-NX/Atmosphere --archive zip
Without specifying a release tag, the command defaults to the latest release.
Just running curl on the url gives this:
curl https://github.com/Atmosphere-NX/Atmosphere/releases/latest
<html><body>You are being redirected.</body>
So, you easily see something is amiss straight off. Checking the curl help, you find options, command below to pinpoint what you need:
curl --help | grep redirect
-L, --location Follow redirects
--max-redirs <num> Maximum number of redirects allowed
--proto-redir <protocols> Enable/disable PROTOCOLS on redirect
--stderr Where to redirect stderr
First clue is redirect in the response and then we see in the help section that there is a flag to handle that.
Running it with th -L command gives the expected output. Pipeing it to grep "browser_download_url.*zip" however gives you nothing. You then investigate to see what the right match would be. But let's try mathing just html link with zip, just to see what happens.
curl -sL https://github.com/Atmosphere-NX/Atmosphere/releases/latest | grep "href=.*zip"
<a href="/Atmosphere-NX/Atmosphere/releases/download/1.2.6/atmosphere-1.2.6-master-173d5c2d3+hbl-2.4.1+hbmenu-3.5.0.zip" rel="nofollow" data-skip-pjax>
<a href="/Atmosphere-NX/Atmosphere/archive/refs/tags/1.2.6.zip" rel="nofollow" data-skip-pjax>
From there you can probably find what you are after to construct your command. As you see, links are relative with this method, so you still have to provide the base url to wget (or a curl equivalent) to finally be able to dowload what you are after.
This is more a reply to get you going on trouble shooting. You already have other answers to actually do what you want. But if you can't install the tools suggested, you could probably do something like this:
curl -sL https://github.com/Atmosphere-NX/Atmosphere/releases/latest |
awk '/releases\/download/ && done != 1 {
sub(/.*href="/, "https://github.com")
sub(/".*/, "")
print
done = 1
}' |
xargs curl -LsO
Not suggesting this is a good way, just a way.

Install CURRENT version of Gradle within CI

Gradle describes their installation in their docs:
https://docs.gradle.org/current/userguide/installation.html
However the docs requires using specific version and there is no easy way to say "get the current version".
I would like to do something like:
export GRADLE_VERSION=`curl https://services.gradle.org/current-version`
curl -sSLO https://services.gradle.org/distributions/gradle-${GRADLE_VERSION}-bin.zip \
&& unzip -q -d /opt/gradle gradle-*.zip \
&& echo "PATH=$PATH:/opt/gradle/gradle-${GRADLE_VERSION}/bin" > ~/.bashrc
Obviously there is no such API service like /current-version. How would you achieve this?
I suggest:
curl -s 'https://services.gradle.org/versions/current' | jq -r '.version'
Output:
7.3.1
or
curl -s 'https://services.gradle.org/versions/current' | jq -r '.downloadUrl'
Output:
https://services.gradle.org/distributions/gradle-7.3.1-bin.zip

Finding latest version of anaconda automatically from bashrc

I'm trying to create a code which will fetch the latest version of anaconda and install it.
Currently we can do this to install the latest version:
mkdir tmp
cd tmp
wget https://repo.anaconda.com/archive/Anaconda3-2020.11-Linux-x86_64.sh
bash Anaconda3-2020.11-Linux-x86_64.sh
I want the script to be more generalized such that the code would automatically find the latest version of anaconda, download the shell script file and install it.
You can use this to get the latest version:
wget https://repo.anaconda.com/archive/ -q -O- |\
grep 'Anaconda3'| \
sed -n 's|.*>Anaconda3-\([0-9]\{4\}\.[0-9]\{2\}\)-.*|\1|p'
uniq |\
sort -r |\
head -1
This solution works only for those versions that use the year format (e.g. 2020-07), but since the latest version will presumably be of that format that should be fine.
Some explanation:
wget to fetch the contents of the archive page, which gives us the HTML content containing all the download URLs. -q quiets the output, -O- prints to stdout. Alternatively, you can use curl -s to the same effect.
grep 'Anaconda3' gives us the lines containing Anaconda, which contain the download links.
Use sed to select the version strings from the download links, e.g. 2020-11. That gives you a list of all versions (of the format YYYY-MM).
Sort that lists and select the first entry, which is the latest version.
Use the version in the rest of your script and you are done. A complete solution would be:
version=$(wget https://repo.anaconda.com/archive/ -q -O- |\
grep 'Anaconda3'|\
sed -n 's|.*>Anaconda3-\([0-9]\{4\}\.[0-9]\{2\}\)-.*|\1|p' |\
uniq |\
sort -r |\
head -1)
wget "https://repo.anaconda.com/archive/Anaconda3-$version-Linux-x86_64.sh"
I'm sure fetching the latest version could be made more efficient, but this should be sufficient for your use case.

wget do not download subirectories only all files in specified directory [duplicate]

I am trying to download the files for a project using wget, as the SVN server for that project isn't running anymore and I am only able to access the files through a browser. The base URLs for all the files is the same like
http://abc.tamu.edu/projects/tzivi/repository/revisions/2/raw/tzivi/*
How can I use wget (or any other similar tool) to download all the files in this repository, where the "tzivi" folder is the root folder and there are several files and sub-folders (upto 2 or 3 levels) under it?
You may use this in shell:
wget -r --no-parent http://abc.tamu.edu/projects/tzivi/repository/revisions/2/raw/tzivi/
The Parameters are:
-r //recursive Download
and
--no-parent // Don´t download something from the parent directory
If you don't want to download the entire content, you may use:
-l1 just download the directory (tzivi in your case)
-l2 download the directory and all level 1 subfolders ('tzivi/something' but not 'tivizi/somthing/foo')
And so on. If you insert no -l option, wget will use -l 5 automatically.
If you insert a -l 0 you´ll download the whole Internet, because wget will follow every link it finds.
You can use this in a shell:
wget -r -nH --cut-dirs=7 --reject="index.html*" \
http://abc.tamu.edu/projects/tzivi/repository/revisions/2/raw/tzivi/
The Parameters are:
-r recursively download
-nH (--no-host-directories) cuts out hostname
--cut-dirs=X (cuts out X directories)
This link just gave me the best answer:
$ wget --no-clobber --convert-links --random-wait -r -p --level 1 -E -e robots=off -U mozilla http://base.site/dir/
Worked like a charm.
wget -r --no-parent URL --user=username --password=password
the last two options are optional if you have the username and password for downloading, otherwise no need to use them.
You can also see more options in the link https://www.howtogeek.com/281663/how-to-use-wget-the-ultimate-command-line-downloading-tool/
use the command
wget -m www.ilanni.com/nexus/content/
you can also use this command :
wget --mirror -pc --convert-links -P ./your-local-dir/ http://www.your-website.com
so that you get the exact mirror of the website you want to download
try this working code (30-08-2021):
!wget --no-clobber --convert-links --random-wait -r -p --level 1 -E -e robots=off --adjust-extension -U mozilla "yourweb directory with in quotations"
I can't get this to work.
Whatever I try, I just get some http file.
Just looking at these commands for simply downloading a directory?
There must be a better way.
wget seems the wrong tool for this task, unless it is a complete failure.
This works:
wget -m -np -c --no-check-certificate -R "index.html*" "https://the-eye.eu/public/AudioBooks/Edgar%20Allan%20Poe%20-%2"
This will help
wget -m -np -c --level 0 --no-check-certificate -R"index.html*"http://www.your-websitepage.com/dir

git grep and xargs in Windows Batch file?

I am trying to create a Windows friendly .bat implementation of the following .sh script. The top few lines are all fine, just add SET and cd is fine. git grep is fine, however, xargs isn't... What would the git grep | xargs logic look like in .bat ?
INFINITY=10000
TOPDIR=$(pwd)
METEOR_DIR="./code"
cd "$METEOR_DIR"
# Call git grep to find all js files with the appropriate comment tags,
# and only then pass it to JSDoc which will parse the JS files.
# This is a whole lot faster than calling JSDoc recursively.
git grep -al "#summary" | xargs -L ${INFINITY} -t \
"$TOPDIR/node_modules/.bin/jsdoc" \
-t "$TOPDIR/jsdoc/docdata-jsdoc-template" \
-c "$TOPDIR/jsdoc/jsdoc-conf.json" \
2>&1 | grep -v 'WARNING: JSDoc does not currently handle'
Any recent Git for Windows release has more than 200 Linux commands packaged in it.
Add to your PATH <path\to\Git\usr\bin and you will have xargs.
vonc#VONCM D:\prgs\git\PortableGit-2.9.2-64-bit\usr\bin
> dir xargs.exe
Directory of D:\prgs\git\PortableGit-2.9.2-64-bit\usr\bin
20/01/2016 10:17 64 058 xargs.exe

Resources