I want to download the two (.bin and .zip) binaries from the latest releases.
I tried using the following command
curl -s https://github.com/Atmosphere-NX/Atmosphere/releases/latest | grep "browser_download_url.*zip" | cut -d : -f 2,3 | tr -d "\" | wget -qi -
but nothing happens, output being SYSTEM_WGETRC = c:/progra~1/wget/etc/wgetrc
I'm open to using any other (wget, ecurl etc) commands.
Is it trying to extract the download link from the HTML page? That's error prone and may break any time.
For such operations, check if they offer an API first.
They do: https://docs.github.com/en/rest/reference/releases#get-the-latest-release
You could write something like (pseudo code):
curl \
-H "Accept: application/vnd.github.v3+json" \
https://api.github.com/repos/Atmosphere-NX/Atmosphere/releases/latest \
| jq .assets[0].browser_download_url \
| xargs wget -qi -
Like suggested in the comments, test each command (pipe separated) individually.
You can use the GitHub CLI, specifically the release download command:
gh release download --repo Atmosphere-NX/Atmosphere --pattern '*.bin'
gh release download --repo Atmosphere-NX/Atmosphere --archive zip
Without specifying a release tag, the command defaults to the latest release.
Just running curl on the url gives this:
curl https://github.com/Atmosphere-NX/Atmosphere/releases/latest
<html><body>You are being redirected.</body>
So, you easily see something is amiss straight off. Checking the curl help, you find options, command below to pinpoint what you need:
curl --help | grep redirect
-L, --location Follow redirects
--max-redirs <num> Maximum number of redirects allowed
--proto-redir <protocols> Enable/disable PROTOCOLS on redirect
--stderr Where to redirect stderr
First clue is redirect in the response and then we see in the help section that there is a flag to handle that.
Running it with th -L command gives the expected output. Pipeing it to grep "browser_download_url.*zip" however gives you nothing. You then investigate to see what the right match would be. But let's try mathing just html link with zip, just to see what happens.
curl -sL https://github.com/Atmosphere-NX/Atmosphere/releases/latest | grep "href=.*zip"
<a href="/Atmosphere-NX/Atmosphere/releases/download/1.2.6/atmosphere-1.2.6-master-173d5c2d3+hbl-2.4.1+hbmenu-3.5.0.zip" rel="nofollow" data-skip-pjax>
<a href="/Atmosphere-NX/Atmosphere/archive/refs/tags/1.2.6.zip" rel="nofollow" data-skip-pjax>
From there you can probably find what you are after to construct your command. As you see, links are relative with this method, so you still have to provide the base url to wget (or a curl equivalent) to finally be able to dowload what you are after.
This is more a reply to get you going on trouble shooting. You already have other answers to actually do what you want. But if you can't install the tools suggested, you could probably do something like this:
curl -sL https://github.com/Atmosphere-NX/Atmosphere/releases/latest |
awk '/releases\/download/ && done != 1 {
sub(/.*href="/, "https://github.com")
sub(/".*/, "")
print
done = 1
}' |
xargs curl -LsO
Not suggesting this is a good way, just a way.
Related
Looking for a simple way to download a .zip from a latest GitHub release.
There are other similar questions, but I havent been able to get them to work. :(
Trying to pull latest release from https://github.com/CTCaer/hekate
Currently ive got:
#!/bin/bash
curl -s https://api.github.com/repos/CTCaer/hekate/releases/latest | jq -r ".assets[] | select(.name | test(\"hekate_ctcaer\")) | .browser_download_url"
trying to fetch the url of the latest .zip and only grab the "hekate_ctcaer_X.X.X_Nyx_X.X.X.zip"
I saw someone trying to achieve this with 'Xidel', so im open to trying that if someone knows the syntax to grab a specific file from the GitHub api.
As I understand it (?), the Github API spits out an array for the release 'assets', so im trying to specify an item in this array that matches "hekate_ctcaer", and download the specified file.
Github is also a compatible git repo. I provide a new train of thought.
use git ls-remote to fetch last release tag.
git -c 'versionsort.suffix=-' ls-remote --tags --sort='v:refname' http://github.com/CTCaer/hekate.git
| tail --lines=1
| cut --delimiter='/' --fields=3
Here this examples outputs v5.8.0
then clone remote repo
git clone --branch v5.8.0 http://github.com/CTCaer/hekate.git
zip repos to zipped file.
zip hekate.zip -r hekate/
This will print out the url to the zip file of the latest release:
curl -sL https://api.github.com/repos/CTCaer/hekate/tags \
| jq -r '.[0].zipball_url' \
| xargs -I {} curl -sL {} -o latest.zip
I saw someone trying to achieve this with 'Xidel'
I assume you're referring to my answer here. That answer is tagged batch-file, so you first of all have to swop the quotes for bash ("function('string')" --> 'function("string")'). And secondly, you're right. You have to select the appropriate object in the "assets"-array.
$ xidel -s "https://api.github.com/repos/CTCaer/hekate/releases/latest" \
-f '$json/(assets)()[starts-with(name,"hekate_ctcaer")]/browser_download_url' \
--download '{substring-after($headers[starts-with(.,"Content-Disposition")],"filename=")}'
This downloads 'hekate_ctcaer_5.8.0_Nyx_1.3.0.zip' in the current dir.
With r8389 or newer you can just use --download ..
also how would I modify this for the following: github.com/Atmosphere-NX/Atmosphere/releases/tag/1.3.2 the .zip AND the .bin
Strictly speaking you'd have to raise a new question for this, but ok.
It appears that (at the moment) v1.3.2 is also the latest release for this repo, so you can use...
$ xidel -s "https://api.github.com/repos/Atmosphere-NX/Atmosphere/releases/latest" \
-e '$json'
or alternatively...
$ xidel -s "https://api.github.com/repos/Atmosphere-NX/Atmosphere/releases" \
-e '$json()[tag_name="1.3.2"]'
The "assets"-array here has just 2 objects; one with the zip-file and one with the bin-file, so just "follow" (--follow / -f) the 2 "browser_download_url"-keys to download:
$ xidel -s "https://api.github.com/repos/Atmosphere-NX/Atmosphere/releases" \
-f '$json()[tag_name="1.3.2"]//browser_download_url' \
--download .
Simple question, where I already spent hours searching for an answer:
The following HTTP check doesn't work, despite it's validity can be proven:
./check_http --sni -H www.wikimedia.de -u /ueber-uns/ -S -s "cli-privacy-readmore"
Verify this with a visit to the source code and search for it: https://www.wikimedia.de/ueber-uns/.
The verbose switch gives more insight, though that's where I'm at the end of my knowledge:
> ./check_http --sni -H www.wikimedia.de -u /ueber-uns/ -S -s "cli-privacy-readmore" -v | grep cli-privacy-rea -A2 -n
1348: <a class="cli-privacy-rea
1349-3000
1350-dmore" data-readmore-text="Zeig mehr" data-readless-text="Zeige weniger"></a> </div>
Why is there a 3000 (on line 1349) in it? I've already found out that it's HEX and somehow used for length estimation. But neither a look inside tcpdump (the value is somehow transmitted, or added afterwards to the packets), nor the test with other variables (other gateway, os, site) brings me closer to a solution.
Can anyone help me?
Edit:
Tested on:
Archlinux check_http v2.3 (monitoring-plugins 2.3)
Ubuntu 18.04 check_http v2.2 (monitoring-plugins 2.2)
Your issue is not reproducible.
# check_http --version
check_http v2.3.3 (nagios-plugins 2.3.3)
# check_http --sni -H www.wikimedia.de -u /ueber-uns/ -S -s "cli-privacy-readmore" -v | grep cli-privacy-rea -A2 -n
1340: <a class="cli-privacy-readmore" data-readmore-text="Zeig mehr" data-readless-text="Zeige weniger"></a> </div>
1341- </div>
1342- <div class="cli-col-12 cli-align-items-stretch cli-px-0 cli-tab-section-container">
# check_http --sni -H www.wikimedia.de -u /ueber-uns/ -S -s "cli-privacy-readmore"
HTTP OK: HTTP/1.1 200 OK - 95466 bytes in 0.375 second response time |time=0.375144s;;;0.000000 size=95466B;;;0
Try a newer version of the plugin.
edit: You also may not need to use a Nagios plugin for this as it's a pretty basic use case, unless you need the perfdata or similar you can just do this:
# curl --silent https://www.wikimedia.de/ueber-uns/ | grep -q "cli-privacy-readmore" ; echo $?
0
# curl --silent https://www.wikimedia.de/ueber-uns/ | grep -q "banana" ; echo $?
1
It works ok as a single tool:
curl "someURL"
curl -o - "someURL"
but it doesn't work in a pipeline:
curl "someURL" | tr -d '\n'
curl -o - "someURL" | tr -d '\n'
it returns:
(23) Failed writing body
What is the problem with piping the cURL output? How to buffer the whole cURL output and then handle it?
This happens when a piped program (e.g. grep) closes the read pipe before the previous program is finished writing the whole page.
In curl "url" | grep -qs foo, as soon as grep has what it wants it will close the read stream from curl. cURL doesn't expect this and emits the "Failed writing body" error.
A workaround is to pipe the stream through an intermediary program that always reads the whole page before feeding it to the next program.
E.g.
curl "url" | tac | tac | grep -qs foo
tac is a simple Unix program that reads the entire input page and reverses the line order (hence we run it twice). Because it has to read the whole input to find the last line, it will not output anything to grep until cURL is finished. Grep will still close the read stream when it has what it's looking for, but it will only affect tac, which doesn't emit an error.
For completeness and future searches:
It's a matter of how cURL manages the buffer, the buffer disables the output stream with the -N option.
Example:
curl -s -N "URL" | grep -q Welcome
Another possibility, if using the -o (output file) option - the destination directory does not exist.
eg. if you have -o /tmp/download/abc.txt and /tmp/download does not exist.
Hence, ensure any required directories are created/exist beforehand, use the --create-dirs option as well as -o if necessary
The server ran out of disk space, in my case.
Check for it with df -k .
I was alerted to the lack of disk space when I tried piping through tac twice, as described in one of the other answers: https://stackoverflow.com/a/28879552/336694. It showed me the error message write error: No space left on device.
You can do this instead of using -o option:
curl [url] > [file]
So it was a problem of encoding. Iconv solves the problem
curl 'http://www.multitran.ru/c/m.exe?CL=1&s=hello&l1=1' | iconv -f windows-1251 | tr -dc '[:print:]' | ...
If you are trying something similar like source <( curl -sS $url ) and getting the (23) Failed writing body error, it is because sourcing a process substitution doesn't work in bash 3.2 (the default for macOS).
Instead, you can use this workaround.
source /dev/stdin <<<"$( curl -sS $url )"
Trying the command with sudo worked for me. For example:
sudo curl -O -k 'https url here'
note: -O (this is capital o, not zero) & -k for https url.
I had the same error but from different reason. In my case I had (tmpfs) partition with only 1GB space and I was downloading big file which finally filled all memory on that partition and I got the same error as you.
I encountered the same problem when doing:
curl -L https://packagecloud.io/golang-migrate/migrate/gpgkey | apt-key add -
The above query needs to be executed using root privileges.
Writing it in following way solved the issue for me:
curl -L https://packagecloud.io/golang-migrate/migrate/gpgkey | sudo apt-key add -
If you write sudo before curl, you will get the Failed writing body error.
For me, it was permission issue. Docker run is called with a user profile but root is the user inside the container. The solution was to make curl write to /tmp since that has write permission for all users , not just root.
I used the -o option.
-o /tmp/file_to_download
In my case, I was doing:
curl <blabla> | jq | grep <blibli>
With jq . it worked: curl <blabla> | jq . | grep <blibli>
I encountered this error message while trying to install varnish cache on ubuntu. The google search landed me here for the error (23) Failed writing body, hence posting a solution that worked for me.
The bug is encountered while running the command as root curl -L https://packagecloud.io/varnishcache/varnish5/gpgkey | apt-key add -
the solution is to run apt-key add as non root
curl -L https://packagecloud.io/varnishcache/varnish5/gpgkey | apt-key add -
The explanation here by #Kaworu is great: https://stackoverflow.com/a/28879552/198219
This happens when a piped program (e.g. grep) closes the read pipe before the previous program is finished writing the whole page. cURL doesn't expect this and emits the "Failed writing body" error.
A workaround is to pipe the stream through an intermediary program that always reads the whole page before feeding it to the next program.
I believe the more correct implementation would be to use sponge, as already suggested by #nisetama in the comments:
curl "url" | sponge | grep -qs foo
I got this error trying to use jq when I didn't have jq installed. So... make sure jq is installed if you're trying to use it.
In Bash and zsh (and perhaps other shells), you can use process substitution (Bash/zsh) to create a file on the fly, and then use that as input to the next process in the pipeline chain.
For example, I was trying to parse JSON output from cURL using jq and less, but was getting the Failed writing body error.
# Note: this does NOT work
curl https://gitlab.com/api/v4/projects/ | jq | less
When I rewrote it using process substitution, it worked!
# this works!
jq "" <(curl https://gitlab.com/api/v4/projects/) | less
Note: jq uses its 2nd argument to specify an input file
Bonus: If you're using jq like me and want to keep the colorized output in less, use the following command line instead:
jq -C "" <(curl https://gitlab.com/api/v4/projects/) | less -r
(Thanks to Kowaru for their explanation of why Failed writing body was occurring. However, their solution of using tac twice didn't work for me. I also wanted to find a solution that would scale better for large files and tries to avoid the other issues noted as comments to that answer.)
I was getting curl: (23) Failed writing body . Later I noticed that I did not had sufficient space for downloading an rpm package via curl and thats the reason I was getting issue. I freed up some space and issue for resolved.
I had the same question because of my own typo mistake:
# fails because of reasons mentioned above
curl -I -fail https://www.google.com | echo $?
curl: (23) Failed writing body
# success
curl -I -fail https://www.google.com || echo $?
I added flag -s and it did the job. eg: curl -o- -s https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash
In the janus project, they use curl to download and pipe a bootstrap script into bash.
https://github.com/carlhuda/janus
It looks like this:
$ curl -Lo- https://bit.ly/janus-bootstrap | bash
Why would one want to use the args -Lo-?
-o is supposed to be for output, but wouldn't that happen anyway (i.e. to stdout)?
It's all in the man pages:
-L in case the page has moved (3xx response) curl will redirect the request to the new address
-o output to a file instead of stdout (usually the screen). In your case the o flag is redundant since the output is piped to bash (for execution) - not to a file.
The -o is redundant, they produce the exact same output:
$ curl --silent example.com | sha256sum
3587cb776ce0e4e8237f215800b7dffba0f25865cb84550e87ea8bbac838c423 *-
$ curl --silent --output - example.com | sha256sum
3587cb776ce0e4e8237f215800b7dffba0f25865cb84550e87ea8bbac838c423 *-
They have used that syntax since that line was first introduced in 2011.
You might ask Wael Nasreddine (#kalbasit on GitHub) why he did it. He
is still active on that repo.
I have a list of URLS that I need to check, to see if they still work or not. I would like to write a bash script that does that for me.
I only need the returned HTTP status code, i.e. 200, 404, 500 and so forth. Nothing more.
EDIT Note that there is an issue if the page says "404 not found" but returns a 200 OK message. It's a misconfigured web server, but you may have to consider this case.
For more on this, see Check if a URL goes to a page containing the text "404"
Curl has a specific option, --write-out, for this:
$ curl -o /dev/null --silent --head --write-out '%{http_code}\n' <url>
200
-o /dev/null throws away the usual output
--silent throws away the progress meter
--head makes a HEAD HTTP request, instead of GET
--write-out '%{http_code}\n' prints the required status code
To wrap this up in a complete Bash script:
#!/bin/bash
while read LINE; do
curl -o /dev/null --silent --head --write-out "%{http_code} $LINE\n" "$LINE"
done < url-list.txt
(Eagle-eyed readers will notice that this uses one curl process per URL, which imposes fork and TCP connection penalties. It would be faster if multiple URLs were combined in a single curl, but there isn't space to write out the monsterous repetition of options that curl requires to do this.)
wget --spider -S "http://url/to/be/checked" 2>&1 | grep "HTTP/" | awk '{print $2}'
prints only the status code for you
Extending the answer already provided by Phil. Adding parallelism to it is a no brainer in bash if you use xargs for the call.
Here the code:
xargs -n1 -P 10 curl -o /dev/null --silent --head --write-out '%{url_effective}: %{http_code}\n' < url.lst
-n1: use just one value (from the list) as argument to the curl call
-P10: Keep 10 curl processes alive at any time (i.e. 10 parallel connections)
Check the write_out parameter in the manual of curl for more data you can extract using it (times, etc).
In case it helps someone this is the call I'm currently using:
xargs -n1 -P 10 curl -o /dev/null --silent --head --write-out '%{url_effective};%{http_code};%{time_total};%{time_namelookup};%{time_connect};%{size_download};%{speed_download}\n' < url.lst | tee results.csv
It just outputs a bunch of data into a csv file that can be imported into any office tool.
This relies on widely available wget, present almost everywhere, even on Alpine Linux.
wget --server-response --spider --quiet "${url}" 2>&1 | awk 'NR==1{print $2}'
The explanations are as follow :
--quiet
Turn off Wget's output.
Source - wget man pages
--spider
[ ... ] it will not download the pages, just check that they are there. [ ... ]
Source - wget man pages
--server-response
Print the headers sent by HTTP servers and responses sent by FTP servers.
Source - wget man pages
What they don't say about --server-response is that those headers output are printed to standard error (sterr), thus the need to redirect to stdin.
The output sent to standard input, we can pipe it to awk to extract the HTTP status code. That code is :
the second ($2) non-blank group of characters: {$2}
on the very first line of the header: NR==1
And because we want to print it... {print $2}.
wget --server-response --spider --quiet "${url}" 2>&1 | awk 'NR==1{print $2}'
Use curl to fetch the HTTP-header only (not the whole file) and parse it:
$ curl -I --stderr /dev/null http://www.google.co.uk/index.html | head -1 | cut -d' ' -f2
200
wget -S -i *file* will get you the headers from each url in a file.
Filter though grep for the status code specifically.
I found a tool "webchk” written in Python. Returns a status code for a list of urls.
https://pypi.org/project/webchk/
Output looks like this:
▶ webchk -i ./dxieu.txt | grep '200'
http://salesforce-case-status.dxi.eu/login ... 200 OK (0.108)
https://support.dxi.eu/hc/en-gb ... 200 OK (0.389)
https://support.dxi.eu/hc/en-gb ... 200 OK (0.401)
Hope that helps!
Keeping in mind that curl is not always available (particularly in containers), there are issues with this solution:
wget --server-response --spider --quiet "${url}" 2>&1 | awk 'NR==1{print $2}'
which will return exit status of 0 even if the URL doesn't exist.
Alternatively, here is a reasonable container health-check for using wget:
wget -S --spider -q -t 1 "${url}" 2>&1 | grep "200 OK" > /dev/null
While it may not give you exact status out, it will at least give you a valid exit code based health responses (even with redirects on the endpoint).
Due to https://mywiki.wooledge.org/BashPitfalls#Non-atomic_writes_with_xargs_-P (output from parallel jobs in xargs risks being mixed), I would use GNU Parallel instead of xargs to parallelize:
cat url.lst |
parallel -P0 -q curl -o /dev/null --silent --head --write-out '%{url_effective}: %{http_code}\n' > outfile
In this particular case it may be safe to use xargs because the output is so short, so the problem with using xargs is rather that if someone later changes the code to do something bigger, it will no longer be safe. Or if someone reads this question and thinks he can replace curl with something else, then that may also not be safe.