I download a .tar.gz file using wget using this command:
wget hello.tar.gz
This is a part of a long script, sometimes when I want to download this file, an error occurs and when for the second time the file is downloaded the name of the downloaded file changes to something like this:
hello.tar.gz.2
the third time:
hello.tar.gz.3
How can I say that the whatever the name of the downloaded is, change it to hello.tar.gz?
In other words I don't want the name of the downloaded file be anything other than hello.tar.gz?
wget hello.tar.gz -O <fileName>
wget have internal option like -r, -p to change default behavior
So just try the following:
wget -p <url>
wget -r <url>
Since now you noticed the incremental change. Discard any repeated files and rely on the following as initial condition:
wget hello.tar.gz
mv hello.tar.gz.2 hello.tar.gz
Related
wget -E -H -k -K -p -e robots=off -P ./images/ -i./list.txt
./list.txt: No such file or directory
No URLs found in ./list.txt.
Converted links in 0 files in 0 seconds.
I downloaded and installed brew. Further, I installed wget and it's letting me download images one image at a time. However, when I tried the aforementioned command to download images from multiple urls, it's not doing anything. Can someone tell me what I could be doing wrong here?
wget is pretty lucid with description of issue
./list.txt: No such file or directory
apparently there is not file named list.txt inside current dir. Please trying giving full path to list.txt.
A script regularly downloads some data file from a remote server with wget:
CERTDIR=folder1
SPOOLDIR=folder2
URL="http://..."
FILENAME="$SPOOLDIR/latest.xml.gz"
/usr/bin/wget \
-N \
--quiet \
--private-key=${CERTDIR}/keynopass.pem \
--ca-certificate=${CERTDIR}/ca.pem \
--certificate=${CERTDIR}/client.pem \
"$URL" \
--output-document ${FILENAME}
The -N switch is used to turn on timestamping. (possibly redundant, this seems to be the default)
I was anticipating that the file only will be downloaded if there is a newer remote version.
But this is not the case. The actual download is done, no matter if the remote file has the same timestamp as the local file.
The file is a bit lengthy, so my plan was to check for a new version frequently, but download only as needed. Unfortunately this seems impossible with that approach.
Just guessing: the URL references no file, but is an api call. Could this be the reason?
But: the timestamp of the local file is set to the timestamp of the remote file - so I know, that the timestamp information is available.
Do I miss something?
Notes:
the remote server is not controlled by me
the local server runs ubuntu 16.04
wget --version: GNU Wget 1.17.1 built on linux-gnu.
The documentation mentions that:
Use of -O is not intended to mean simply "use the name file instead of
the one in the URL;" rather, it is analogous to shell redirection:
wget -O file http://foo is intended to work like wget -O - http://foo > file;
file will be truncated immediately, and all downloaded content will be written there.
For this reason, -N (for timestamp-checking) is not supported in
combination with -O: since file is always newly created, it will
always have a very new timestamp. A warning will be issued if this
combination is used.
Thus one option would be to leave out the -O option, let wget download the file (if needed), and just create a symlink in your target directory called latest.xml.gz pointing to the downloaded file...
Im trying to do a bash script and i need to download certain files with wget
like libfat-nds-1.0.11.tar.bz2 but after some times the version of this file may change so i would like to download a file that start with libfatnds and ends in .tar.bz2 .Is this possible with wget?
Using only wget, it can be achieved by specifying filename with wildcards in the list of accepted extensions.
wget -r -np -nd --accept='libfat-nds-*.tar.bz2'
The problem is that HTTP doesn't support wildcard downloads
. But if there is content listing enabled on the server or you have a index.html containing the available file names you could download that, extract the file name you need and then download the file with wget.
Something in this order
Download the index with curl
Use grep and/or sed to extract the exact file name
Download the file with wget (or curl)
If you pipe the commands you can do it on one line.
Hi i have a text file where download links are given like -
http://www.example.com/10.10.11/abc.jpg
http://www.example.com/10.10.12/abc.jpg
http://www.example.com/10.10.13/abc.jpg
Here 10.10.* is the date of the image.
I need to download all the images using wget where the image name will be the corresponding date (eg. 10.10.111.jpg).
PS. I tried using:
wget -i download.txt
So, any solution?
Thanks in advance
You can instruct Wget to create subdirectories based on the URL, and then do the renaming after the download has finished.
I'd suggest a batch script that downloads the files one by one using the -O option, and a bit of sed/awk magic to get the names right
But careful! given the -O option, you have to call wget on a per file basis
This should do the trick.
#!/bin/sh
while read url; do
urldir=${url%/*}
dir=${urldir##*/}
wget -O $dir.jpg $url
done < download.txt
This might work for you:
sed '\|/\([^/]*\)/[^/]*\1[^/.]*.jpg|!d' download.txt | wget -i -
Explanation:
Filter the download.txt file to contain only those files which you require and then pass them on to wget.
I have developed a script that does just this bulkGetter. Super easy to use, you just need an input file with all the links you want to download and use option "-rb" (refer to link).
I'm using mac's terminal.
I want to copy images from remote url: http://media.pragprog.com/titles/rails4/code/depot_b/public/images/ to a local directory.
What's the command to do that?
Tnx,
You can use curl
curl -O "http://media.pragprog.com/titles/rails4/code/depot_b/public/images/*.jpg"
for example.
alternatively you may want just all the images, from a website. wget can do this with a recursive option such as:
$ wget -r -A=jpeg,jpg,bmp,png,gif,tiff,xpm,ico http://www.website.com/
This should only download the comma delimited extensions recursively starting at the site index. This works like a web-spider so if its not referenced anywhere on the site it will miss the image.
wget will work, assuming the server has directory listing:
wget -m http://media.pragprog.com/titles/rails4/code/depot_b/public/images
You can do this with Wget or cURL. If I recall correctly, neither come out-of-the-box w/ OS X, so you may need to install them with MacPorts or something similar.