I have figured out how to download all images from a particular website: wget -i wget -qO- http://example.com | sed -n '/<img/s/.src="([^"])"./\1/p' | awk '{gsub("thumb-350-", "");print}' What I HAVEN'T figured out is how to download images from ALL pages of a website which increments like this (http://example.com/page/) and restrict the download to images of a certain size or bigger. Can you help me?
I used that command in Ubuntu's terminal and managed to download all the images from the original page. Now, to avoid repeating the same process 131 times, the number of pages in the blog, and to avoid downloading images that are too little, I'd like you to help me tweak that command.
Related
I am trying to download the google search results page for Gop primaries results using wget but I am not able to do that (this page). But, I noticed that the webpage is getting the data from this file https://goo.gl/KPGSqS which it gets using a GET request.
So, I was wondering if there is a way to download that file with wget? The usual way i do is using wget -c url but that is not working. So, any ideas on what i should do for this?
I tried with the user-agent option, but even that isn't working.
If you want to download a webpage content (parsed to simple text file or source html code) you could consider using lynx. Download lynx by typing sudo apt-get lynx and then you could save the webpage content using lynx http://your.url/ > savefile.txt.
You can find how to use lynx in this page
I'm trying to download all the images that appear on the page with WGET, it seems that eveything is fine but the command is actually downloading only the first 6 images, and no more. I can't figure out why.
The command i used:
wget -nd -r -P . -A jpeg,jpg http://www.edpeers.com/2013/weddings/umbria wedding-photographer/
It's downloading only the first 6 images relevant of the page and all other stuff that i don't need, look at the page, any idea why it's only getting the first 6 relevant images?
Thanks in advance.
I'm currently using a bash script to download several images using wget.
Unfortunately the server I am downloading from is less than reliable and therefore sometimes when I'm downloading a file, the server will disconnect and the script will move onto the next file, leaving the previous one incomplete.
In order to remedy this I've tried to add a second line after the script fetches all incomplete files using:
wget -c myurl.com/image{1..3}.png
This seems to work as wget goes back and completes download of the files, but the problem then comes from this: ImageMagick which I use to stich the images in a pdf, claims there are errors with the headers of the images.
My thought of what to with deleting the incomplete files is:
wget myurl.com/image{1..3}.png
wget -rmincompletefiles
wget -N myurl.com/image{1..3}.png
convert *.png mypdf.pdf
So the question is, what can I use in place of -rmincompletefiles that actually exists, or is there a better I should be approaching this issue?
I made surprising discovery when attempting to implement tvm's suggestion.
It turns out, and this something I didn't realize, that when you run wget -N, wget actually checks file sizes and verifies they are the same. If they are not, the files are deleted and then downloaded again.
So cool tip if you're having the same issue I am!
I've found this solution to work for my use case.
From the answer:
wget http://www.example.com/mysql.zip -O mysql.zip || rm -f mysql.zip
This way, the file will only be deleted if an error or cancellation occurred.
Well, I would try hard to download the files with wget (you can specify extra parameters like larger --timeout to give the server some extra time). wget assumes certain things about the partial downloads and even with proper resume, they can sometimes end up mangled (unless you check their eg. MD5 sums by other means).
Since you are using convert and bash, there will be most likely another tool available from the Imagemagick package - namely identify.
While certain features are surely poorly documented, it has one awesome functionality - it can identify broken (or partially downloaded images).
➜ ~ identify b.jpg; echo $?
identify.im6: Invalid JPEG file structure: ...
1
It will return exit status 1 if you call it on the inconsistent image. You can remove these inconsistent images using simple loop such as:
for i in *.png;
do identify "$i" || rm -f "$i";
done
Then I would try to download again the files that are broken.
I'd like to download multiple numbered images from a website.
The images are structured like this:
http://website.com/images/foo1bar.jpg
http://website.com/images/foo2bar.jpg
http://website.com/images/foo3bar.jpg
... And I'd like to download all of the images within a specific interval.
Are there simple browser addons that could do this, or should I use "wget" or the like?
Thank you for your time.
Crudely, on Unix-like systems:
#!/bin/sh
for i in {1..3}
do
wget http://website.com/images/foo"$i"bar.jpg
done
Try googling "bash for loop".
Edit LOL! Indeed, in a haste I omitted the name of the very program that downloads the image files. Also, this goes into a text editor, then you save it with an arbitrary file name, make it executable with the command
chmod u+x the_file_name
and finally you run it with
./the_file_name
I need to create a static copy of a web page (all media resources, like CSS, images and JS included) in a shell script. This copy should be openable offline in any browser.
Some browsers have a similar functionality (Save As... Web Page, complete) which create a folder from a page and rewrite external resources as relative static resources in this folder.
What's a way to accomplish and automatize this on Linux command line to a given URL?
You can use wget like this:
wget --recursive --convert-links --domains=example.org http://www.example.org
this command will recursively download any page reachable by hyperlinks from the page at www.example.org not following links outside the example.org domain.
Check wget manual page for more options for controlling recursion.
You want the tool wget to mirror a site do:
$ wget -mk http://www.example.com/
Options:
-m --mirror
Turn on options suitable for mirroring. This option turns on recursion and time-stamping, sets infinite recursion depth and keeps
FTP
directory listings. It is currently equivalent to -r -N -l inf --no-remove-listing.
-k --convert-links
After the download is complete, convert the links in the document to make them suitable for local viewing. This affects not
only the
visible hyperlinks, but any part of the document that links to external content, such as embedded images, links to style sheets,
hyperlinks to non-HTML content, etc.