I need to download a pdf from a website which does not provide a link ending with (.pdf) using ruby. Manually, when i click on the link to download the pdf, it takes me to a new page and the dialog box to save/open the file appears after some time.
Please help me in downloading the file.
The link
You an do this
require 'open-uri'
File.open('my_file_name.pdf', "wb") do |file|
file.write open('http://someurl.com/2013-1-2/somefile/download').read
end
I have been doing this for my projects and it works.
If you just need a simple ruby script to do it, I'd just run wget. Like this exec 'wget "http://path.to.the.file/and/some/params"'
At that point though, you might as well run wget.
The other way, is to just run a get on the page that you know the pdf is at
source = Net::HTTP.get("http://the.website.com", "/and/some/params")
There are a number of other http clients that you could use, but as long as you make a get request to the endpoint that the pdf is at, it should give you the raw data. Then you can just rename the file, and you'll have the pdf
In your case, I ran the following commands to get the pdf
wget http://www.lawcommission.gov.np/en/documents/prevailing-laws/constitution/func-download/129/chk,d8c4644b0f086a04d8d363cb86fb1647/no_html,1/
mv index.html thefile.pdf
Then open the pdf. Note that these are linux commands. If you want to get the file with a ruby script, you could use something like what I previously mentioned.
Update:
There is an added complication that was not initially stated, which is that the url to the pdf changes every time there is an update to the pdf. In order to make this work, you probably want to do something involving web scraping. I suggest nokogiri. This way you can look at the page where the download is and then perform a get request on the desired URL. Furthermore, the server that hosts the pdf is misconfigured, and breaks chrome within a few seconds of opening the page.
How to solve this problem: I went to the site, and refreshed it. Then broke the connection to the server (press the X where there would otherwise be a refresh button). Then right click next to the download link, and select inspect element. Then browse the dom to find something that is definitively identifying (like an id). Thankfully, I found something <strong id="telecharger"> Download</strong>. This means that you can use something like page.css('strong#telecharger')[0].parent['href'] This should give you a URL. Then you can perform a get request as described above. I don't have time to make the script for you (too much work to do), but this should be enough to solve the problem.
Related
We have a saved .webarchive and we want to retrieve the original URL. Is that possible?
Background. My wife filled out a long application on the web and saved a local copy, .webarchive. The instructions said that to make changes you have to go to URL of where you were when you were at a certain step of the submission. The instructions are complicated/confusing and like most of these long applications hard to deal with anyway. She doesn't have that URL. We went back to her Safari history and one URL for the site that day but that just came up with an error.
To give a sense on how the sophistication of the site, they have a link for downloading Flash Player.
We're trying to contact the site. Due in 24 hours. Fortunately what they have is OK, she just wanted to make some edits and add some info.
I looked at the 13k lines of the .webarchive in a text editor and skimming through it don't see anything obvious. There is some com.apple.print plist embedded but no URL. I looked at Get Info and no URL (some things I download from the web have the original URL).
Thank you for any help.
Actually, webarchive itself is in the binary plist format and can be read like .plist files. The original URL of a webarchive file, if there is one, should be stored at :WebMainResource:WebResourceURL, which can be read with:
/usr/libexec/PlistBuddy -c 'print ":WebMainResource:WebResourceURL"' /path/to/file
Has anyone been able to find a way to test pdf's with ruby within the browser? I have tried a few different ways and the only way I have been able to get any pdf testing to work is to save off the pdf and use the pdf_reader gem. This only seems to work on pdf's that, when the link is clicked, opens up a dialog box with the options to open or save the pdf. Unfortunately I have not been able to find a way to do anything like this with pdf's that are opened in browser, with no dialog box options to save it. Any ideas?
Maybe testing it in the browser isnt the best way. When you say test the pdf what are you trying to do? I wouldnt test the pdf in the browser if I was you.
Try docsplit, if you want to verify its contents.
Docsplit is a command-line utility and Ruby library for splitting apart documents into their component parts: searchable UTF-8 plain text via OCR if necessary, page images or thumbnails in any format, PDFs, single pages, and document metadata (title, author, number of pages...)
You are not inventing a browser, or a PDF generator.
Use unit tests to check your back-end modules can take data in, and write PDF out, then serve the PDF in a website and let the browser do its thing. Test (as what Rails calls a "functional test") that the MVC will produce a web page containing a link to the PDF, and you are done.
You can use gem 'mechanize' to download an online PDF (the PDF with in a browser) on your computer and then read it via gem PDF reader.
I would like to save a web page programmatically.
I don't mean merely save the HTML. I would also like automatically to store all associated files (images, CSS files, maybe embedded SWF, etc), and hopefully rewrite the links for local browsing.
The intended usage is a personal bookmarks application, in which link content is cached in case the original copy is taken down.
Take a look at wget, specifically the -p flag
−p −−page−requisites
This option causes Wget to download all the files
that are necessary to properly display
a givenHTML page. Thisincludes such
things as inlined images, sounds, and
referenced stylesheets.
The following command:
wget -p http://<site>/1.html
Will download page.html and all files it requires.
On Windows: you can run IE as a com object and pull everything out.
On other thing, you can take the source of Mozilla.
In Java, Lobo.
Or commons-httpclient and write a lot of code.
You could try the MHTML format (which is what IE uses). http://en.wikipedia.org/wiki/MHTML
In other words, you'd be downloading each object (image, css, etc.) to your computer, and then "embedding" them, via Base64, into a single file.
Back in the earlier days of the internet I remember that in certain browsers, every time you downloaded an image or a file, the URL of where that file was downloaded from would be written into that file's properties (I guess the summary tab?). I think Netscape v2 did this if I remember correctly.
I really miss that kind of functionality as every once in a while I'll run into a neat little program stored somewhere in the depths of my hard drive and wonder where I got it from originally.
I googled around but I'm not quite sure what terms to use to describe what I'm looking for. So I'm wondering if anyone knows of a Firefox plug-in or something similar that would do this?
If you use the DownThemAll! extension for Firefox, you can tell it to prepend the URL of the site to the downloaded file name...
thus you end up with files like:
download.com_utils_compression_ABCD32.exe
It also works really well when you want to download/queue a bunch of files.
You download http://example.com/foo to ~/Desktop/foo, and you want to see the originating URL in the properties of the local file foo?
Back when I used OS X, I remember Safari used to record the original URL in the resource fork of the downloaded file. Can't remember what the named fork is, well, named, but it'll show up in the properties panel from Finder. Since it's there, Spotlight will probably index it, too, but I haven't used OS X since 10.3.
If you use Opera, and haven't cleared the file out from your download manager, select the download and it'll show the original URL that the file is from in the properties pane.
Is this what you want? If so... well, I don't know of a similar Firefox extension, but it'll clarify the question.
For the IE Browser I use the hell out of Fidler to look at all traffic going across the wire.
For FireFox, you can use the FireBug plugin. There is a "Net" tab that will show you request information that is going across the wire.
Most of the time you can use one of these tools to see what URL was requested in order to start a download. You can also view all the get and post information that might need to be sent in order to have your request succeed.
Fidler is here: http://www.fiddlertool.com/fiddler/
FireBug is here: https://addons.mozilla.org/en-US/firefox/addon/1843
Best of Luck!
I'm trying to load an image from the Firefox cache as the title suggests. I'm running Ubuntu, so the location of my cache is /home/me/.mozilla/firefox/xxxxxx.default/Cache
However, in the Cache (and this is on Mac, too) the filenames are just ridiculous combinations of letters and numbers. Is there a way to pinpoint a certain file?
You should take a look at the source code of the CacheViewer Add-on.
Download the file instead of installing it (right click and save as) and then extract it (it's just a Zip file, even though it has a .xpi extension), then extract the cacheviewer.jar file inside the resulting chrome folder. Finally go into content and then cacheviewer to find the javascript and XUL files.
From my brief investigation, the useful routines are in the cacheviewer.js file, though if you were hoping there would be a simple javascript one liner for accessing cached items you're probably going to be disappointed. The XUL files (which are just XML) are helpful in working out which JS functions are called to perform particular tasks. I'm not too sure how all this maps into Greasemonkey, rather than the extension environment, but hopefully there's enough code to get you started.
Ummm, that really is an internal implementation detail. But I suggest looking at how about:cache?device=disk and about:cache-entry?client=HTTP&sb=1&key=https://stackoverflow.com/Content/img/wmd/blockquote.png are implemented.
Also, http://www.securityfocus.com/infocus/1832 gives details, too. Note that Firefox doesn't use a separate file for everything...
And of course, Firefox may change the format at any time.
Just give your img src= attribute the full URL. If the image happens to be cacheable (the server sends an appropriate Expires: or Cache-control: header, for example) and it's already in the cache, Firefox will not hit the network.
HTTP caching is supposed to be invisible. When you're generating content, you generally shouldn't worry about it.
You can point REDbot at a URL to see all sorts of delicious information about its cacheability.