Proper way of reading a page and saving it into a html file? - ruby

Question

I have the following:
require 'rubygems'
require 'anemone'
require 'nokogiri'
require 'open-uri'
Anemone.crawl("http://www.findbrowsenodes.com/", :delay => 3) do |anemone|
anemone.on_pages_like(/http:\/\/www.findbrowsenodes.com\/us\/.+\/[\d]*/) do | page |
doc = Nokogiri::HTML(open(page.url))
id = doc.at_css("#n_info #clipnode").text unless doc.at_css("#n_info #clipnode").nil?
File.open("#{node_id}.html", "wb") do |f|
f.write(open(page).read)
end
end
end
So I'm trying to save each URL as a html file with this:
File.open("#{id}.html", "wb") do |f|
f.write(open(page).read)
end
But I get this error:
alex#alex-K43U:~/rails/anemone$ ruby anemone.rb
/home/alex/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/open-uri.rb:35:in
open': can't convert Anemone::Page into String (TypeError) from
/home/alex/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/open-uri.rb:35:in
open' from anemone.rb:27:in block (3 levels) in <main>' from
anemone.rb:26:inopen' from anemone.rb:26:in `block (2 levels) in
'
What's the right way of doing this?

Answer 1

There are several problems / confusions:
As the error says, the open methods expects a String (i.e. the url), but you're providing an Anemone::Page object.
This object has a url method, which you already use on line 9.
On line 9: open(page.url)
You're already opening the page, so you could reuse that. But:
According to the docs http://anemone.rubyforge.org/doc/classes/Anemone/Page.html Anemone::Page contains a body method that may already contain the content (I'm just guessing, haven't use or tried that library). If that's the case, there's no need to use open.
As I see it, the following untested code may be more like what you're looking for:
doc = Nokogiri::HTML(page.body)
# [snip]
File.open("#{node_id}.html", "wb") do |f|
f.write(page.body)
end