Retrieving images under a specific Div with Mechanize and Ruby - ruby

Question

I need to retrieve all the images present under a specific div using Ruby and Mechanize. The relevant DOM structure is as follows:
<div id="item_img">
<a href="JavaScript:imageview('000000018693.jpg')">
<img src="/shop/doubleimages/0000000186932.jpg" border="0" width="500" height="500" alt="関係ないコメント z1808">
</a>
<img src="/shop/doubleimages/000000018693_1.jpg"><br><br>
<img src="/shop/doubleimages/000000018693_2.jpg"><br><br>
<img src="/shop/doubleimages/000000018693_3.jpg"><br><br>
</div>
So, I initially got all the images after spinning up a new agent by doing:
agent = Mechanize.new
agent.user_agent_alias = 'Mac Safari'
page = agent.get ('http://double14.com/shopdetail/000000018693/')
puts page.images
This was nice, but it every image on the page (as it should), and seems to strip out the div id above it, making it impossible to decide what comes from where. As a result, I had every image on the page (no bueno).
I got it down to this:
agent = Mechanize.new
agent.user_agent_alias = 'Mac Safari'
page = agent.get ('http://double14.com/shopdetail/000000018693/')
node = page.search "#item_img img"
node.each do |n|
puts n.attributes['src']
end
Unfortunately, that outputs the following -
/shop/doubleimages/0000000186932.jpg
/shop/doubleimages/000000018693_1.jpg
/shop/doubleimages/000000018693_2.jpg
/shop/doubleimages/000000018693_3.jpg
Is there a way to take the full URL and use that instead? Ultimately, I would like to save these images to a database, but I need the full URL to serialize them to disk for later upload.

Answer 1

Yes. You can get the full URL for the images with the #resolve method:
require 'mechanize'
mechanize = Mechanize.new
mechanize.user_agent_alias = 'Mac Safari'
page = mechanize.get('http://double14.com/shopdetail/000000018693/')
page.search('#item_img img').each do |img|
puts mechanize.resolve(img['src'])
end
Alternatively you can use the #download method to download them directly.

Answer 2

This is how I did it for a collection of images. In this case the base_uri is the url that you are passing to get. Let me know if you have any questions.
def self.qualify_images(base_uri, images)
images.map do |image|
next unless has_src?(image)
qualify_image(base_uri, image)
end.compact
end
def self.qualify_image(base_uri, image)
src = image.attributes["src"].value
if src =~ /^\/[\/]/
result = "#{scheme(base_uri)}#{src}"
elsif src =~ /^\//
result = "#{base_uri}#{src}"
else
result = src
end
http?(result) ? result : nil
end
def self.has_src?(image)
image.attributes["src"].value
rescue NoMethodError
false
end
def self.scheme(uri)
uri = URI.parse(uri)
"#{uri.scheme}:"
end
def self.http?(uri)
uri = URI.parse(uri)
uri.kind_of?(URI::HTTP)
rescue URI::InvalidURIError
false
end
This will ensure a fully qualified uri for each image.

Answer 3

It will look something like:
page.search("#item_img img").each do |img|
puts page.uri.merge(img[:src]).to_s
end