How to download specific links from a page with Nokogiri - ruby

Question

I have a webpage with a list of names (which are regular links). When I click on the names of the first page, this opens up another page which has a list of files for download as links. I want to download only the links that end with fq.qz for all of the page1 links.
To do this I have been trying to use Nokogiri:
require 'nokogiri'
require 'open-uri'
url = 'http://myURL/'
doc = Nokogiri::HTML(open(url))
puts doc.css('li')[2]['href']
doc.traverse do |el|
[el[:src], el[:href]].grep(/\.(fq.gz)$/i).map{|l| URI.join(url, l).to_s}.each do |link|
File.open(File.basename(link),'wb'){|f| f << open(link,'rb').read}
end
end
However, I don't think this opens up each of the page 1 links to get the fq.gz ending files in the next level.
The format of the links I am interested in is:
<td>SLX-7998.blabla.fq.gz</td>
I tried using this code which is heavily adapted from one of the answers below but nothing gets downloaded and I get the array as below
master_page.links_with(:href => /ViewSample/).map {|link| link.click
link = agent.get(agent.page.uri.to_s)
if link.content.include?("fq.gz")
out_file = File.new("downloaded_file", "w")
out_file.puts(agent.get_file(link[:href]))
out_file.close
end
=> [nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil]

Answer 1

This is the basis for a quick search for anchors containing certain sub-strings in the linked-text:
require 'nokogiri'
doc = Nokogiri::HTML(<<EOT)
foo.fq.gz
bar.fq.gz
baz
EOT
nodes = doc.search('a').select{ |node| node.text[/fq\.gz$/] }
At this point nodes is a NodeSet of Nodes that match the /fq\.gz$/ pattern in their text:
nodes
# => [#(Element:0x3fd9818bda2c {
# name = "a",
# attributes = [
# #(Attr:0x3fd982027060 { name = "href", value = "http://foo" })],
# children = [ #(Text "foo.fq.gz")]
# }),
# #(Element:0x3fd9818bd928 {
# name = "a",
# attributes = [
# #(Attr:0x3fd982035ef8 { name = "href", value = "http://bar" })],
# children = [ #(Text "bar.fq.gz")]
# })]
We can walk through those and extract just the href parameters:
hrefs = nodes.map{ |node| node['href'] }
Resulting in an array that can be iterated over:
hrefs
# => ["http://foo", "http://bar"]
You should be able to figure out the rest.

Answer 2

You sound like you could use Mechanize, which is a tool for automating interaction with web pages that uses Nokogiri as dependency. You could probably do something like this:
require 'mechanize'
$agent = Mechanize.new
master_page = $agent.get("http://master_page")
master_page.search("a.download_list_link") do |download_list_link|
download_list_page = $agent.get(download_list_link[:href])
download_list_page.search("td > a") do |link|
if link.content.include?("fq.gz")
out_file = File.new("downloaded_file", "w")
out_file.puts($agent.get_file(link[:href]))
out_file.close
end
end
end
Some things that I wrote there will depend on the specific names of elements on the pages you're visiting, but I think that the general idea there will solve your problem.
Edit:
Regarding the errors you're getting with an array of nil objects, one problem that I see is that you forgot to close the block:
master_page.links_with(:href => /ViewSample/).map {|link| link.click
...
# no terminating curly brace