Let's say my HTML document is like:
<div class="headline">News</div>
<p>Some interesting news here</p>
<div class="headline">Sports</div>
<p>Baseball is fun!</p>
I can get the headline divs with the following code:
require 'rubygems'
require 'nokogiri'
require 'open-uri'
url = "mypage.html"
doc = Nokogiri::HTML(open(url))
doc.css(".headline").each do |item|
puts item.text
end
But how do I access the content in the following p tag so that News is related to Some interesting news here, etc?
You want Node#next_element:
doc.css(".headline").each do |item|
puts item.text
puts item.next_element.text
end
There is also item.next, but that will also return text nodes, where item.next_element will return only element nodes (like p).
Related
I am trying to extract only the <p> that exist between Vigentes and Finalizados without achieving it.
require 'nokogiri'
require 'open-uri'
require 'time'
#url = "http://www.caru.org.uy/web/servicios/llamados-a-concurso-publico-para-contratar-personal/"
page = Nokogiri::HTML(open(#url))
div_content = page.css('.contenido')
div_content.each do |item|
puts item.text
break if item.css('h3').text == "Finalizados"
end
You should be able to do:
css = 'h3:contains(Vigentes) ~ p:has(~ h3:contains(Finalizados))'
But unfortunately, nokogiri doesn't behave properly for this one so we'll use xpath:
xpath = "//h3[contains(text(), 'Vigentes')]/following-sibling::p[./following-sibling::h3[contains(text(), 'Finalizados')]]"
page.search(xpath).each do |p|
# do something
end
What I am trying to do: Parse links from website (http://nytm.org/made-in-nyc) that all have the exact same content. "(hiring)" Then I will write to a file 'jobs.html' a list of links. (If it is a violation to publish these websites I will quickly take down the direct URL. I thought it might be useful as a reference to what I am trying to do. First time posting on stack)
DOM Structure:
<article>
<ol>
<li>#waywire</li>
<li><a href="http://1800Postcards.com" target="_self" class="vt-p">1800Postcards.com</a</li>
<li>Adafruit Industries</li>
<li><a href="http://www.adafruit.com/jobs/" target="_self" class="vt-p">(hiring)</a</li>
etc...
What I have tried:
require 'nokogiri'
require 'open-uri'
def find_jobs
doc = Nokogiri::HTML(open('http://nytm.org/made-in-nyc'))
hire_links = doc.css("a").select{|link| link.text == "(hiring)"}
results = hire_links.each{|link| puts link['href']}
begin
file = File.open("./jobs.html", "w")
file.write("#{results}")
rescue IOError => e
ensure
file.close unless file == nil
end
puts hire_links
end
find_jobs
Here is a Gist
Example Result:
[344] #<Nokogiri::XML::Element:0x3fcfa2e2276c name="a" attributes=[#<Nokogiri::XML::Attr:0x3fcfa2e226e0 name="href" value="http://www.zocdoc.com/careers">, #<Nokogiri::XML::Attr:0x3fcfa2e2267c name="target" value="_blank">] children=[#<Nokogiri::XML::Text:0x3fcfa2e1ff1c "(hiring)">]>
So it successfully writes these entries into the jobs.html file but it is in XML format? Not sure how to target just the value and create a link from that. Not sure where to go from here. Thanks!
The problem is with how results is defined. results is an array of Nokogiri::XML::Element:
results = hire_links.each{|link| puts link['href']}
p results.class
#=> Array
p results.first.class
#=> Nokogiri::XML::Element
When you go to write the Nokogiri::XML::Element to the file, you get the results of inspecting it:
puts results.first.inspect
#=> "#<Nokogiri::XML::Element:0x15b9694 name="a" attributes=...."
Given that you want the href attribute of each link, you should collect that in the results instead:
results = hire_links.map{ |link| link['href'] }
Assuming you want each href/link displayed as a line in the file, you can join the array:
File.write('./jobs.html', results.join("\n"))
The modified script:
require 'nokogiri'
require 'open-uri'
def find_jobs
doc = Nokogiri::HTML(open('http://nytm.org/made-in-nyc'))
hire_links = doc.css("a").select { |link| link.text == "(hiring)"}
results = hire_links.map { |link| link['href'] }
File.write('./jobs.html', results.join("\n"))
end
find_jobs
#=> produces a jobs.html with:
#=> http://www.20x200.com/jobs/
#=> http://www.8coupons.com/home/jobs
#=> http://jobs.about.com/index.html
#=> ...
Try using Mechanize. It leverages Nokogiri, and you can do something like
require 'mechanize'
browser = Mechanize.new
page = browser.get('http://nytm.org/made-in-nyc')
links = page.links_with(text: /(hiring)/)
Then you will have an array of link objects that you can get whatever info you want. You can also use the link.click method that Mechanize provides.
I have the below code which is a part of an html:
<td>YouTube</td>
<td><a data-category="news" href=http://kathack.com/party/aems/dic/list">Reddit</a></td>
<td>Kathack</td>
<td><a data-category="news" href="http://www.nytimes.com">New York Times</a></td>
now how would I search the /aems/dic/list and get the full url stored?
So, with nokogiri, something like this:
fragment = Nokogiri::HTML::DocumentFragment.parse text
fragment.css("a").each do |link|
href = link['href']
return href if href =~ /\/aems\/dic\/list/
end
Let's say you have a Mechanize::Page object page:
page.at('a[href*="/aems/dic/list"]')[:href]
#=> "http://kathack.com/party/aems/dic/list"
Update
For a longer example:
require 'mechanize'
agent = Mechanize.new
page = agent.get 'http://www.example.com/'
page.at('a[href*="/aems/dic/list"]')[:href]
#=> "http://kathack.com/party/aems/dic/list"
For example:
content=Nokogiri::HTML(open(url)).at_css(".appwindow").text
This example parse text from .appwindow (only text).
How can I parse this text with <p> tag?
I think you want to find either the full HTML of the first element that has an appwindow class, or perhaps the inner HTML. If so:
require 'nokogiri'
html = Nokogiri::HTML <<ENDHTML
<div id='menu'>menu</div>
<div class='appwindow'><p>Hello <b>World</b>!</p></div>
ENDHTML
puts html.at_css('.appwindow').text
#=> Hello World!
puts html.at_css('.appwindow').to_html
#=> <div class="appwindow"><p>Hello <b>World</b>!</p></div>
puts html.at_css('.appwindow').inner_html
#=> <p>Hello <b>World</b>!</p>
See the list of methods on Nokogiri::XML::Node for other options available to you.
what would be the most efficient way of grabbing all texts between html tags ?
<div>
<a> hi </a>
....
bunch of texts surrounded by html tags.
doc = Nokogiri::HTML(your_html)
doc.xpath("//text()").to_s
Use a Sax parser. Much faster than the XPath option.
require "nokogiri"
some_html = <<-HTML
<html>
<head>
<title>Title!</title>
</head>
<body>
This is the body!
</body>
</html>
HTML
class TextHandler < Nokogiri::XML::SAX::Document
def initialize
#chunks = []
end
attr_reader :chunks
def cdata_block(string)
characters(string)
end
def characters(string)
#chunks << string.strip if string.strip != ""
end
end
th = TextHandler.new
parser = Nokogiri::HTML::SAX::Parser.new(th)
parser.parse(some_html)
puts th.chunks.inspect
Just do:
doc = Nokogiri::HTML(your_html)
doc.xpath("//text()").text
Here's how to get all the text in the question div of this page:
require 'rubygems'
require 'nokogiri'
require 'open-uri'
doc = Nokogiri::HTML(open("http://stackoverflow.com/questions/1512850/grabbing-text-between-all-tags-in-nokogiri"))
puts doc.css("#question").to_s