I'm trying to parse a html raw file using nokogiri.
html_file = URI.open(url).read
html_doc = Nokogiri::HTML(html_file)
puts html_doc.search("p", "h2").map(&:text)
When I do this, I get all the "p" text and then all the "h2" text. Is there a way to get them in the order that they appear in the original text?
I tried something like this below but it doesn't quite work
puts html_doc.search("p" || "h2").map(&:text)
Sorry, found my own answer.
puts html_doc.search("p, h2").map(&:text)
Related
I'm trying to write a Ruby script that will take the Flickr BBCode from an image and only find the actual image link and ignore all of the other stuff.
The BBCode from Flickr looks like this:
<img src="https://farm3.staticflickr.com/2864/92917419471_248187_c.jpg" width="800" height="526" alt="Wiggle Wiggle">
and I'm trying to get my output to be just the link, so: https://farm3.staticflickr.com/2864/92917419471_248187_c.jpg
So far, my code is this
#!/usr/bin/ruby
require 'rubygems'
str1 = ""
puts "What text would you like me to use? "
text = gets
text.scan(/"([^"]*)"/) { str1 = $1}
puts str1
and I need to know how I can scan through the input and only find the part that starts at https and ends with the quote. Any help is appreciated
Don't try to parse HTML with a regex.
Instead, use an HTML parser. Something like Nokogiri http://nokogiri.org/
require 'nokogiri'
doc = Nokogiri::HTML.parse '<img src="https://farm3.staticflickr.com/2864/92917419471_248187_c.jpg" width="800" height="526" alt="Wiggle Wiggle">'
doc.css('a').each do |link|
puts link.attr(:href)
end
You should really use a proper HTML parser if you're trying to parse HTML.
For example, this is trivial in Nokogiri:
require 'nokogiri'
bbcode = %Q[<img src="https://farm3.staticflickr.com/2864/92917419471_248187_c.jpg" width="800" height="526" alt="Wiggle Wiggle">]
Nokogiri::HTML(bbcode).css('a')[0]['href']
# => "http://www.flickr.com/photos/user/9049969465/"
You'll obviously have to add some error checking in there, but that's the basics.
require 'nokogiri'
doc = Nokogiri::HTML (<<-eol)
<img src="https://farm3.staticflickr.com/2864/92917419471_248187_c.jpg" width="800" height="526" alt="Wiggle Wiggle">
eol
doc.at_css("a")['href']
# => "http://www.flickr.com/photos/user/9049969465/"
doc.at("a")['href']
# => "http://www.flickr.com/photos/user/9049969465/"
So I am using this:
Net::HTTP.get(URI.parse(url))
Works perfect.
Issue I am having is that the page it gets is formatted with head, html, body, etc tags.
There is a label element in the body with an id of "Result" I only want to get me back the text of "Result". Not all the html formatting.
Can this be done?
Well, to get only a part of a content in HTML you have to use a HTML parser, which will be Nokogiri in this case .
doc = Nokogiri::HTML(open(url))
doc.css('#Result').each do |re|
puts re.to_s
#puts re.content
end
I'm way new to working with XML but just had a need dropped in my lap. I have been given an usual (to me) XML format. There are colons within the tags.
<THING1:things type="Container">
<PART1:Id type="Property">1234</PART1:Id>
<PART1:Name type="Property">The Name</PART1:Name>
</THING1:things>
It is a large file and there is much more to it than this but I hope this format will be familiar to someone. Does anyone know a way to approach an XML document of this sort?
I'd rather not just write a brute-force way of parsing the text but I can't seem to make any headway with REXML or Hpricot and I suspect it is due to these unusual tags.
my ruby code:
require 'hpricot'
xml = File.open( "myfile.xml" )
doc = Hpricot::XML( xml )
(doc/:things).each do |thg|
[ 'Id', 'Name' ].each do |el|
puts "#{el}: #{thg.at(el).innerHTML}"
end
end
...which is just lifted from: http://railstips.org/blog/archives/2006/12/09/parsing-xml-with-hpricot/
And I figured I would be able to figure some stuff out from here but this code returns nothing. It doens't error. It just returns.
As #pguardiario mentioned, Nokogiri is the de facto XML and HTML parsing library. If you wanted to print out the Id and Name values in your example, here is how you would do it:
require 'nokogiri'
xml_str = <<EOF
<THING1:things type="Container">
<PART1:Id type="Property">1234</PART1:Id>
<PART1:Name type="Property">The Name</PART1:Name>
</THING1:things>
EOF
doc = Nokogiri::XML(xml_str)
thing = doc.at_xpath('//things')
puts "ID = " + thing.at_xpath('//Id').content
puts "Name = " + thing.at_xpath('//Name').content
A few notes:
at_xpath is for matching one thing. If you know you have multiple items, you want to use xpath instead.
Depending on your document, namespaces can be problematic, so calling doc.remove_namespaces! can help (see this answer for a brief discussion).
You can use the css methods instead of xpath if you're more comfortable with those.
Definitely play around with this in irb or pry to investigate methods.
Resources
Parsing an HTML/XML document
Getting started with Nokogiri
Update
To handle multiple items, you need a root element, and you need to remove the // in the xpath query.
require 'nokogiri'
xml_str = <<EOF
<root>
<THING1:things type="Container">
<PART1:Id type="Property">1234</PART1:Id>
<PART1:Name type="Property">The Name1</PART1:Name>
</THING1:things>
<THING2:things type="Container">
<PART2:Id type="Property">2234</PART2:Id>
<PART2:Name type="Property">The Name2</PART2:Name>
</THING2:things>
</root>
EOF
doc = Nokogiri::XML(xml_str)
doc.xpath('//things').each do |thing|
puts "ID = " + thing.at_xpath('Id').content
puts "Name = " + thing.at_xpath('Name').content
end
This will give you:
Id = 1234
Name = The Name1
ID = 2234
Name = The Name2
If you are more familiar with CSS selectors, you can use this nearly identical bit of code:
doc.css('things').each do |thing|
puts "ID = " + thing.at_css('Id').content
puts "Name = " + thing.at_css('Name').content
end
If in a Rails environment, the Hash object is extended and one can take advantage of the the method from_xml:
xml = File.open("myfile.xml")
data = Hash.from_xml(xml)
I have some HTML that looks like:
<dt>
Hello
(2009)
</dt>
I already have all my HTML loaded into a variable called record. I need to parse out the year i.e. 2009 if it exists.
How can I get the text inside the dt tag but not the text inside the a tag? I've used record.search("dt").inner_text and this gives me everything.
It's a trivial question but I haven't managed to figure this out.
To get all the direct children with text, but not any further sub-children, you can use XPath like so:
doc.xpath('//dt/text()')
Or if you wish to use search:
doc.search('dt').xpath('text()')
Using XPath to select exactly what you want (as suggested by #Casper) is the right answer.
def own_text(node)
# Find the content of all child text nodes and join them together
node.xpath('text()').text
end
Here's an alternative, fun answer :)
def own_text(node)
node.clone(1).tap{ |copy| copy.element_children.remove }.text
end
Seen in action:
require 'nokogiri'
root = Nokogiri.XML('<r>hi <a>BOO</a> there</r>').root
puts root.text #=> hi BOO there
puts own_text(root) #=> hi there
The dt element has two children, so you can access it by:
doc.search("dt").children.last.text
So far I have been able to stitch this together :)
begin
open("http://www.somemain.com/" + path + "/" + blah)
rescue OpenURI::HTTPError
#failure += painting.permalink
else
#success += painting.permalink
end
But how do I read the output of the service that I would be calling?
Open-URI extends open, so you'll get a type of IO stream returned:
open('http://www.example.com') #=> #<StringIO:0x00000100977420>
You have to read that to get content:
open('http://www.example.com').read[0 .. 10] #=> "<!DOCTYPE h"
A lot of times a method will let you pass different types as a parameter. They check to see what it is and either use the contents directly, in the case of a string, or read the handle if it's a stream.
For HTML and XML, such as RSS feeds, we'll typically pass the handle to a parser and let it grab the content, parse it, and return an object suitable for searching further:
require 'nokogiri'
doc = Nokogiri::HTML(open('http://www.example.com'))
doc.class #=> Nokogiri::HTML::Document
doc.to_html[0 .. 10] #=> "<!DOCTYPE h"
doc.at('h1').text #=> "Example Domains"
doc = open("http://etc..")
content = doc.read
More often people want to be able to parse the returned document, for this use something like hpricot or nokogiri
I'm not sure if you want to do this yourself for the hell of it or not but if you don't.. Mecanize is a really nice gem for doing this.
It will visit the page you want and automatically wrap the page with nokogiri so that you can access it's elements with css selectors such as "div#header h1". Ryan Bates has a video tutorial on it which will teach you everything you need to know to use it.
Basically you can just
require 'rubygems'
require 'mechanize'
agent = Mechanize.new
agent.get("http://www.google.com")
agent.page.at("some css selector").text
It's that simple.