Should REXML ignore identation or whitespacing?
I am debugging an issue with a simple HTML to Markdown convertor. For some reason it fails on
<blockquote><p>foo</p></blockquote>
But not on
<blockquote>
<p>foo</p>
</blockquote>
The reason is, that in the first case, type.children.first.value is not set, in the latter case it is.
The original code can be found at link above, but a condensed snipped to show the problem is below:
require 'rexml/document'
include REXML
def parse_string(string)
doc = Document.new("<root>\n"+string+"\n</root>")
root = doc.root
root.elements.each do |element|
parse_element(element, :root)
end
end
def parse_element(element, parent)
#output = ''
# ...
#output << opening(element, parent)
#...
end
def opening(type, parent)
case type.name.to_sym
#...
when :blockquote
# remove leading newline
type.children.first.value = ""
"> "
end
end
#Parses just fine
puts parse_string("<blockquote>\n<p>foo</p>\n</blockquote>")
# Fails with undefined method `value=' for <p> ... </>:REXML::Element (NoMethodError)
puts parse_string("<blockquote><p>foo</p></blockquote>")
I am quite certain, this is due to some parameter that makes REXML require whitespacing and identation: why else would it parse the first XML different from the latter?
Can I force REXML to parse both the same? Or am I looking at a whole different kind of bug?
Try passing the option :ignore_whitespace_nodes=>:all to Document.new().
Related
Is there a matcher for comparing REXML elements for logical equality in RSpec? I tried writing a custom matcher that converts them to formatted strings, but it fails if the attribute order is different. (As noted in the XML spec, the order of attributes should not be significant.)
I could grind through writing a custom matcher that compares the name, namespace, child nodes, attributes, etc., etc., but this seems time-consuming and error-prone, and if someone else has already done it I'd rather not reinvent the wheel.
I ended up using the equivalent-xml gem and writing an RSpec custom matcher to convert the REXML to Nokogiri, compare with equivalent-xml, and pretty-print the result if needed.
The test assertion is pretty simple:
expect(actual).to be_xml(expected)
or
expect(actual).to be_xml(expected, path)
if you want to display the file path or some sort of identifier (e.g. if you're comparing a lot of documents).
The match code is a little fancier than it needs to be because it handles REXML, Nokogiri, and strings.
module XMLMatchUtils
def self.to_nokogiri(xml)
return nil unless xml
case xml
when Nokogiri::XML::Element
xml
when Nokogiri::XML::Document
xml.root
when String
to_nokogiri(Nokogiri::XML(xml, &:noblanks))
when REXML::Element
to_nokogiri(xml.to_s)
else
raise "be_xml() expected XML, got #{xml.class}"
end
end
def self.to_pretty(nokogiri)
return nil unless nokogiri
out = StringIO.new
save_options = Nokogiri::XML::Node::SaveOptions::FORMAT | Nokogiri::XML::Node::SaveOptions::NO_DECLARATION
nokogiri.write_xml_to(out, encoding: 'UTF-8', indent: 2, save_with: save_options)
out.string
end
def self.equivalent?(expected, actual, filename = nil)
expected_xml = to_nokogiri(expected) || raise("expected value #{expected || 'nil'} does not appear to be XML#{" in #{filename}" if filename}")
actual_xml = to_nokogiri(actual)
EquivalentXml.equivalent?(expected_xml, actual_xml, element_order: false, normalize_whitespace: true)
end
def self.failure_message(expected, actual, filename = nil)
expected_string = to_pretty(to_nokogiri(expected))
actual_string = to_pretty(to_nokogiri(actual)) || actual
# Uncomment this to dump expected/actual to file for manual diffing
#
# now = Time.now.to_i
# FileUtils.mkdir('tmp') unless File.directory?('tmp')
# File.open("tmp/#{now}-expected.xml", 'w') { |f| f.write(expected_string) }
# File.open("tmp/#{now}-actual.xml", 'w') { |f| f.write(actual_string) }
diff = Diffy::Diff.new(expected_string, actual_string).to_s(:text)
"expected XML differs from actual#{" in #{filename}" if filename}:\n#{diff}"
end
def self.to_xml_string(actual)
to_pretty(to_nokogiri(actual))
end
def self.failure_message_when_negated(actual, filename = nil)
"expected not to get XML#{" in #{filename}" if filename}:\n\t#{to_xml_string(actual) || 'nil'}"
end
end
The actual matcher is fairly straightforward:
RSpec::Matchers.define :be_xml do |expected, filename = nil|
match do |actual|
XMLMatchUtils.equivalent?(expected, actual, filename)
end
failure_message do |actual|
XMLMatchUtils.failure_message(expected, actual, filename)
end
failure_message_when_negated do |actual|
XMLMatchUtils.failure_message_when_negated(actual, filename)
end
end
Apparently Nokogiri's add_class method only works on NodeLists, making this code invalid:
doc.search('a').each do |anchor|
anchor.inner_text = "hello!"
anchor.add_class("whatever") # WHOOPS!
end
What can I do to make this code work? I figured it'd be something like
doc.search('a').each do |anchor|
anchor.inner_text = "hello!"
Nokogiri::XML::NodeSet.new(anchor).add_class("whatever")
end
but this doesn't work either. Please tell me I don't have to implement my own add_class for single nodes!
A CSS class is just another attribute on an element:
doc.search('a').each do |anchor|
anchor.inner_text = "hello!"
anchor['class']="whatever"
end
Since CSS classes are space-delimited in the attribute, if you're not sure if one or more classes might already exist you'll need something like
anchor['class'] ||= ""
anchor['class'] = anchor['class'] << " whatever"
You need to explicitly set the attribute using = instead of just mutating the string returned for the attribute. This, for example, will not change the DOM:
anchor['class'] ||= ""
anchor['class'] << " whatever"
Even though it results in more work being done, I'd probably do this like so:
class Nokogiri::XML::Node
def add_css_class( *classes )
existing = (self['class'] || "").split(/\s+/)
self['class'] = existing.concat(classes).uniq.join(" ")
end
end
If you don't want to monkey-patch the class, you could alternatively:
module ClassMutator
def add_css_class( *classes )
existing = (self['class'] || "").split(/\s+/)
self['class'] = existing.concat(classes).uniq.join(" ")
end
end
anchor.extend ClassMutator
anchor.add_css_class "whatever"
Edit: You can see that this is basically what Nokogiri does internally for the add_class method you found by clicking on the class to view the source:
# File lib/nokogiri/xml/node_set.rb, line 136
def add_class name
each do |el|
next unless el.respond_to? :get_attribute
classes = el.get_attribute('class').to_s.split(" ")
el.set_attribute('class', classes.push(name).uniq.join(" "))
end
self
end
Nokogiri's add_class, works on a NodeSet, like you found. Trying to add the class inside the each block wouldn't work though, because at that point you are working on an individual node.
Instead:
require 'nokogiri'
html = '<p>one</p><p>two</p>'
doc = Nokogiri::HTML(html)
doc.search('p').tap{ |ns| ns.add_class('boo') }.each do |n|
puts n.text
end
puts doc.to_html
Which outputs:
# >> one
# >> two
# >> <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
# >> <html><body>
# >> <p class="boo">one</p>
# >> <p class="boo">two</p>
# >> </body></html>
The tap method, implemented in Ruby 1.9+, gives access to the nodelist itself, allowing the add_class method to add the "boo" class to the <p> tags.
Old thread, but it's the top Google hit. You can now do this with the append_class method without having to mess with space-delimiters:
doc.search('a').each do |anchor|
anchor.inner_text = "hello!"
anchor.append_class('whatever')
end
I have a simple but huge xml file like below. I want to parse it using SAX and only print out text between the title tag.
<root>
<site>some site</site>
<title>good title</title>
</root>
I have the following code:
require 'rubygems'
require 'nokogiri'
include Nokogiri
class PostCallbacks < XML::SAX::Document
def start_element(element, attributes)
if element == 'title'
puts "found title"
end
end
def characters(text)
puts text
end
end
parser = XML::SAX::Parser.new(PostCallbacks.new)
parser.parse_file("myfile.xml")
problem is that it prints text between all the tags. How can I just print text between the title tag?
You just need to keep track of when you're inside a <title> so that characters knows when it should pay attention. Something like this (untested code) perhaps:
class PostCallbacks < XML::SAX::Document
def initialize
#in_title = false
end
def start_element(element, attributes)
if element == 'title'
puts "found title"
#in_title = true
end
end
def end_element(element)
# Doesn't really matter what element we're closing unless there is nesting,
# then you'd want "#in_title = false if element == 'title'"
#in_title = false
end
def characters(text)
puts text if #in_title
end
end
The accepted answer above is correct, however it has a drawback that it will go through the whole XML file even if it finds <title> right at the beginning.
I did have similar needs and I ended up writing a saxy ruby gem that is aimed to be efficient in such situations. Under the hood it implements Nokogiri's SAX Api.
Here's how you'd use it:
require 'saxy'
title = Saxy.parse(path_to_your_file, 'title').first
It will stop right when it finds first occurrence of <title> tag.
I've got a complex XML file, and I want to extract a content of a specific tag from it.
I use a ruby script with XmlSimple gem. I retrieve an XML file with HTTP request, then strip all the unnecessary tags and pull out necessary info. That's the script itself:
data = XmlSimple.xml_in(response.body)
hash_1 = Hash[*data['results']]
def find_value(hash, value)
hash.each do |key, val|
if val[0].kind_of? Hash then
find_value(val[0], value)
else
if key.to_s.eql? value
puts val
end
end
end
end
hash_1['book'].each do |arg|
find_value(arg, "title")
puts("\n")
end
The problem is, that when I change replace puts val with return val, and then call find_value method with puts find_value (arg, "title"), i get the whole contents of hash_1[book] on the screen.
How to correct the find_value method?
A "complex XML file" and XmlSimple don't mix. Your task would be solved a lot easier with Nokogiri, and be faster as well:
require 'nokogiri'
doc = Nokogiri::XML(response.body)
puts doc.xpath('//book/title/text()')
I am parsing a text file and want to be able to extend the sets of tokens that can be recognized easily. Currently I have the following:
if line =~ /!DOCTYPE/
puts "token doctype " + line[0,20]
#ast[:doctype] << line
elsif line =~ /<html/
puts "token main HTML start " + line[0,20]
html_scanner_off = false
elsif line =~ /<head/ and not html_scanner_off
puts "token HTML header starts " + line[0,20]
html_header_scanner_on = true
elsif line =~ /<title/
puts "token HTML title " + line[0,20]
#ast[:HTML_header_title] << line
end
Is there a way to write this with a yield block, e.g. something like:
scanLine("title", :HTML_header_title, line)
?
Don't parse HTML with regexes.
That aside, there are several ways to do what you're talking about. One:
class Parser
class Token
attr_reader :name, :pattern, :block
def initialize(name, pattern, block)
#name = name
#pattern = pattern
#block = block
end
def process(line)
#block.call(self, line)
end
end
def initialize
#tokens = []
end
def scanLine(line)
#tokens.find {|t| line =~ t.pattern}.process(line)
end
def addToken(name, pattern, &block)
#tokens << Token.new(name, pattern, block)
end
end
p = Parser.new
p.addToken("title", /<title/) {|token, line| puts "token #{token.name}: #{line}"}
p.scanLine('<title>This is the title</title>')
This has some limitations (like not checking for duplicate tokens), but works:
$ ruby parser.rb
token title: <title>This is the title</title>
$
If you're intending to parse HTML content, you might want to use one of the HTML parsers like nokogiri (http://nokogiri.org/) or Hpricot (http://hpricot.com/) which are really high-quality. A roll-your-own approach will probably take longer to perfect than figuring out how to use one of these parsers.
On the other hand, if you're dealing with something that's not quite HTML, and can't be parsed that way, then you'll need to roll your own somehow. There's a few Ruby parser frameworks out there that may help, but for simple tasks where performance isn't a critical factor, you can get by with a pile of regexps like you have here.