Probably a pretty easy question:
I'm using Mechanize, Nokogori, and Xpath to parse through some html as such:
category = a.page.at("//li//a[text()='Test']")
Now, I want the term that I'm searching for in text()= to be dynamic...i.e. I want to create a local variable:
term = 'Test'
and embed that local ruby variable in the Xpath, if that makes sense.
Any ideas how?
My intuition was to treat this like string concatenation, but that doesn't work out:
term = 'Test'
category = a.page.at("//li//a[text()=" + term + "]")
When you use category = a.page.at("//li//a[text()=" + term + "]"). The final result to method is //li//a[text()=Test] where test is not in quotes. So to put quotes around string you need to use escape character \.
term = 'Test'
category = a.page.at("//li//a[text()=\"#{term}\"]")
or
category = a.page.at("//li//a[text()='" + term + "']")
or
category = a.page.at("//li//a[text()='#{term}']")
For example:
>> a="In quotes" #=> "In quotes"
>> puts "This string is \"#{a}\"" #=> This string is "In quotes"
>> puts "This string is '#{a}'" #=> This string is 'In quotes'
>> puts "This string is '"+a+"'" #=> This string is 'In quotes'
A little-used feature that might be relevant to your question is Nokogiri's ability to call a ruby callback while evaluating an XPath expression.
You can read more about this feature at http://nokogiri.org under the method docs for Node#xpath (http://nokogiri.org/Nokogiri/XML/Node.html#method-i-xpath), but here's an example addressing your question:
#! /usr/bin/env ruby
require 'nokogiri'
xml = <<-EOXML
<root>
<a n='1'>foo</a>
<a n='2'>bar</a>
<a n='3'>baz</a>
</root>
EOXML
doc = Nokogiri::XML xml
dynamic_query = Class.new do
def text_matching node_set, string
node_set.select { |node| node.inner_text == string }
end
end
puts doc.at_xpath("//a[text_matching(., 'bar')]", dynamic_query.new)
# => <a n="2">bar</a>
puts doc.at_xpath("//a[text_matching(., 'foo')]", dynamic_query.new)
# => <a n="1">foo</a>
HTH.
Related
While using Nokogiri::XML::Builder I need to be able to generate a node that also replaces a regex match on the text with some other XML.
Currently I'm able to add additional XML inside the node. Here's an example;
def xml
Nokogiri::XML::Builder.new do |xml|
xml.chapter {
xml.para {
xml.parent.add_child("Testing[1] footnote paragraph.")
add_footnotes(xml, 'An Entry')
}
}
end.to_xml
end
# further child nodes WILL be added to footnote
def add_footnotes(xml, text)
xml.footnote text
end
which produces;
<chapter>
<para>Testing[1] footnote paragraph.<footnote>An Entry</footnote></para>
</chapter>
But I need to be able to run a regex replace on the reference [1], replacing it with the <footnote> XML, producing output like the following;
<chapter>
<para>Testing<footnote>An Entry</footnote> footnote paragraph.</para>
</chapter>
I'm making the assumption here that the add_footnotes method would receive the reference match (e.g. as $1), which would be used to pull the appropriate footnote from a collection.
That method would also be adding additional child nodes, such as the following;
<footnote>
<para>Words.</para>
<para>More words.</para>
</footnote>
Can anyone help?
Here's a spin on your code that shows how to generate the output. You'll need to refit it to your own code....
require 'nokogiri'
FOOTNOTES = {
'1' => 'An Entry'
}
child_text = "Testing[1] footnote paragraph."
pre_footnote, footnote_id, post_footnote = /^(.+)\[(\d+)\](.+)/.match(child_text).captures
doc = Nokogiri::XML::Builder.new do |xml|
xml.chapter {
xml.para {
xml.text(pre_footnote)
xml.footnote FOOTNOTES[footnote_id]
xml.text(post_footnote)
}
}
end
puts doc.to_xml
Which outputs:
<?xml version="1.0"?>
<chapter>
<para>Testing<footnote>An Entry</footnote> footnote paragraph.</para>
</chapter>
The trick is you have to grab the text preceding and following your target so you can insert those as text nodes. Then you can figure out what needs to be added. For clarity in your code you should preprocess all the text, get your variables figured out, then fall into the XML generator. Don't try to do any calculations inside the Builder block, instead just reference variables. Think of Builder like a view in an MVC-type application if that helps.
FOOTNOTES could actually be a database lookup, a hash or some other data container.
You should also look at the << method, which lets you inject XML source, so you could pre-build the footnote XML, then loop over an array containing the various footnotes and inject them. Often it's easier to pre-process, then use gsub to treat things like [1] as placeholders. See "gsub(pattern, hash) → new_str" in the documentation, along with this example:
'hello'.gsub(/[eo]/, 'e' => 3, 'o' => '*') #=> "h3ll*"
For instance:
require 'nokogiri'
text = 'this is[1] text and[2] text'
footnotes = {
'[1]' => 'some',
'[2]' => 'more'
}
footnotes.keys.each do |k|
v = footnotes[k]
footnotes[k] = "<footnote>#{ v }</footnote>"
end
replacement_xml = text.gsub(/\[\d+\]/, footnotes) # => "this is<footnote>some</footnote> text and<footnote>more</footnote> text"
doc = Nokogiri::XML::Builder.new do |xml|
xml.chapter {
xml.para { xml.<<(replacement_xml) }
}
end
puts doc.to_xml
# >> <?xml version="1.0"?>
# >> <chapter>
# >> <para>this is<footnote>some</footnote> text and<footnote>more</footnote> text</para>
# >> </chapter>
I can try as below :
require 'nokogiri'
def xml
Nokogiri::XML::Builder.new do |xml|
xml.chapter {
xml.para {
xml.parent.add_child("Testing[1] footnote paragraph.")
add_footnotes(xml, 'add text',"[1]")
}
}
end.to_xml
end
def add_footnotes(xml, text,ref)
string = xml.parent.child.content
xml.parent.child.content = ""
string.partition(ref).each do |txt|
next xml.text(txt) if txt != ref
xml.footnote text
end
end
puts xml
# >> <?xml version="1.0"?>
# >> <chapter>
# >> <para>Testing<footnote>add text</footnote> footnote paragraph.</para>
# >> </chapter>
I'm trying to extract the first href link from a website. Just the full link alone.
I am expecting to get http://www.iana.org/domains/example as the output but instead I am getting just http://www.iana.org/domains/ex
require 'net/http'
source = Net::HTTP.get('www.example.org', '/index.html')
def findhref(page) #returns rest of the html after href
return page[page.index('href')..-1]
end
def findlink(page)
text = findhref(page)
firstquote = text.index('"') #first position of quote
secondquote = text[firstquote+1..-1].index('"') #2nd quote
puts text #for debugging
puts firstquote+1 #for debugging
puts secondquote #for debugging
return text[firstquote+1..secondquote]
end
print findlink(source)
I would suggest using Nokogiri for HTML parsing. The solution to your problem would be as simple as:
doc = Nokogiri::HTML(open('www.example.org/index.html'))
first_anchor = doc.css('a').first
first_href = first_anchor['href']
I have this code, and I need to add a regex ahead of "href=" for integers:
f = File.open("us.html")
doc = Nokogiri::HTML(f)
ans = doc.css('a[href=]')
puts doc
I tried doing:
ans = doc.css('a[href=\d]
or:
ans = doc.css('a[href="\d"])
but it doesn't work. Can anyone suggest a workaround?
If you want to use a regular expression, I believe you will have to do that manually. It cannot be done with a CSS or XPath selector.
You can do it by iterating through the elements and comparing their href attribute to your regular expression. For example:
html = %q{
<html>
<a href='1'></a>
<a href='adf'></a>
</html>
}
doc = Nokogiri::HTML(html)
ans = doc.css('a[href]').select{ |e| e['href'] =~ /\d/}
#=>
You can do it in XPath:
require 'nokogiri'
html = %q{
<html>
<a href='1'></a>
<a href='adf'></a>
</html>
}
doc = Nokogiri::HTML(html)
puts doc.xpath('//a[#href[number(.) = .]]')
#=>
The XPath function number() does a conversion to a number. If it equals the node itself, then the node is a number. It is even possible to check a range using inequality operators.
I know this is an easy question, but I want to extract one part of a string with rails.
I would do this like Java, by knowing the beginning and end character of the string and extract it, but I want to do this by ruby way, that's why I need your help.
My string is:
STACK OVER AND FLOW
And I want the numerical values between quotation marks => 99999 and the value of the link => STACK OVER AND FLOW
How should I parse this string in ruby ?
Thanks.
If you need to parse html:
> require 'nokogiri'
> str = %q[STACK OVER AND FLOW]
> doc = Nokogiri.parse(str)
> link = doc.at('a')
> link.text
=> "STACK OVER AND FLOW"
> link['href'][/(\d+)/, 1]
=> "99999"
http://nokogiri.org/
This should work if you have only one link in string
str = %{STACK OVER AND FLOW }
num = str.match(/href=".*?'(\d*)'.*?/)[1].to_i
name = str.match(/>(.*?)</)[1].strip
Way to get both at a time:
str = "STACK OVER AND FLOW "
num, name = str.scan(/launchRemote\('(\d+)'[^>]+>\s*(.*?)\s*</).first
# => ["99999", "STACK OVER AND FLOW"]
I'm way new to working with XML but just had a need dropped in my lap. I have been given an usual (to me) XML format. There are colons within the tags.
<THING1:things type="Container">
<PART1:Id type="Property">1234</PART1:Id>
<PART1:Name type="Property">The Name</PART1:Name>
</THING1:things>
It is a large file and there is much more to it than this but I hope this format will be familiar to someone. Does anyone know a way to approach an XML document of this sort?
I'd rather not just write a brute-force way of parsing the text but I can't seem to make any headway with REXML or Hpricot and I suspect it is due to these unusual tags.
my ruby code:
require 'hpricot'
xml = File.open( "myfile.xml" )
doc = Hpricot::XML( xml )
(doc/:things).each do |thg|
[ 'Id', 'Name' ].each do |el|
puts "#{el}: #{thg.at(el).innerHTML}"
end
end
...which is just lifted from: http://railstips.org/blog/archives/2006/12/09/parsing-xml-with-hpricot/
And I figured I would be able to figure some stuff out from here but this code returns nothing. It doens't error. It just returns.
As #pguardiario mentioned, Nokogiri is the de facto XML and HTML parsing library. If you wanted to print out the Id and Name values in your example, here is how you would do it:
require 'nokogiri'
xml_str = <<EOF
<THING1:things type="Container">
<PART1:Id type="Property">1234</PART1:Id>
<PART1:Name type="Property">The Name</PART1:Name>
</THING1:things>
EOF
doc = Nokogiri::XML(xml_str)
thing = doc.at_xpath('//things')
puts "ID = " + thing.at_xpath('//Id').content
puts "Name = " + thing.at_xpath('//Name').content
A few notes:
at_xpath is for matching one thing. If you know you have multiple items, you want to use xpath instead.
Depending on your document, namespaces can be problematic, so calling doc.remove_namespaces! can help (see this answer for a brief discussion).
You can use the css methods instead of xpath if you're more comfortable with those.
Definitely play around with this in irb or pry to investigate methods.
Resources
Parsing an HTML/XML document
Getting started with Nokogiri
Update
To handle multiple items, you need a root element, and you need to remove the // in the xpath query.
require 'nokogiri'
xml_str = <<EOF
<root>
<THING1:things type="Container">
<PART1:Id type="Property">1234</PART1:Id>
<PART1:Name type="Property">The Name1</PART1:Name>
</THING1:things>
<THING2:things type="Container">
<PART2:Id type="Property">2234</PART2:Id>
<PART2:Name type="Property">The Name2</PART2:Name>
</THING2:things>
</root>
EOF
doc = Nokogiri::XML(xml_str)
doc.xpath('//things').each do |thing|
puts "ID = " + thing.at_xpath('Id').content
puts "Name = " + thing.at_xpath('Name').content
end
This will give you:
Id = 1234
Name = The Name1
ID = 2234
Name = The Name2
If you are more familiar with CSS selectors, you can use this nearly identical bit of code:
doc.css('things').each do |thing|
puts "ID = " + thing.at_css('Id').content
puts "Name = " + thing.at_css('Name').content
end
If in a Rails environment, the Hash object is extended and one can take advantage of the the method from_xml:
xml = File.open("myfile.xml")
data = Hash.from_xml(xml)