I have an xml file which includes the nested elements below:
<SourceDetails>
<Origin>Origin</Origin>
<Identifier>Identifier</Identifier>
<Version>0</Version>
</SourceDetails>
I have already used the function at_xpath to extract the above xml snippet from an xml file which has been stored in a variable. Is it possible to iterate through this variable and store the contents of nested xml elements using Ruby Nokogiri? If so, how is this done?
I would like to append each element within SourceDetails to another variable followed by a forward slash. For the above example, I would like to get the content in the format Origin/Identifier/0
There is an easy way
require "nokogiri"
xmlFileData = Nokogiri::XML(File.open('./xmlFile.xml'))
dataArr = xmlFileData.at_xpath("//SourceDetails").text.split("\n")
dataArr.delete_at(0)
puts dataArr.join("/").gsub(/(\s+)/, '')
Here's a quick and dirty one. Since I'm not sure how you're storing your variable containing the XML, to be sure I'm getting the actual XML data I actually read the the XML data from a file, which gives us:
require 'nokogiri'
xml = File.open('source_of_xml.xml') { |f| Nokogiri::XML(f) }
values = []
xml.xpath('SourceDetails').each do |elem|
values << elem.text.gsub(/\n/, "").split
end
p values.first.join("/") #assing this to variable you want.
# => "Origin/Identifier/0"
Does this help or guide you in anyway?
Related
I have an array of objects, where the objects are instances of a class. I would like to save this array into a file in such format that I could read the file back to an array and the objects and its' instance variable values would be as they were before saving. Does someone know how this could be achieved?
The class instance objects that I would like to save to a file are fairly complex containing tens of instance variables that are often other class instance variables themselves.
WHAT I HAVE TRIED:
According to this post I tried the following:
TRIAL1:
Save file:
require 'pp'
$stdout = File.open('path/to/file.txt', 'w')
pp myArray
Load file:
require 'rubygems'
require 'json'
buffer = File.open('path/to/file.txt', 'r').read
myArray = JSON.parse(buffer)
but I got a JSON::ParserError
TRIAL2:
Save file
serialized_array = Marshal.dump(myArray)
File.open('./myArray.txt', 'w') {|f| f.write(serialized_array) }
received Encoding::UndefinedConversionError
TRIAL1 doesn't work because pp "prints arguments in pretty form" and that's not necessarily JSON.
TRIAL2 probably isn't working because Marshal produces binary data (not text) and you're not working with your file in binary mode, that could lead to encoding and EOL problems. Besides, Marshal isn't a great format for persistence since the format is tied to the version of Ruby you're using.
A modification of TRIAL1 to write JSON is probably the best solution these days:
require 'json'
File.open('path/to/file.json', 'w') { |f| JSON.dump(myArray, f) }
Finally managed to find a solution that worked!
dump = Marshal.dump(myArray)
File.write('./myarray', myArray, mode: 'r+b')
dump = File.read('./myarray')
user = Marshal.restore(dump)
Marshall was able to do the trick after changing the encoding to binary mode
I have an XML file, and before I process it I need to make sure that a certain element exists and is not blank.
Here is the code I have:
CSV.open("#{csv_dir}/products.csv","w",{:force_quotes => true}) do |out|
out << headers
Dir.glob("#{xml_dir}/*.xml").each do |xml_file|
gdsn_doc = GDSNDoc.new(xml_file)
logger.info("Processing xml file #{xml_file}")
:x
#desc_exists = #gdsn_doc.xpath("//productData/description")
if !#desc_exists.empty?
row = []
headers.each do |col|
row << product[col]
end
out << row
end
end
end
The following code is not working to find the "description" element and to check whether it is blank or not:
#desc_exists = #gdsn_doc.xpath("//productData/description")
if !#desc_exists.empty?
Here is a sample of the XML file:
<productData>
<description>Chocolate biscuits </description>
<productData>
This is how I have defined the class and Nokogiri:
class GDSNDoc
def initialize(xml_file)
#doc = File.open(xml_file) {|f| Nokogiri::XML(f)}
#doc.remove_namespaces!
The code had to be moved up to an earlier stage, where Nokogiri was initialised. It doesn't get runtime errors, but it does let XML files with blank descriptions get through and it shouldn't.
class GDSNDoc
def initialize(xml_file)
#doc = File.open(xml_file) {|f| Nokogiri::XML(f)}
#doc.remove_namespaces!
desc_exists = #doc.xpath("//productData/descriptions")
if !desc_exists.empty?
You are creating your instance like this:
gdsn_doc = GDSNDoc.new(xml_file)
then use it like this:
#desc_exists = #gdsn_doc.xpath("//productData/description")
#gdsn_doc and gdsn_doc are two different things in Ruby - try just using the version without the #:
#desc_exists = gdsn_doc.xpath("//productData/description")
The basic test is to use:
require 'nokogiri'
doc = Nokogiri::XML(<<EOT)
<productData>
<description>Chocolate biscuits </description>
<productData>
EOT
# using XPath selectors...
doc.xpath('//productData/description').to_html # => "<description>Chocolate biscuits </description>"
doc.xpath('//description').to_html # => "<description>Chocolate biscuits </description>"
xpath works fine when the document is parsed correctly.
I get an error "undefined method 'xpath' for nil:NilClass (NoMethodError)
Usually this means you didn't parse the document correctly. In your case it's because you're not using the right variable:
gdsn_doc = GDSNDoc.new(xml_file)
...
#desc_exists = #gdsn_doc.xpath("//productData/description")
Note that gdsn_doc is not the same as #gdsn_doc. The later doesn't appear to have been initialized.
#doc = File.open(xml_file) {|f| Nokogiri::XML(f)}
While that should work, it's idiomatic to write it as:
#doc = Nokogiri::XML(File.read(xml_file))
File.open(...) do ... end is preferred if you're processing inside the block and want Ruby to automatically close the file. That isn't necessary when you're simply reading then passing the content to something else for processing, hence the use of File.read(...) which slurps the file. (Slurping isn't necessary a good practice because it can have scalability problems, but for reasonable sized XML/HTML it's OK because it's easier to use DOM-based parsing than SAX.)
If Nokogiri doesn't raise an exception it was able to parse the content, however that still doesn't mean the content was valid. It's a good idea to check
#doc.errors
to see whether Nokogiri/libXML had to do some fix-ups on the content just to be able to parse it. Fixing the markup can change the DOM from what you expect, making it impossible to find a tag based on your assumptions for the selector. You could use xmllint or one of the XML validators to check, but Nokogiri will still have to be happy.
Nokogiri includes a command-line version nokogiri that accepts a URL to the document you want to parse:
nokogiri http://example.com
It'll open IRB with the content loaded and ready for you to poke at it. It's very convenient when debugging and testing. It's also a decent way to make sure the content actually exists if you're dealing with HTML containing DHTML that loads parts of the page dynamically.
I have an XML document:
<cred>
<login>Tove</login>
<pass>Jani</pass>
</cred>
My code is:
require 'nokogiri'
require 'selwet'
context "parse xml" do doc = Nokogiri::XML(File.open("test.xml"))
doc.xpath("cred/login").each do
|char_element|
puts char_element.text
end
should "check" do
Unit.go_to "http://www.ya.ru/"
Unit.click '.b-inline'
Unit.fill '[name="login"]', #login
end
When I run my test I get:
Tove
0
But I want to insert the parse result to #login. How can I get variables with the parsing result? Do I need to insert the login and pass values from the XML to fields in the web page?
You can get value of login from your XML with
#login = doc.xpath('//cred/login').text
I'd use something like this to get the values:
require 'nokogiri'
doc = Nokogiri::XML(<<EOT)
<cred>
<login>Tove</login>
<pass>Jani</pass>
</cred>
EOT
login = doc.at('login').text # => "Tove"
pass = doc.at('pass').text # => "Jani"
Nokogiri makes it really easy to access values using CSS, so use it for readability when possible. The same thing can be done using XPath:
login = doc.at('//login').text # => "Tove"
pass = doc.at('//pass').text # => "Jani"
but having to add // twice to accomplish the same thing is usually wasted effort.
The important part is at, which returns the first occurrence of the target. at allows us to use either CSS or XPath, but CSS is usually less visually noisy.
I'm trying to parse this website using Ruby and Nokogiri:
Here's my code:
require 'nokogiri'
require 'open-uri'
class StreamsController < ApplicationController
def index
end
def updateall
doc = Nokogiri::HTML(open('http://www.own3d.tv/game/League+of+Legends'))
# Grab all the Live streams from the front page.
doc.css('div#top_live .VIDEOS-1grid-box').each do |stream|
s = Stream.new
# Parse the URL.
s.url = stream.css('a.small-tn')['href']
end
end
end
On the # Parse the URL bit, I get the error Cannot convert String to Integer.
I'm kind of confused on how to use Nokogiri for this simple use case.
How can I get the href attribute of each link inside each |stream| object?
The problem is that stream.css returns a NodeSet (like an array) of matches, hence the string can't be converted to an array index.
To get the first match, use stream.at_css, which is what I think you want.
stream.css('a.small-tn') will return a collection of nodes. So calling ['href'] on the collection isn't going to work since the collection acts as an array, and it thinks you're trying to access an element at a certain index (hence the error). Rather, you need to decide if you want to iterate through them, or just grab the first:
s.url = stream.css('a.small-tn').first['href']
If you want to make this a bit more safe, you can check for nils:
node = stream.css('a.small-tn').first
s.url = node['href'] if node
Or you can use the at_css helper (as #AJcodez) pointed out, which does the same thing.
The XML file I am trying to parse has all the data contained in attributes. I found how to build the string to insert into the text file.
I have this XML file:
<ig:prescribed_item class_ref="0161-1#01-765557#1">
<ig:prescribed_property property_ref="0161-1#02-016058#1" is_required="false" combination_allowed="false" one_of_allowed="false">
<dt:measure_number_type representation_ref="0161-1#04-000005#1">
<dt:real_type>
<dt:real_format pattern="\d(1,)\.\d(1,)"/>
</dt:real_type>
<dt:prescribed_unit_of_measure UOM_ref="0161-1#05-003260#1"/>
</dt:measure_number_type>
</ig:prescribed_property>
<ig:prescribed_property property_ref="0161-1#02-016059#1" is_required="false" combination_allowed="false" one_of_allowed="false">
<dt:measure_number_type representation_ref="0161-1#04-000005#1">
<dt:real_type>
<dt:real_format pattern="\d(1,)\.\d(1,)"/>
</dt:real_type>
<dt:prescribed_unit_of_measure UOM_ref="0161-1#05-003260#1"/>
</dt:measure_number_type>
</ig:prescribed_property>
</ig:prescribed_item>
</ig:identification_guide>
And I want to parse it into a text file like this with the class ref duplicated for each property:
class_ref|property_ref|is_required|UOM_ref
0161-1#01-765557#1|0161-1#02-016058#1|false|0161-1#05-003260#1
0161-1#01-765557#1|0161-1#02-016059#1|false|0161-1#05-003260#1
This is the code I have so far:
require 'nokogiri'
doc = Nokogiri::XML(File.open("file.xml"), 'UTF-8') do |config|
config.strict
end
content = doc.xpath("//ig:prescribed_item/#class_ref").map {|i|
i.search("//ig:prescribed_item/ig:prescribed_property/#property_ref").map { |d| d.text }
}
puts content.inspect
content.each do |c|
puts c.join('|')
end
I'd simplify it a bit using CSS accessors:
xml = <<EOT
<ig:prescribed_item class_ref="0161-1#01-765557#1">
<ig:prescribed_property property_ref="0161-1#02-016058#1" is_required="false" combination_allowed="false" one_of_allowed="false">
<dt:measure_number_type representation_ref="0161-1#04-000005#1">
<dt:real_type>
<dt:real_format pattern="\d(1,)\.\d(1,)"/>
</dt:real_type>
<dt:prescribed_unit_of_measure UOM_ref="0161-1#05-003260#1"/>
</dt:measure_number_type>
</ig:prescribed_property>
<ig:prescribed_property property_ref="0161-1#02-016059#1" is_required="false" combination_allowed="false" one_of_allowed="false">
<dt:measure_number_type representation_ref="0161-1#04-000005#1">
<dt:real_type>
<dt:real_format pattern="\d(1,)\.\d(1,)"/>
</dt:real_type>
<dt:prescribed_unit_of_measure UOM_ref="0161-1#05-003260#1"/>
</dt:measure_number_type>
</ig:prescribed_property>
</ig:prescribed_item>
</ig:identification_guide>
EOT
require 'nokogiri'
doc = Nokogiri::XML(xml)
data = [ %w[ class_ref property_ref is_required UOM_ref] ]
doc.css('|prescribed_item').each do |pi|
pi.css('|prescribed_property').each do |pp|
data << [
pi['class_ref'],
pp['property_ref'],
pp['is_required'],
pp.at_css('|prescribed_unit_of_measure')['UOM_ref']
]
end
end
puts data.map{ |row| row.join('|') }
Which outputs:
class_ref|property_ref|is_required|UOM_ref
0161-1#01-765557#1|0161-1#02-016058#1|false|0161-1#05-003260#1
0161-1#01-765557#1|0161-1#02-016059#1|false|0161-1#05-003260#1
Could you explain this line in greater detail "pp.at_css('|prescribed_unit_of_measure')['UOM_ref']"
In Nokogiri, there are two types of "find a node" methods: The "search" methods return all nodes that match a particular accessor as a NodeSet, and the "at" methods return the first Node of the NodeSet which will be the first encountered Node that matched the accessor.
The "search" methods are things like search, css, xpath and /. The "at" methods are things like at, at_css, at_xpath and %. Both search and at accept either XPath or CSS accessors.
Back to pp.at_css('|prescribed_unit_of_measure')['UOM_ref']: At that point in the code pp is a local variable containing a "prescribed_property" Node. So, I'm telling the code to find the first node under pp that matches the CSS |prescribed_unit_of_measure accessor, in other words the first <dt:prescribed_unit_of_measure> tag contained by the pp node. When Nokogiri finds that node, it returns the value of the UOM_ref attribute of the node.
As a FYI, the / and % operators are aliased to search and at respectively in Nokogiri. They're part of its "Hpricot" compatability; We used to use them a lot when Hpricot was the XML/HTML parser of choice, but they're not idiomatic for most Nokogiri developers. I suspect it's to avoid confusion with the regular use of the operators, at least it is in my case.
Also, Nokogiri's CSS accessors have some extra-special juiciness; They support namespaces, like the XPath accessors do, only they use |. Nokogiri will let us ignore the namespaces, which is what I did. You'll want to nose around in the Nokogiri docs for CSS and namespaces for more information.
There are definitely ways of parsing based on attributes.
The Engine yard article "Getting started with Nokogiri" has a full description.
But quickly, the examples they give are:
To match “h3″ tags that have a class
attribute, we write:
h3[#class]
To match “h3″ tags whose class
attribute is equal to the string “r”,
we write:
h3[#class = "r"]
Using the attribute matching
construct, we can modify our previous
query to:
//h3[#class = "r"]/a[#class = "l"]