I have a TMX translation memory file that I need to parse to be able to import it into a new DB. I'm using Ruby + Nokogiri. This is the TMX (xml) structure:
<body>
<tu creationdate="20181001T113609Z" creationid="some_user">
<prop type="Att::Attribute1">Value1</prop>
<prop type="Txt::Attribute2">Value2</prop>
<prop type="Txt::Attribute3">Value3</prop>
<prop type="Txt::Attribute4">Value4</prop>
<tuv xml:lang="EN-US">
<seg>Testing</seg>
</tuv>
<tuv xml:lang="SL">
<seg>Testiranje</seg>
</tuv>
</tu>
</body>
I've only included 1 TU node here for simplicity.
This is my current script:
require 'nokogiri'
doc = File.open("test_for_import.xml") { |f| Nokogiri::XML(f) }
doc.xpath('//tu').each do |x|
puts "Creation date: " + x.attributes["creationdate"]
puts "User: " + x.attributes["creationid"]
x.children.each do |y|
puts y.children
end
end
This yields the following:
Creation date: 20181001T113609Z
User: some_user
Value1
Value2
Value3
Value4
<seg>Testing</seg>
<seg>Testiranje</seg>
What I need to do get is to search for Attribute1 and it's corresponding value and assign to a variable. These will then be used as attributes when creating translation records in the new DB. I need the same for seg to get the source and the translation. I don't want to rely on the sequence, even though it should/is always the same.
What is the best way to continue? All the elements are of class Nokogiri::XML::NodeSet . Even after looking at the docs for this I'm still stuck.
Can someone help?
Best, Sebastjan
The easiest way to traverse a node tree like this is using XPath. You've already used XPath for getting your top-level tu element, but you can extend XPath queries much further to get specific elements like you're looking for.
Here on DevHints is a handy cheat-sheet for what you can do with XPath.
Relative to your x variable which points to the tu element, here are the XPaths you'll want to use:
prop[#type="Att::Attribute1"] for finding your prop for Attribute 1
//seg or tuv/seg for finding the seg elements
Here's a complete code example using those XPaths. The at_xpath method returns one result, whereas the xpath method returns all results.
require 'nokogiri'
doc = File.open("test_for_import.xml") { |f| Nokogiri::XML(f) }
doc.xpath('//tu').each do |x|
puts "Creation date: " + x.attributes["creationdate"]
puts "User: " + x.attributes["creationid"]
# Get Attribute 1
# There should only be one result for this, so using `at_xpath`
attr1 = x.at_xpath('prop[#type="Att::Attribute1"]')
puts "Attribute 1: " + attr1.text
# Get each seg
# There will be many results, so using `xpath`
segs = x.xpath('//seg')
segs.each do |seg|
puts "Seg: " + seg.text
end
end
This outputs:
Creation date: 20181001T113609Z
User: some_user
Attribute 1: Value1
Seg: Testing
Seg: Testiranje
Related
I am currently trying to create a hash out of an xml documen, with the help of the ox gem
Input xml:
<?xml version="1.0"?>
<expense>
<payee>starbucks</payee>
<amount>5.75</amount>
<date>2017-06-10</date>
</expense>
with the following ruby/ox code:
doc = Ox.parse(xml)
plist = doc.root.nodes
I get the following output:
=> [#<Ox::Element:0x00007f80d985a668 #value="payee", #attributes={}, #nodes=["starbucks"]>, #<Ox::Element:0x00007f80d9839198 #value="amount", #attributes={}, #nodes=["5.75"]>, #<Ox::Element:0x00007f80d9028788 #value="date", #attributes={}, #nodes=["2017-06-10"]>]
The output I want is a hash in the format:
{'payee' => 'Starbucks',
'amount' => 5.75,
'date' => '2017-06-10'}
to save in my sqllite database. How can I transform the objects array into a hash like above.
Any help is highly appreciated.
The docs suggest you can use the following:
require 'ox'
xml = %{
<top name="sample">
<middle name="second">
<bottom name="third">Rock bottom</bottom>
</middle>
</top>
}
puts Ox.load(xml, mode: :hash)
puts Ox.load(xml, mode: :hash_no_attrs)
#{:top=>[{:name=>"sample"}, {:middle=>[{:name=>"second"}, {:bottom=>[{:name=>"third"}, "Rock bottom"]}]}]}
#{:top=>{:middle=>{:bottom=>"Rock bottom"}}}
I'm not sure that's exactly what you're looking for though.
Otherwise, it really depends on the methods available on the Ox::Element instances in the array.
From the docs, it looks like there are two handy methods here: you can use [] and text.
Therefore, I'd use reduce to coerce the array into the hash format you're looking for, using something like the following:
ox_nodes = [#<Ox::Element:0x00007f80d985a668 #value="payee", #attributes={}, #nodes=["starbucks"]>, #<Ox::Element:0x00007f80d9839198 #value="amount", #attributes={}, #nodes=["5.75"]>, #<Ox::Element:0x00007f80d9028788 #value="date", #attributes={}, #nodes=["2017-06-10"]>]
ox_nodes.reduce({}) do |hash, node|
hash[node['#value']] = node.text
hash
end
I'm not sure whether node['#value'] will work, so you might need to experiment with that - otherwise perhaps node.instance_variable_get('#value') would do it.
node.text does the following, which sounds about right:
Returns the first String in the elements nodes array or nil if there is no String node.
N.B. I prefer to tidy the reduce block a little using tap, something like the following:
ox_nodes.reduce({}) do |hash, node|
hash.tap { |h| h[node['#value']] = node.text }
end
Hope that helps - let me know how you get on!
I found the answer to the question in my last comment by myself:
def create_xml(expense)
Ox.default_options=({:with_xml => false})
doc = Ox::Document.new(:version => '1.0')
expense.each do |key, value|
e = Ox::Element.new(key)
e << value
doc << e
end
Ox.dump(doc)
end
The next question would be how can i transform the value of the amount key from a string to an integer befopre saving it to the database
I have the following XML here:
<listing>
<seller_info>
<payment_types>Visa, Mastercard, , , , 0, Discover, American Express </payment_types>
<shipping_info>siteonly, Buyer Pays Shipping Costs </shipping_info>
<buyer_protection_info/>
<auction_info>
<bid_history>
<item_info>
</listing>
The following code works fine for displaying first child of the first //listing node:
require 'nokogiri'
require 'open-uri'
html_data = open('http://aiweb.cs.washington.edu/research/projects/xmltk/xmldata/data/auctions/321gone.xml')
nokogiri_object = Nokogiri::XML(html_data)
listing_elements = nokogiri_object.xpath("//listing")
puts listing_elements[0].children[1]
This also works:
puts listing_elements[0].children[3]
I tried to access the second node <payment_types> with the the following code:
puts listing_elements[0].children[2]
but a blank line was displayed. Looking through Firebug, it is clearly the 2nd child of the listing node. In general, only odd numbers work with the children method.
Is this a bug in Nokogiri? Any thoughts?
It's not a bug, its the space created while parsing strings that contain "\n" (or empty nodes), but you could use the noblanks option to avoid them:
nokogiri_object = Nokogiri::XML(html_data) { |conf| conf.noblanks }
Use that and you will have no blanks in your array.
The problem is you are not parsing the document correctly. children returns more than you think, and its use is painting you into a corner.
Here's a simplified example of how I'd do it:
require 'nokogiri'
doc = Nokogiri::XML(DATA.read)
auctions = doc.search('listing').map do |listing|
seller_info = listing.at('seller_info')
auction_info = listing.at('auction_info')
hash = [:seller_name, :seller_rating].each_with_object({}) do |s, h|
h[s] = seller_info.at(s.to_s).text.strip
end
[:current_bid, :time_left].each do |s|
hash[s] = auction_info.at(s.to_s).text.strip
end
hash
end
__END__
<?xml version='1.0' ?>
<!DOCTYPE root SYSTEM "http://www.cs.washington.edu/research/projects/xmltk/xmldata/data/auctions/321gone.dtd">
<root>
<listing>
<seller_info>
<seller_name>537_sb_3 </seller_name>
<seller_rating> 0</seller_rating>
</seller_info>
<auction_info>
<current_bid> $839.93</current_bid>
<time_left> 1 Day, 6 Hrs</time_left>
</auction_info>
</listing>
<listing>
<seller_info>
<seller_name> lapro8</seller_name>
<seller_rating> 0</seller_rating>
</seller_info>
<auction_info>
<current_bid> $210.00</current_bid>
<time_left> 4 Days, 21 Hrs</time_left>
</auction_info>
</listing>
</root>
After running, auctions will be:
auctions
# => [{:seller_name=>"537_sb_3",
# :seller_rating=>"0",
# :current_bid=>"$839.93",
# :time_left=>"1 Day, 6 Hrs"},
# {:seller_name=>"lapro8",
# :seller_rating=>"0",
# :current_bid=>"$210.00",
# :time_left=>"4 Days, 21 Hrs"}]
Notice there are no empty text nodes to deal with because I told Nokogiri exactly which nodes to grab text from. You should be able to extend the code to grab any information you want easily.
A typically formatted XML or HTML document that displays nesting or indentation uses text nodes to provide that indenting:
require 'nokogiri'
doc = Nokogiri::HTML(<<EOT)
<html>
<body>
<p>foo</p>
</body>
</html>
EOT
Here's what your code is seeing:
doc.at('body').children.map(&:to_html)
# => ["\n" +
# " ", "<p>foo</p>", "\n" +
# " "]
The Text nodes are what are confusing you:
doc.at('body').children.first.class # => Nokogiri::XML::Text
doc.at('body').children.first.text # => "\n "
If you don't drill down far enough you will pick up the Text nodes and have to clean up the results:
doc.at('body')
.text # => "\n foo\n "
.strip # => "foo"
Instead, explicitly find the node you want and extract the information:
doc.at('body p').text # => "foo"
In the suggested code above I used strip because the incoming XML had spaces surrounding some text:
h[s] = seller_info.at(s.to_s).text.strip
which is the result of the original XML creation code not cleaning the lines prior to generating the XML. So sometimes we have to clean up their mess, but the proper accessing of the node can reduce that a lot.
The problem is that children includes text nodes such as the whitespace between elements. If instead you use element_children you get just the child elements (i.e. the contents of the tags, not the surrounding whitespace).
I'm way new to working with XML but just had a need dropped in my lap. I have been given an usual (to me) XML format. There are colons within the tags.
<THING1:things type="Container">
<PART1:Id type="Property">1234</PART1:Id>
<PART1:Name type="Property">The Name</PART1:Name>
</THING1:things>
It is a large file and there is much more to it than this but I hope this format will be familiar to someone. Does anyone know a way to approach an XML document of this sort?
I'd rather not just write a brute-force way of parsing the text but I can't seem to make any headway with REXML or Hpricot and I suspect it is due to these unusual tags.
my ruby code:
require 'hpricot'
xml = File.open( "myfile.xml" )
doc = Hpricot::XML( xml )
(doc/:things).each do |thg|
[ 'Id', 'Name' ].each do |el|
puts "#{el}: #{thg.at(el).innerHTML}"
end
end
...which is just lifted from: http://railstips.org/blog/archives/2006/12/09/parsing-xml-with-hpricot/
And I figured I would be able to figure some stuff out from here but this code returns nothing. It doens't error. It just returns.
As #pguardiario mentioned, Nokogiri is the de facto XML and HTML parsing library. If you wanted to print out the Id and Name values in your example, here is how you would do it:
require 'nokogiri'
xml_str = <<EOF
<THING1:things type="Container">
<PART1:Id type="Property">1234</PART1:Id>
<PART1:Name type="Property">The Name</PART1:Name>
</THING1:things>
EOF
doc = Nokogiri::XML(xml_str)
thing = doc.at_xpath('//things')
puts "ID = " + thing.at_xpath('//Id').content
puts "Name = " + thing.at_xpath('//Name').content
A few notes:
at_xpath is for matching one thing. If you know you have multiple items, you want to use xpath instead.
Depending on your document, namespaces can be problematic, so calling doc.remove_namespaces! can help (see this answer for a brief discussion).
You can use the css methods instead of xpath if you're more comfortable with those.
Definitely play around with this in irb or pry to investigate methods.
Resources
Parsing an HTML/XML document
Getting started with Nokogiri
Update
To handle multiple items, you need a root element, and you need to remove the // in the xpath query.
require 'nokogiri'
xml_str = <<EOF
<root>
<THING1:things type="Container">
<PART1:Id type="Property">1234</PART1:Id>
<PART1:Name type="Property">The Name1</PART1:Name>
</THING1:things>
<THING2:things type="Container">
<PART2:Id type="Property">2234</PART2:Id>
<PART2:Name type="Property">The Name2</PART2:Name>
</THING2:things>
</root>
EOF
doc = Nokogiri::XML(xml_str)
doc.xpath('//things').each do |thing|
puts "ID = " + thing.at_xpath('Id').content
puts "Name = " + thing.at_xpath('Name').content
end
This will give you:
Id = 1234
Name = The Name1
ID = 2234
Name = The Name2
If you are more familiar with CSS selectors, you can use this nearly identical bit of code:
doc.css('things').each do |thing|
puts "ID = " + thing.at_css('Id').content
puts "Name = " + thing.at_css('Name').content
end
If in a Rails environment, the Hash object is extended and one can take advantage of the the method from_xml:
xml = File.open("myfile.xml")
data = Hash.from_xml(xml)
I have this script that is a part of a bigger one. I have tree diffrent XML files that looks a litle diffrent from each other and I need some type of control structure to handle nil-object and xpath expressions better
The script that I have right now, outputs nil objects:
require 'open-uri'
require 'rexml/document'
include REXML
#urls = Array.new()
#urls << "http://testnavet.skolverket.se/SusaNavExport/EmilObjectExporter?id=186956355&strId=info.uh.kau.KTADY1&EMILVersion=1.1"
#urls << "http://testnavet.skolverket.se/SusaNavExport/EmilObjectExporter?id=184594606&strId=info.uh.gu.GS5&EMILVersion=1.1"
#urls << "http://testnavet.skolverket.se/SusaNavExport/EmilObjectExporter?id=185978100&strId=info.uh.su.ARO720&EMILVersion=1.1"
#urls.each do |url|
doc = REXML::Document.new(open(url).read)
doc.elements.each("/educationInfo/extensionInfo/nya:textualDescription/nya:textualDescriptionPhrase | /ns:educationInfo/ns:extensionInfo/gu:guInfoExtensions/gu:guSubject/gu:descriptions/gu:description | //*[name()='ct:text']"){
|e| m = e.text
m.gsub!(/<.+?>/, "")
puts "Description: " + m
puts ""
}
end
OUTPUT:
Description: bestrykning, kalandrering, tryckning, kemiteknik
Description: Vill du jobba med internationella och globala frågor med...
Description: The study of globalisation is becoming ever more
important for our understanding of today´s world and the School of
Global Studies is a unique environment for research.
Description:
Description:
Description: Kursen behandlar identifieringen och beskrivningen av
sjukliga förändringar i mänskliga skelett. Kursen ger en
ämneshistorisk bakgrund och skelettförändringars förhållanden till
moderna kliniska data diskuteras.
See this post on how to skip over entries when using a block in ruby. The method each() on doc.elements is being called with a block (which is you code containing gsub and puts calls). The "next" keyword will let you stop executing the block for the current element and move on to the next one.
doc.elements.each("/educationInfo/extensionInfo/nya:textualDescription/nya:textualDescriptionPhrase | /ns:educationInfo/ns:extensionInfo/gu:guInfoExtensions/gu:guSubject/gu:descriptions/gu:description | //*[name()='ct:text']"){
|e| m = e.text
m.gsub!(//, "")
next if m.empty?
puts "Description: " + m
puts ""
}
We know that "m" is a string (and not nil) when using the "next" keyword because we just called gsub! on it, which did not throw an error when executing that line. That means the blank Descriptions are caused by empty strings, not nil objects.
I have this XML:
<Experiment>
<mzData version="1.05" accessionNumber="1635">
<description>
<admin>
<sampleName>Fas-induced and control Jurkat T-lymphocytes</sampleName>
<sampleDescription>
<cvParam cvLabel="MeSH" accession="D017209" name="apoptosis" />
<cvParam cvLabel="UNITY" accession="D2135" name="Jurkat cells" />
<cvParam cvLabel="MeSH" accession="D019014" name="Antigens, CD95" />
</sampleDescription>
</admin>
</description>
</mzData>
</Experiment>
</ExperimentCollection>
I also have the following code:
require 'rubygems'
require 'nokogiri'
doc = Nokogiri::XML(File.open("my.xml"))
sampleName = doc.xpath( "/ExperimentCollection/Experiment/mzData/description/admin/sampleName" ).text
sampleDescription = doc.xpath( "/ExperimentCollection/Experiment/mzData/description/admin/sampleDescription/MeSH/#accession" ).text
puts sampleName + " " + sampleDescription
foo = sampleName + " " + sampleDescription
f = File.new("my.txt","w")
f.write(foo)
f.close()
The code grabs the sampleName just fine, but not the accession letters/numbers. I only want to grab all the letters/numbers after MeSH -> accession (D017209 and D019014). What do I have to change in the doc.xpath command to make this work?
doc.xpath( "/ExperimentCollection/Experiment/mzData/description/admin/sampleDescription/MeSH/#accession" )
Returns nothing because there is no tag MeSH. You need to replace MeSH with cvParam[#cvLabel=\"MeSH\"] (read: a cvParam tag which has an attribute cvLabel with the value MeSH).
Once you fixed that xpath will return a collection of Nokogiri::XML::Attr objects. By calling text on that collection you will get back the string value of the first element. Since you want all of the elements you should instead use map(&:text) (or map {|n| n.text} in ruby 1.8.6) which will return an array containing the string value of each accession attribute (i.e. ["D017209", "D019014"] for the example XML-file).
Since you seem to be confused, here's a clarification:
#Bobby: When I said "xpath will return a collection of Nokogiri::XML::Attr objects", I meant just that. You call xpath and then xpath creates and returns a collection of Attr objects. In no way did I mean that you should manually create any Attr objects yourself.
And when I said you should use map, I just meant you should call map on the collection returned by xpath (though instead of using map you can just call puts with the collection as an argument).
So what you need to do is 1. fix your xpath like I described.
use xpath with the fixed xpath to get a collection
use puts to print it
In other words:
require 'rubygems'
require 'nokogiri'
doc = Nokogiri::XML(File.open("my.xml"))
common_prefix = "/ExperimentCollection/Experiment/mzData/description/admin"
sample_name = doc.xpath( common_prefix+"/sampleName" ).text
accessions = doc.xpath( common_prefix+
"/sampleDescription/cvParam[#cvLabel=\"MeSH\"]/#accession" )
puts sample_name
puts accessions
Here is a simple way to do it, although this is probably too clever, because you'll probably want to do other things as well:
File.open("my.txt","w") do |f|
doc.xpath('//cvParam[#cvLabel="MeSH"]').each {|n| f << "#{n['name']} #{n['accession']}\n"}
end
You may need a more selective xpath statement.