Extracting nested object with Logstash and Ruby - ruby

Question

I have this XML structure:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE us-patent-application SYSTEM "us-patent-application-v44-2014-04-03.dtd" [ ]>
<us-patent-application lang="EN" dtd-version="v4.4 2014-04-03" file="US20180000001A1-20180104.XML" status="PRODUCTION" id="us-patent-application" country="US" date-produced="20171219" date-publ="20180104">
<us-bibliographic-data-application lang="EN" country="US">
<us-parties>
<inventors>
<inventor sequence="00" designation="us-only">
<addressbook>
<last-name>Evans</last-name>
<first-name>Mike</first-name>
<address>
<city>Emerald Park</city>
<country>CA</country>
</address>
</addressbook>
</inventor>
<inventor sequence="01" designation="us-only">
<addressbook>
<last-name>Lucas</last-name>
<first-name>Lisa</first-name>
<address>
<city>Regina</city>
<country>CA</country>
</address>
</addressbook>
</inventor>
<inventor sequence="02" designation="us-only">
<addressbook>
<last-name>Smith</last-name>
<first-name>John R.</first-name>
<address>
<city>Regina</city>
<country>CA</country>
</address>
</addressbook>
</inventor>
</inventors>
</us-parties>
</us-bibliographic-data-application>
</us-patent-application>
I would like Logstash to output this structure:
{
"us-patent-application": {
"us-bibliographic-data-application": {
"us-parties": {
"inventors": [
"Mike Evans",
"Lisa Lucas",
"John R. Smith"
]
}
}
}
}
I have attempted to solve this 'combination' of names into one array in Logstash, but I can't find a working solution.
As of now, I am focused on using a ruby script in the Logstash ruby filter plugin. I am using this approach because I was not able to find a working solution using Xpath in the Logstash XML filter.
Here is the Logstash 'main.conf' configuration file:
input {
file {
path => [
"/opt/uspto/*.xml"
]
start_position => "beginning"
#use for testing
sincedb_path => "/dev/null"
# set this sincedb path when not testing
#sincedb_path => "/opt/logstash/tmp/sincedb"
exclude => "*.gz"
type => "xml"
codec => multiline {
#pattern => "<wo-ocr-published-application"
pattern => "<?xml version=\"1.0\" encoding=\"UTF-8\"\?>"
negate => "true"
what => "previous"
max_lines => 300000
}
}
}
filter {
if "multiline" in [tags] {
xml {
source => "message"
#store_xml => false # this limits the data indexed to only xpath and grok created fields
store_xml => true #saves ALL xml nodes if it can - can be VERY large
target => "xmldata" # only used with store_xml => true
}
ruby {
path => "/etc/logstash/rubyscripts/inventors.rb"
}
}
}
output {
file {
path => [ "/tmp/logstash_output_text_file" ]
codec => rubydebug
}
}
And here is the inventors.rb script:
# the value of `params` is the value of the hash passed to `script_params`
# in the logstash configuration
def register(params)
#drop_percentage = params["percentage"]
end
def filter(event)
# get the number of inventors to loop over
# convert the array key string number 0 to an integer
n = event.get('[num_inventors][0]').to_i
# set a loop number to start with
i = 0
#create empty arrays to fill
firstname = []
lastname = []
# loop over inventors until n is reached
while (i < n) do
#get the inventors first name
fname = event.get('[event][us-patent-application][us-bibliographic-data-application][us-parties][inventors][inventor][addressbook][last-name]')
#puts"first name #{fname}"
# push the first name into firstname array
firstname.push(fname)
#get the inventors last name
lname = event.get('[event][us-patent-application][us-bibliographic-data-application][us-parties][inventors][inventor][addressbook][last-name]')
#puts"last name #{lname}"
# push the last name into firstname array
lastname.push(lname)
#increment n up 1
i += 1
end
#merge firstname and lastname arrays
names = firstname.zip(lastname)
# push the names array to the event
event.set('allnames', names)
return [event]
end
Finally, here is the Elasticsearch output:
{
"host" => "localhost.localdomain",
"allnames" => [],
"type" => "xml",
"#timestamp" => 2018-09-20T17:28:05.332Z,
"message" => "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\r\n<!DOCTYPE us-patent-application SYSTEM \"us-patent-application-v44-2014-04-03.dtd\" [ ]>\r\n<us-patent-application lang=\"EN\" dtd-version=\"v4.4 2014-04-03\" file=\"US20180000001A1-20180104.XML\" status=\"PRODUCTION\" id=\"us-patent-application\" country=\"US\" date-produced=\"20171219\" date-publ=\"20180104\">\r\n\t<us-bibliographic-data-application lang=\"EN\" country=\"US\">\r\n\t\t<us-parties>\r\n\t\t\t<inventors>\r\n\t\t\t\t<inventor sequence=\"00\" designation=\"us-only\">\r\n\t\t\t\t\t<addressbook>\r\n\t\t\t\t\t\t<last-name>Evans</last-name>\r\n\t\t\t\t\t\t<first-name>Mike</first-name>\r\n\t\t\t\t\t\t<address>\r\n\t\t\t\t\t\t\t<city>Emerald Park</city>\r\n\t\t\t\t\t\t\t<country>CA</country>\r\n\t\t\t\t\t\t</address>\r\n\t\t\t\t\t</addressbook>\r\n\t\t\t\t</inventor>\r\n\t\t\t\t<inventor sequence=\"01\" designation=\"us-only\">\r\n\t\t\t\t\t<addressbook>\r\n\t\t\t\t\t\t<last-name>Lucas</last-name>\r\n\t\t\t\t\t\t<first-name>Lisa</first-name>\r\n\t\t\t\t\t\t<address>\r\n\t\t\t\t\t\t\t<city>Regina</city>\r\n\t\t\t\t\t\t\t<country>CA</country>\r\n\t\t\t\t\t\t</address>\r\n\t\t\t\t\t</addressbook>\r\n\t\t\t\t</inventor>\r\n\t\t\t\t<inventor sequence=\"02\" designation=\"us-only\">\r\n\t\t\t\t\t<addressbook>\r\n\t\t\t\t\t\t<last-name>Smith</last-name>\r\n\t\t\t\t\t\t<first-name>Scott R.</first-name>\r\n\t\t\t\t\t\t<address>\r\n\t\t\t\t\t\t\t<city>Regina</city>\r\n\t\t\t\t\t\t\t<country>CA</country>\r\n\t\t\t\t\t\t</address>\r\n\t\t\t\t\t</addressbook>\r\n\t\t\t\t</inventor>\r\n\t\t\t</inventors>\r\n\t\t</us-parties>\r\n\t</us-bibliographic-data-application>\r",
"#version" => "1",
"tags" => [
[0] "multiline",
[1] "_xmlparsefailure"
],
"path" => "/opt/uspto/test.xml"
}
The behavior I am looking for the to have the "allnames" => [] array look like:
"allnames" => ["Mike Evans",
"Lisa Lucas",
"John R. Smith"],
I cannot figure out how to properly grab the 'first-name' and last-name' nodes in my ruby script. I'm pulling my hair out trying to get a working solution. Any ideas are welcome!

Answer 1

Please see if this solves your requirement. I was really hopeful that this can be solved with xml + xpath. But I guess all xpath functions are not supported. :(
input {
file {
path => [
"/opt/uspto/*.xml"
]
start_position => "beginning"
#use for testing
sincedb_path => "/dev/null"
# set this sincedb path when not testing
#sincedb_path => "/opt/logstash/tmp/sincedb"
exclude => "*.gz"
type => "xml"
codec => multiline {
#pattern => "<wo-ocr-published-application"
pattern => "<?xml version=\"1.0\" encoding=\"UTF-8\"\?>"
negate => "true"
what => "previous"
max_lines => 300000
}
}
}
filter {
if "multiline" in [tags] {
xml {
source => "message"
store_xml => false
target => "xmldata" # only used with store_xml => true
force_array=> false
xpath => [ "//us-bibliographic-data-application/us-parties/inventors/inventor/addressbook/first-name/text()" , "[xmldata][us-bibliographic-data-application][us-parties][inventors][first_name]" ,
"//us-bibliographic-data-application/us-parties/inventors/inventor/addressbook/last-name/text()", "[xmldata][us-bibliographic-data-application][us-parties][inventors][last_name]" ]
}
ruby {
code => ' first_name = event.get("[xmldata][us-bibliographic-data-application][us-parties][inventors][first_name]")
last_name = event.get("[xmldata][us-bibliographic-data-application][us-parties][inventors][last_name]")
event.set("[xmldata][us-bibliographic-data-application][us-parties][inventors][names]", first_name.zip(last_name).map{ |a| a.join(" ") })
'
}
mutate {
remove_field => ["message", "host", "path", "[xmldata][us-bibliographic-data-application][us-parties][inventors][first_name]", "[xmldata][us-bibliographic-data-application][us-parties][inventors][last_name]" ]
}
}
}
output {
file {
path => [ "/tmp/logstash_output_text_file" ]
codec => rubydebug
}
}
Output is
{
"#timestamp": "2018-09-21T12:00:26.428Z",
"#version": "1",
"tags": [
"multiline"
],
"type": "xml",
"xmldata": {
"us-bibliographic-data-application": {
"us-parties": {
"inventors": {
"names": [
"Mike Evans",
"Lisa Lucas",
"John R. Smith"
]
}
}
}
}
}