aggregate multiple recursive logstash - elasticsearch

Question

I am using logstash with input jdbc, and would like to embed one object inside another with aggregate.
How can I use add recursive?
Ie add an object inside another object?
This would be an example:
{
"_index": "my-index",
"_type": "test",
"_id": "1",
"_version": 1,
"_score": 1,
"_source": {
"id": "1",
"properties": {
"nested_1": [
{
"A": 0,
"B": "true",
"C": "PEREZ, MATIAS ROGELIO Y/O",
"Nested_2": [
{
"Z1": "true",
"Z2": "99999"
}
},
{
"A": 0,
"B": "true",
"C": "SALVADOR MATIAS ROMERO",
"Nested_2": [
{
"Z1": "true",
"Z2": "99999"
}
}
]
}
}
}
I'm using something like that but it doesn't work
aggregate {
task_id => "%{id}"
code => "
map['id'] = event.get('id')
map['nested_1_list'] ||= []
map['nested_1'] ||= []
if (event.get('id') != nil)
if !( map['nested_1_list'].include?event.get('id') )
map['nested_1_list'] << event.get('id')
map['nested_1'] << {
'A' => event.get('a'),
'B' => event.get('b'),
'C' => event.get('c'),
map['nested_2_list'] ||= []
map['nested_2'] ||= []
if (event.get('id_2') != nil)
if !( map['nested_2_list'].include?event.get('id_2') )
map['nested_2_list'] << event.get('id_2')
map['nested_2'] << {
'Z1' => event.get('z1'),
'Z2' => event.get('z2')
}
end
end
}
end
end
event.cancel()
"
push_previous_map_as_event => true
timeout => 3
}
Any idea how to implement this?........................
..........

Answer 1

Finally what I did was, generate the JSON from the input, that is, from a stored procedure that is consumed from a view (vw) from the input statement of logstash.
Once consumed, I process it as json and I already have that json to work as one more variable.
# Convierto el string a json real (quita comillas y barras invertidas)
ruby {
code => "
require 'json'
json_value = JSON.parse(event.get('field_db').to_s)
event.set('field_convert_to_json',json_value)
"
}

Answer 2

Maybe you can try this. Note This will be applicable only when you want to have a single object and not an array of object.
Please do visit my blog for other formats.
https://xyzcoder.github.io/2020/07/29/indexing-documents-using-logstash-and-python.html
input {
jdbc {
jdbc_driver_library => "/usr/share/logstash/javalib/mssql-jdbc-8.2.2.jre11.jar"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_connection_string => "jdbc:sqlserver://host.docker.internal;database=StackOverflow2010;user=pavan;password=pavankumar#123"
jdbc_user => "pavan"
jdbc_password => "pavankumar#123"
statement => "select top 500 p.Id as PostId,p.AcceptedAnswerId,p.AnswerCount,p.Body,u.Id as userid,u.DisplayName,u.Location
from StackOverflow2010.dbo.Posts p inner join StackOverflow2010.dbo.Users u
on p.OwnerUserId=u.Id"
}
}
filter {
aggregate {
task_id => "%{postid}"
code => "
map['postid'] = event.get('postid')
map['accepted_answer_id'] = event.get('acceptedanswerid')
map['answer_count'] = event.get('answercount')
map['body'] = event.get('body')
map['user'] = {
'id' => event.get('userid'),
'displayname' => event.get('displayname'),
'location' => event.get('location')
}
map['user']['test'] = {
'test_body' => event.get('postid')
}
event.cancel()"
push_previous_map_as_event => true
timeout => 30
}
}
output {
elasticsearch {
hosts => ["http://elasticsearch:9200", "http://elasticsearch:9200"]
index => "stackoverflow_top"
}
stdout {
codec => rubydebug
}
}
and my output is
{
"_index" : "stackoverflow_top",
"_type" : "_doc",
"_id" : "S8WEmnMBrXsRTNbKO0JJ",
"_score" : 1.0,
"_source" : {
"#version" : "1",
"body" : """<p>How do I store binary data in MySQL?</p>
""",
"#timestamp" : "2020-07-29T12:20:22.649Z",
"answer_count" : 10,
"user" : {
"displayname" : "Geoff Dalgas",
"location" : "Corvallis, OR",
"test" : {
"test_body" : 17
},
"id" : 2
},
"postid" : 17,
"accepted_answer_id" : 26
}
Here test object is nested into the user object