Logstash - parses only one JSON event - elasticsearch

Question

I am using ELK 5.3.0. I am trying to parse simple JSON document. It does work creating key/values, however it writes in Elasticsearch only one event. And it does that randomly. Sometimes is first, sometimes second or third. But is is always one event.
Filesetup (Created in Mac. One line per JSON object), three events:
{"timestamp":"2012-01-01 02:00:01", "severity":"ERROR",
"messages":"Foo failed", "fieldone": "I am first entry... if the value
of a field one", "fieldtwo": "ttthis if the value of a field two"}
{"timestamp":"2013-01-01 02:04:02", "severity":"INFO", "messages":"Bar
was successful", "fieldone": "I am second entry... if the value of a
field one", "fieldtwo": "this if the value of a field two"}
{"timestamp":"2017-01-01 02:10:12", "severity":"DEBUG",
"messages":"Baz was notified", "fieldone": "I am third entry... if the
value of a field one", "fieldtwo": "this if the value of a field two"}
Filebeatsetup:
- input_type: log
paths: Downloads/elk/small/jsontest.log
document_type: jsonindex
Logstashsetup:
filter {
if [#metadata][type] == "jsonindex" {
json {
source => "message"
}
}
}
Logstash output (shows three events):
{
"severity" => "DEBUG",
"offset" => 544,
"#uuid" => "a316bb67-98e5-4551-8243-f8538023cfd9",
"input_type" => "log",
"source" => "/Users/xxx/Downloads/elk/small/jsontest.log",
"fieldone" => "this if the value of a field one",
"type" => "jsonindex",
"tags" => [
[0] "beats_input_codec_json_applied",
[1] "_dateparsefailure"
],
"fieldtwo" => "this if the value of a field two",
"#timestamp" => 2017-05-08T11:25:41.586Z,
"#version" => "1",
"beat" => {
"hostname" => "C700893",
"name" => "C700893",
"version" => "5.3.0"
},
"host" => "C700893",
"fingerprint" => "bcb57f445084cc0e474366bf892f6b4ab9162a4e",
"messages" => "Baz was notified",
"timestamp" => "2017-01-01 02:10:12"
}
{
"severity" => "INFO",
"offset" => 361,
"#uuid" => "6d4b4401-a440-4894-b0de-84c97fc4eaf5",
"input_type" => "log",
"source" => "/Users/xxx/Downloads/elk/small/jsontest.log",
"fieldone" => "this if the value of a field one",
"type" => "jsonindex",
"tags" => [
[0] "beats_input_codec_json_applied",
[1] "_dateparsefailure"
],
"fieldtwo" => "this if the value of a field two",
"#timestamp" => 2017-05-08T11:25:41.586Z,
"#version" => "1",
"beat" => {
"hostname" => "C700893",
"name" => "C700893",
"version" => "5.3.0"
},
"host" => "C700893",
"fingerprint" => "bcb57f445084cc0e474366bf892f6b4ab9162a4e",
"messages" => "Bar was successful",
"timestamp" => "2013-01-01 02:04:02"
}
{
"severity" => "ERROR",
"offset" => 177,
"#uuid" => "d9bd0a0b-0021-48fd-8d9e-d6f82cd1e506",
"input_type" => "log",
"source" => "/Users/xxx/Downloads/elk/small/jsontest.log",
"fieldone" => "this if the value of a field one",
"type" => "jsonindex",
"tags" => [
[0] "beats_input_codec_json_applied",
[1] "_dateparsefailure"
],
"fieldtwo" => "this if the value of a field two",
"#timestamp" => 2017-05-08T11:25:41.586Z,
"#version" => "1",
"beat" => {
"hostname" => "C700893",
"name" => "C700893",
"version" => "5.3.0"
},
"host" => "C700893",
"fingerprint" => "bcb57f445084cc0e474366bf892f6b4ab9162a4e",
"messages" => "Foo failed",
"timestamp" => "2012-01-01 02:00:01"
}
ElasticSearch (document viewed in as JSON):
"tags": [
"beats_input_codec_json_applied",
"_dateparsefailure"
],
There is no JSON failure. _dateparsefailure is expected.
What is going on in here?
EDIT (Solution):
After some time, I figured I was shooting myself in the leg. Since I am parsing many different logs and also log types, I need to make certain I do not have duplicates, this in my Logstash output section I have this piece of code to ensure no duplicate log entires:
uuid {
target => "#uuid"
overwrite => true
}
fingerprint {
source => ["message"]
target => "fingerprint"
key => "78787878"
method => "SHA1"
concatenate_sources => true
}
}
End also in the same section I call ElasticSearch like this:
if [#metadata][type] == "jsonindex" {
elasticsearch {
hosts => [ "localhost:9200" ]
index => "%{[#metadata][type]}"
document_id => "%{fingerprint}"
}
}
Since my JSON objects do not contain message property, it is always virtually identical:
fingerprint {
source => ["message"]
...
Small edit to index creation fixed the problem:
if [#metadata][type] == "jsonindex" {
elasticsearch {
hosts => [ "localhost:9200" ]
index => "%{[#metadata][type]}"
}
}

Answer 1

your data need to separate by lines
the grok need to parse it as different line,it will become 3 data
for example :
{"timestamp":"2012-01-01 02:00:01", "severity":"ERROR", "messages":"Foo failed", "fieldone": "I am first entry... if the value of a field one", "fieldtwo": "ttthis if the value of a field two"}
{"timestamp":"2013-01-01 02:04:02", "severity":"INFO", "messages":"Bar was successful", "fieldone": "I am second entry... if the value of a field one", "fieldtwo": "this if the value of a field two"}
{"timestamp":"2017-01-01 02:10:12", "severity":"DEBUG", "messages":"Baz was notified", "fieldone": "I am third entry... if the value of a field one", "fieldtwo": "this if the value of a field two"}
you are in one line so the result is parse the last one which means the timestamp is the last one
"timestamp":"2017-01-01 02:10:12
if you change by line ,i think maybe not but you could use this
- input_type: log
paths: Downloads/elk/small/jsontest.log
document_type: jsonindex
multiline.pattern: '^{"timestamp":"[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}, '
multiline.negate: true
multiline.match: after
add the mutiline to change ,but i am afraid your data not separate by lines