Logstash not creating indexes in Windows 10 - elasticsearch

I have used the zip files to start logstash, kibana and elasticsearch. I am ingesting a csv file from logstash to elastic search
input {
file {
path => "D:\tls202_part01\tls202_part01.csv"
start_position => "beginning"
}
}
filter {
csv {
separator => ","
columns => ["appln_id", "appln_title_lg", "appln_title"]
}
mutate {
convert => ["appln_id", "integer"]
convert => ["appln_title_lg", "string"]
convert => ["appln_title", "string"]
}
}
output {
elasticsearch {
hosts => "localhost"
index => "title"
}
stdout {
codec => rubydebug
}
}
this is my config file. When I search for index title it is not there and logstash logs are these:
Sending Logstash logs to D:/logstash-6.5.4/logs which is now configured via log4j2.properties
[2018-12-26T10:22:35,672][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-12-26T10:22:35,699][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.5.4"}
[2018-12-26T10:22:41,588][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-12-26T10:22:42,051][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2018-12-26T10:22:42,297][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2018-12-26T10:22:42,370][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-12-26T10:22:42,376][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-12-26T10:22:42,417][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost"]}
[2018-12-26T10:22:42,439][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-12-26T10:22:42,473][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"#timestamp"=>{"type"=>"date"}, "#version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-12-26T10:22:43,330][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"D:/logstash-6.5.4/data/plugins/inputs/file/.sincedb_bb5ff7ebd070422c5b611ac87e9e7087", :path=>["D:\\tls202_part01\\tls202_part01.csv"]}
[2018-12-26T10:22:43,390][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x389cc614 run>"}
[2018-12-26T10:22:43,499][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2018-12-26T10:22:43,532][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2018-12-26T10:22:43,842][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
The CSV file is large of 2GB csv data.
Also, kibana is showing no Elasticsearch data found for creating indexes.

It seems that logstash didn't found your file, change your path from backslash to forward slash and see if it works.
path => "D:/tls202_part01/tls202_part01.csv"

Related

Logstash starts but it does not load the data to elasticsearch

I am trying to read the information in json format from log file and load it in elasticsearch. I am using logstash.
OS: windows 10
ElastiSearch version is 7.6.2
LogStash version is 7.6.2
The log file content is like below:
{"#timestamp":"2020-05-03T15:09:38.255+02:00","#version":1,"message":"The following profiles are active: default","logger_name":"payroll.employee.EmployeeApplication","thread_name":"main","level":"INFO","level_value":20000,"springAppName":"employee"}
{"#timestamp":"2020-05-03T15:09:59.136+02:00","#version":1,"message":"Started EmployeeApplication in 24.892 seconds (JVM running for 27.193)","logger_name":"payroll.employee.EmployeeApplication","thread_name":"main","level":"INFO","level_value":20000,"springAppName":"employee"}
Below it is the logstash.conf file
input {
file {
path => "C:/Users/User/Desktop/Git-Repos/Microservice/elk-logs/employee.log"
start_position => "beginning"
sincedb_path => "NUL"
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "logback-%{+YYYY.MM.dd}"
#user => "elastic"
#password => "changeme"
}
}
I run the logstash like below :
logstash.bat -f C:\Users\User\Desktop\Apps\logstash-7.6.2\config\logstash.conf
And the output is like below:
[2020-05-03T17:52:08,394][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}
[2020-05-03T17:52:08,456][INFO ][logstash.outputs.elasticsearch][main] Using default mapping template
[2020-05-03T17:52:08,519][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been created for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[2020-05-03T17:52:08,534][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["C:/Users/User/Desktop/Apps/logstash-7.6.2/config/logstash.conf"], :thread=>"#<Thread:0x6e5e96fc run>"}
[2020-05-03T17:52:08,597][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"#timestamp"=>{"type"=>"date"}, "#version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2020-05-03T17:52:10,327][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-05-03T17:52:10,380][INFO ][filewatch.observingtail ][main] START, creating Discoverer, Watch with file and sincedb collections
[2020-05-03T17:52:10,411][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-05-03T17:52:10,929][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
It seeems that the logstash starts correctly but it fails to insert the data in elasticsearch.
I really would appriciate any help.
Regards
P.S
I added output { stdout { codec => json } } and below it is result:
{"message":"{\"#timestamp\":\"2020-05-03T15:24:33.059+02:00\",\"#version\":1,\"message\":\"findAll called\",\"logger_name\":\"payroll.employee.service.EmployeeServiceImp\",\"thread_name\":\"http-nio-8082-exec-1\",\"level\":\"INFO\",\"level_value\":20000,\"springAppName\":\"employee\",\"traceId\":\"042059d1825b87b6\",\"spanId\":\"042059d1825b87b6\",\"spanExportable\":\"false\",\"X-Span-Export\":\"false\",\"X-B3-SpanId\":\"042059d1825b87b6\",\"X-B3-TraceId\":\"042059d1825b87b6\"}\r","#version":"1","#timestamp":"2020-05-03T17:22:09.423Z","path":"C:/Users/User/Desktop/Git-Repos/Microservice/elk-logs/employee.log","host":"WIN-7C4J6I3B56C"}{"message":"{\"#timestamp\":\"2020-05-03T15:09:34.106+02:00\",\"#version\":1,\"message\":\"Employee Application Started\",\"logger_name\":\"payroll.employee.EmployeeApplication\",\"thread_name\":\"main\",\"level\":\"INFO\",\"level_value\":20000}\r","#version":"1","#timestamp":"2020-05-03T17:22:09.389Z","path":"C:/Users/User/Desktop/Git-Repos/Microservice/elk-logs/employee.log","host":"WIN-7C4J6I3B56C"}{"message":"{\"#timestamp\":\"2020-05-03T15:25:04.625+02:00\",\"#version\":1,\"message\":\"countByDepartmentId called, departmentId: 1\",\"logger_name\":\"payroll.employee.service.EmployeeServiceImp\",\"thread_name\":\"http-nio-8082-exec-2\",\"level\":\"INFO\",\"level_value\":20000,\"springAppName\":\"employee\",\"traceId\":\"6a534061270b8b46\",\"spanId\":\"8b54f9e340a458a2\",\"spanExportable\":\"false\",\"X-Span-Export\":\"false\",\"X-B3-SpanId\":\"8b54f9e340a458a2\",\"X-B3-ParentSpanId\":\"30113ef7e385207d\",\"X-B3-TraceId\":\"6a534061270b8b46\",\"parentId\":\"30113ef7e385207d\"}\r","#version":"1","#timestamp":"2020-05-03T17:22:09.423Z","path":"C:/Users/User/Desktop/Git-Repos/Microservice/elk-logs/employee.log","host":"WIN-7C4J6I3B56C"}{"message":"{\"#timestamp\":\"2020-05-03T15:09:59.136+02:00\",\"#version\":1,\"message\":\"Started EmployeeApplication in 24.892 seconds (JVM running for 27.193)\",\"logger_name\":\"payroll.employee.EmployeeApplication\",\"thread_name\":\"main\",\"level\":\"INFO\",\"level_value\":20000,\"springAppName\":\"employee\"}\r","#version":"1","#timestamp":"2020-05-03T17:22:09.423Z","path":"C:/Users/User/Desktop/Git-Repos/Microservice/elk-logs/employee.log","host":"WIN-7C4J6I3B56C"}{"message":"{\"#timestamp\":\"2020-05-03T15:09:38.255+02:00\",\"#version\":1,\"message\":\"The following profiles are active: default\",\"logger_name\":\"payroll.employee.EmployeeApplication\",\"thread_name\":\"main\",\"level\":\"INFO\",\"level_value\":20000,\"springAppName\":\"employee\"}\r","#version":"1","#timestamp":"2020-05-03T17:22:09.408Z","path":"C:/Users/User/Desktop/Git-Repos/Microservice/elk-logs/employee.log","host":"WIN-7C4J6I3B56C"}{"message":"{\"#timestamp\":\"2020-05-03T15:25:04.709+02:00\",\"#version\":1,\"message\":\"countByDepartmentId called, departmentId: 2\",\"logger_name\":\"payroll.employee.service.EmployeeServiceImp\",\"thread_name\":\"http-nio-8082-exec-3\",\"level\":\"INFO\",\"level_value\":20000,\"springAppName\":\"employee\",\"traceId\":\"6a534061270b8b46\",\"spanId\":\"2391430ee2ce38cb\",\"spanExportable\":\"false\",\"X-Span-Export\":\"false\",\"X-B3-SpanId\":\"2391430ee2ce38cb\",\"X-B3-ParentSpanId\":\"0964d2e2fba9f7e5\",\"X-B3-TraceId\":\"6a534061270b8b46\",\"parentId\":\"0964d2e2fba9f7e5\"}\r","#version":"1","#timestamp":"2020-05-03T17:22:09.423Z","path":"C:/Users/User/Desktop/Git-Repos/Microservice/elk-logs/employee.log","host":"WIN-7C4J6I3B56C"}
Now i am more confused. The problem is with logstash or elasticsearch or kibana that I am not able to see any information in kibana?
As discussed in the chat. The problem was the pattern of the index name.
Changing the pattern to logback-* worked perfectly.

Logstash not inserting records in ElasticSearch

I am trying to insert simple CSV values to ES. it is not going thru.
The conf file i am using is as follows:
input {
file {
path => "D:/ELK/Logs/rahul.csv"
start_position => "beginning"
sincedb_path => "NULL"
}
}
filter {
csv {
columns => [ "record_id", "duration", "src_bytes", "dest_bytes" ]
}
}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "rahul"
}
stdout {}
}
I checked the error log on Logstash\logs folder. following is there
[2020-04-17T17:07:34,598][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-04-17T17:07:34,662][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.6.2"}
[2020-04-17T17:07:35,419][INFO ][org.reflections.Reflections] Reflections took 25 ms to scan 1 urls, producing 20 keys and 40 values
[2020-04-17T17:07:36,217][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2020-04-17T17:07:36,328][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2020-04-17T17:07:36,359][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7}
[2020-04-17T17:07:36,362][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2020-04-17T17:07:36,393][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}
[2020-04-17T17:07:36,423][INFO ][logstash.outputs.elasticsearch][main] Using default mapping template
[2020-04-17T17:07:36,443][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been created for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[2020-04-17T17:07:36,447][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>6, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>750, "pipeline.sources"=>["D:/ELK/Logs/rahul.conf"], :thread=>"#<Thread:0x5e5be359 run>"}
[2020-04-17T17:07:36,467][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"#timestamp"=>{"type"=>"date"}, "#version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2020-04-17T17:07:37,167][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-04-17T17:07:37,188][INFO ][filewatch.observingtail ][main] START, creating Discoverer, Watch with file and sincedb collections
[2020-04-17T17:07:37,205][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-04-17T17:07:37,395][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
resolved by adding sincedb_path => "NULL" to the input plugin, like so:
input {
file {
path => "D:/ELK/Logs/rahul.csv"
start_position => "beginning"
sincedb_path => "NULL"
}
}

How to enter data from csv to elasticsearch using logstash?

I am trying to insert data into elasticsearch using logstash but getting stuck. My config file:
logstashCrime.conf
input {
file {
path => "C:\elk\sampl.csv"
start_position => "beginning"
sincedb_path => "nul"
}
}
filter {
csv {
separator => ","
columns => ["code","name"]
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "crime"
}
stdout {
codec => rubydebug
}
}
I am getting a response like this when I try to insert using logstash-7.2.0\bin\logstash -f c:\elk\logstashCrime.conf:
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to C:/elk/logstash-7.2.0/logs which is now configured via log4j2.properties
[2019-07-15T16:10:22,300][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-07-15T16:10:22,320][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.2.0"}
[2019-07-15T16:10:28,817][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-07-15T16:10:29,009][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-07-15T16:10:29,058][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>7}
[2019-07-15T16:10:29,063][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2019-07-15T16:10:29,087][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-07-15T16:10:29,148][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2019-07-15T16:10:29,202][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"#timestamp"=>{"type"=>"date"}, "#version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2019-07-15T16:10:29,225][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been create for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[2019-07-15T16:10:29,229][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, :thread=>"#<Thread:0x74421f35 run>"}
[2019-07-15T16:10:30,202][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"main"}
[2019-07-15T16:10:30,408][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2019-07-15T16:10:30,416][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-07-15T16:10:30,755][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
My sampl.csv file looks like this:
id,name
------
1,john
2,doe
3,you
4,me
I am new to ELK. Any help appreciated. Also, I am using windows 10 as my OS. I successfully created an index using logstash withoug csv, but with csv it is not creating.
I want to view it in Kibana but since the index is not created, I cant see the index in Kibana.
You cannot use backslash in the path option of a file input. Use forward slash.

logstash configuration pipeline

I have a log file looking like this
116.50.181.5 - - [18/May/2015:19:05:32 +0000] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/style2.css" "Mozilla/5.0 (X11; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0"
my logstash configuration is as below:
input {
file {
path => "C:\Users\PC\Documents\elk\Input\listening.txt"
start_position => "beginning"
}
}
filter {
grok {
match => {
"message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}'
}
}
date {
match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]
locale => en
}
geoip {
source => "clientip"
}
useragent {
source => "agent"
target => "useragent"
}
}
output {
elasticsearch
{
hosts => "http://localhost:9200"
index => "log"
}
}
everything works just fine, I have no errors in logstash , but the data does'nt appear in elasticsearch as expected.
C:\elk\logstash-7.1.1\bin>logstash -f logstashETL.conf
Sending Logstash logs to C:/elk/logstash-7.1.1/logs which is now configured via log4j2.properties
[2019-06-12T16:02:27,371][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-06-12T16:02:27,405][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.1.1"}
[2019-06-12T16:02:36,087][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-06-12T16:02:36,344][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-06-12T16:02:36,428][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>7}
[2019-06-12T16:02:36,428][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2019-06-12T16:02:36,469][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}
[2019-06-12T16:02:36,493][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2019-06-12T16:02:36,513][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, :thread=>"#<Thread:0x75642d2 run>"}
[2019-06-12T16:02:36,753][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"#timestamp"=>{"type"=>"date"}, "#version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2019-06-12T16:02:37,814][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"C:/elk/logstash-7.1.1/data/plugins/inputs/file/.sincedb_636c54fa423804cc695f80e1cb9d6ccd", :path=>["C:\\Users\\PC\\Documents\\elk\\Input\\listening.txt"]}
[2019-06-12T16:02:37,878][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"main"}
[2019-06-12T16:02:37,988][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-06-12T16:02:38,008][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2019-06-12T16:02:38,773][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
maybe there are somethings wrong or misssing in my code.
Add the below code in your input
start_position => "beginning"
sincedb_path => "/dev/null"
sincedb_path => "/dev/null" means it doesn't store sincedb files. These files are keeping byte ofset of where the logstash left on the file.
Then go to logstash/data/plugins/inputs/file directory. After that run below command at this directory
rm -r .sincedb*
Finally run your logstash pipeline. It should work.

Not able to map csv file from logstash to kibana in Window

I'm trying to feed data in csv files into elastic search using logstash. My logsatsh config file looks like this:
input {
file {
path => "D:\Log Anlyser\data\cars.csv"
start_position => "beginning"
sincedb_path => "NUL"
}
}
filter {
csv {
separator => ","
columns => [ "maker", "model", "mileage", "manufacture_year", "engine_displacement", "engine_power", "body_type", "color_slug", "stk_year", "transmission", "door_count", "seat_count", "fuel_type", "date_created", "date_last_seen", "price_eur" ]
}
mutate {convert => ["milage", "integer"] }
mutate {convert => ["price_eur", "float"] }
mutate {convert => ["engine_power", "integer"] }
mutate {convert => ["door_count", "integer"] }
mutate {convert => ["seat_count", "integer"] }
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => ["cars-%{+YYYY.MM.dd}"]
}
}
while firing this command for logstash in window : logstash -f cars.conf i am getting this:-
Sending Logstash logs to D:/Log_Anlyser/logstash/logs which is now configured via log4j2.properties
[2019-02-26T12:05:51,690][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-02-26T12:05:51,721][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.6.1"}
[2019-02-26T12:05:57,133][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-02-26T12:05:57,510][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-02-26T12:05:57,664][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-02-26T12:05:57,711][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>5}
[2019-02-26T12:05:57,742][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-02-26T12:05:57,758][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2019-02-26T12:05:57,852][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"#timestamp"=>{"type"=>"date", "include_in_all"=>false}, "#version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2019-02-26T12:05:58,179][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x274079d5 run>"}
[2019-02-26T12:05:58,226][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-02-26T12:05:58,226][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2019-02-26T12:05:58,547][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
Now While connecting to kibana(localhost:5601) i am not able to map the data. Getting this error:-
Unable to fetch mapping. Do you have indices matching the pattern?
Can you please help.
I got the problem. Mistake is very silly. Path of CSV file was wrong. Earlier path was path => "D:\Log Anlyser\data\cars.csv". Current Path is`
path => "D:/Log_Anlyser/data/cars.csv"
It will work
There might be few reasons - maybe the data is not reaching ES at all. you can check that by verifying the index exists, by running
GET es-url:9200/_cat/indices/cars*
If an index exists then you should be able to create the index pattern in Kibana.
If the index is missing then either Logstash is not reading the input file, or elasticsearch is not reachable. need to check logstash logs, and make sure data reaches ES.

Resources