I am trying to insert simple CSV values to ES. it is not going thru.
The conf file i am using is as follows:
input {
file {
path => "D:/ELK/Logs/rahul.csv"
start_position => "beginning"
sincedb_path => "NULL"
}
}
filter {
csv {
columns => [ "record_id", "duration", "src_bytes", "dest_bytes" ]
}
}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "rahul"
}
stdout {}
}
I checked the error log on Logstash\logs folder. following is there
[2020-04-17T17:07:34,598][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-04-17T17:07:34,662][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.6.2"}
[2020-04-17T17:07:35,419][INFO ][org.reflections.Reflections] Reflections took 25 ms to scan 1 urls, producing 20 keys and 40 values
[2020-04-17T17:07:36,217][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2020-04-17T17:07:36,328][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2020-04-17T17:07:36,359][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7}
[2020-04-17T17:07:36,362][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2020-04-17T17:07:36,393][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}
[2020-04-17T17:07:36,423][INFO ][logstash.outputs.elasticsearch][main] Using default mapping template
[2020-04-17T17:07:36,443][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been created for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[2020-04-17T17:07:36,447][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>6, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>750, "pipeline.sources"=>["D:/ELK/Logs/rahul.conf"], :thread=>"#<Thread:0x5e5be359 run>"}
[2020-04-17T17:07:36,467][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"#timestamp"=>{"type"=>"date"}, "#version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2020-04-17T17:07:37,167][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-04-17T17:07:37,188][INFO ][filewatch.observingtail ][main] START, creating Discoverer, Watch with file and sincedb collections
[2020-04-17T17:07:37,205][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-04-17T17:07:37,395][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
resolved by adding sincedb_path => "NULL" to the input plugin, like so:
input {
file {
path => "D:/ELK/Logs/rahul.csv"
start_position => "beginning"
sincedb_path => "NULL"
}
}
Related
I have this XML file which I stored in D:\ in Window 10:
<?xml version="1.0" encoding="UTF-8"?>
<root>
<ChainId>7290027600007</ChainId>
<SubChainId>001</SubChainId>
<StoreId>001</StoreId>
<BikoretNo>9</BikoretNo>
<DllVerNo>8.0.1.3</DllVerNo>
</root>
I have installed Elastic Search and able to access it at http://localhost:9200/. I have installed Logstash and created logstash-xml.conf to ingest the above XML file. In logstash-xml.conf, the configuration is below:
input
{
file
{
path => "D:\data.xml"
start_position => "beginning"
sincedb_path => "NUL"
exclude => "*.gz"
type => "xml"
codec => multiline {
pattern => "<?xml "
negate => "true"
what => "previous"
}
}
}
filter {
xml{
source => "message"
store_xml => false
target => "root"
xpath => [
"/root/ChainId/text()", "ChainId",
"/root/SubChainId/text()", "SubChainId",
"/root/StoreId/text()", "StoreId",
"/root/BikoretNo/text()", "BikoretNo",
"/root/DllVerNo/text()", "DllVerNo"
]
}
mutate {
gsub => [ "message", "[\r\n]", "" ]
}
}
output{
elasticsearch{
hosts => ["http://localhost:9200/"]
index => "parse_xml"
}
stdout
{
codec => rubydebug
}
}
When I run this configuration in the command line, I see this below:
D:\logstash\bin>logstash -f logstash-xml.conf
"Using bundled JDK: ""
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/C:/Users/CHEEWE~1.NGA/AppData/Local/Temp/jruby-3748/jruby14189572270520245744jopenssl.jar) to field java.security.MessageDigest.provider
WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Sending Logstash logs to D:/logstash/logs which is now configured via log4j2.properties
[2020-12-05T09:27:21,716][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.10.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +indy +jit [mswin32-x86_64]"}
[2020-12-05T09:27:22,053][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-12-05T09:27:24,031][INFO ][org.reflections.Reflections] Reflections took 37 ms to scan 1 urls, producing 23 keys and 47 values
[2020-12-05T09:27:26,083][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2020-12-05T09:27:26,311][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2020-12-05T09:27:26,378][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7}
[2020-12-05T09:27:26,383][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2020-12-05T09:27:26,437][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200/"]}
[2020-12-05T09:27:26,487][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[2020-12-05T09:27:26,621][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"#timestamp"=>{"type"=>"date"}, "#version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2020-12-05T09:27:28,152][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["D:/logstash/bin/logstash-xml.conf"], :thread=>"#<Thread:0x65f57880 run>"}
[2020-12-05T09:27:29,176][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>1.02}
[2020-12-05T09:27:29,640][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-12-05T09:27:29,712][INFO ][filewatch.observingtail ][main][aca15cd3c6850472d105bd7b2b7a43da8ce8ec36a4b0b8c19830d898f1eb1109] START, creating Discoverer, Watch with file and sincedb collections
[2020-12-05T09:27:29,726][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-12-05T09:27:30,020][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
But, going back to ElasticSearch, I can logstash index created, but I can't see the XML data loaded.
I am trying to read the information in json format from log file and load it in elasticsearch. I am using logstash.
OS: windows 10
ElastiSearch version is 7.6.2
LogStash version is 7.6.2
The log file content is like below:
{"#timestamp":"2020-05-03T15:09:38.255+02:00","#version":1,"message":"The following profiles are active: default","logger_name":"payroll.employee.EmployeeApplication","thread_name":"main","level":"INFO","level_value":20000,"springAppName":"employee"}
{"#timestamp":"2020-05-03T15:09:59.136+02:00","#version":1,"message":"Started EmployeeApplication in 24.892 seconds (JVM running for 27.193)","logger_name":"payroll.employee.EmployeeApplication","thread_name":"main","level":"INFO","level_value":20000,"springAppName":"employee"}
Below it is the logstash.conf file
input {
file {
path => "C:/Users/User/Desktop/Git-Repos/Microservice/elk-logs/employee.log"
start_position => "beginning"
sincedb_path => "NUL"
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "logback-%{+YYYY.MM.dd}"
#user => "elastic"
#password => "changeme"
}
}
I run the logstash like below :
logstash.bat -f C:\Users\User\Desktop\Apps\logstash-7.6.2\config\logstash.conf
And the output is like below:
[2020-05-03T17:52:08,394][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}
[2020-05-03T17:52:08,456][INFO ][logstash.outputs.elasticsearch][main] Using default mapping template
[2020-05-03T17:52:08,519][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been created for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[2020-05-03T17:52:08,534][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["C:/Users/User/Desktop/Apps/logstash-7.6.2/config/logstash.conf"], :thread=>"#<Thread:0x6e5e96fc run>"}
[2020-05-03T17:52:08,597][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"#timestamp"=>{"type"=>"date"}, "#version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2020-05-03T17:52:10,327][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-05-03T17:52:10,380][INFO ][filewatch.observingtail ][main] START, creating Discoverer, Watch with file and sincedb collections
[2020-05-03T17:52:10,411][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-05-03T17:52:10,929][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
It seeems that the logstash starts correctly but it fails to insert the data in elasticsearch.
I really would appriciate any help.
Regards
P.S
I added output { stdout { codec => json } } and below it is result:
{"message":"{\"#timestamp\":\"2020-05-03T15:24:33.059+02:00\",\"#version\":1,\"message\":\"findAll called\",\"logger_name\":\"payroll.employee.service.EmployeeServiceImp\",\"thread_name\":\"http-nio-8082-exec-1\",\"level\":\"INFO\",\"level_value\":20000,\"springAppName\":\"employee\",\"traceId\":\"042059d1825b87b6\",\"spanId\":\"042059d1825b87b6\",\"spanExportable\":\"false\",\"X-Span-Export\":\"false\",\"X-B3-SpanId\":\"042059d1825b87b6\",\"X-B3-TraceId\":\"042059d1825b87b6\"}\r","#version":"1","#timestamp":"2020-05-03T17:22:09.423Z","path":"C:/Users/User/Desktop/Git-Repos/Microservice/elk-logs/employee.log","host":"WIN-7C4J6I3B56C"}{"message":"{\"#timestamp\":\"2020-05-03T15:09:34.106+02:00\",\"#version\":1,\"message\":\"Employee Application Started\",\"logger_name\":\"payroll.employee.EmployeeApplication\",\"thread_name\":\"main\",\"level\":\"INFO\",\"level_value\":20000}\r","#version":"1","#timestamp":"2020-05-03T17:22:09.389Z","path":"C:/Users/User/Desktop/Git-Repos/Microservice/elk-logs/employee.log","host":"WIN-7C4J6I3B56C"}{"message":"{\"#timestamp\":\"2020-05-03T15:25:04.625+02:00\",\"#version\":1,\"message\":\"countByDepartmentId called, departmentId: 1\",\"logger_name\":\"payroll.employee.service.EmployeeServiceImp\",\"thread_name\":\"http-nio-8082-exec-2\",\"level\":\"INFO\",\"level_value\":20000,\"springAppName\":\"employee\",\"traceId\":\"6a534061270b8b46\",\"spanId\":\"8b54f9e340a458a2\",\"spanExportable\":\"false\",\"X-Span-Export\":\"false\",\"X-B3-SpanId\":\"8b54f9e340a458a2\",\"X-B3-ParentSpanId\":\"30113ef7e385207d\",\"X-B3-TraceId\":\"6a534061270b8b46\",\"parentId\":\"30113ef7e385207d\"}\r","#version":"1","#timestamp":"2020-05-03T17:22:09.423Z","path":"C:/Users/User/Desktop/Git-Repos/Microservice/elk-logs/employee.log","host":"WIN-7C4J6I3B56C"}{"message":"{\"#timestamp\":\"2020-05-03T15:09:59.136+02:00\",\"#version\":1,\"message\":\"Started EmployeeApplication in 24.892 seconds (JVM running for 27.193)\",\"logger_name\":\"payroll.employee.EmployeeApplication\",\"thread_name\":\"main\",\"level\":\"INFO\",\"level_value\":20000,\"springAppName\":\"employee\"}\r","#version":"1","#timestamp":"2020-05-03T17:22:09.423Z","path":"C:/Users/User/Desktop/Git-Repos/Microservice/elk-logs/employee.log","host":"WIN-7C4J6I3B56C"}{"message":"{\"#timestamp\":\"2020-05-03T15:09:38.255+02:00\",\"#version\":1,\"message\":\"The following profiles are active: default\",\"logger_name\":\"payroll.employee.EmployeeApplication\",\"thread_name\":\"main\",\"level\":\"INFO\",\"level_value\":20000,\"springAppName\":\"employee\"}\r","#version":"1","#timestamp":"2020-05-03T17:22:09.408Z","path":"C:/Users/User/Desktop/Git-Repos/Microservice/elk-logs/employee.log","host":"WIN-7C4J6I3B56C"}{"message":"{\"#timestamp\":\"2020-05-03T15:25:04.709+02:00\",\"#version\":1,\"message\":\"countByDepartmentId called, departmentId: 2\",\"logger_name\":\"payroll.employee.service.EmployeeServiceImp\",\"thread_name\":\"http-nio-8082-exec-3\",\"level\":\"INFO\",\"level_value\":20000,\"springAppName\":\"employee\",\"traceId\":\"6a534061270b8b46\",\"spanId\":\"2391430ee2ce38cb\",\"spanExportable\":\"false\",\"X-Span-Export\":\"false\",\"X-B3-SpanId\":\"2391430ee2ce38cb\",\"X-B3-ParentSpanId\":\"0964d2e2fba9f7e5\",\"X-B3-TraceId\":\"6a534061270b8b46\",\"parentId\":\"0964d2e2fba9f7e5\"}\r","#version":"1","#timestamp":"2020-05-03T17:22:09.423Z","path":"C:/Users/User/Desktop/Git-Repos/Microservice/elk-logs/employee.log","host":"WIN-7C4J6I3B56C"}
Now i am more confused. The problem is with logstash or elasticsearch or kibana that I am not able to see any information in kibana?
As discussed in the chat. The problem was the pattern of the index name.
Changing the pattern to logback-* worked perfectly.
I am trying to insert data into elasticsearch using logstash but getting stuck. My config file:
logstashCrime.conf
input {
file {
path => "C:\elk\sampl.csv"
start_position => "beginning"
sincedb_path => "nul"
}
}
filter {
csv {
separator => ","
columns => ["code","name"]
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "crime"
}
stdout {
codec => rubydebug
}
}
I am getting a response like this when I try to insert using logstash-7.2.0\bin\logstash -f c:\elk\logstashCrime.conf:
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to C:/elk/logstash-7.2.0/logs which is now configured via log4j2.properties
[2019-07-15T16:10:22,300][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-07-15T16:10:22,320][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.2.0"}
[2019-07-15T16:10:28,817][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-07-15T16:10:29,009][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-07-15T16:10:29,058][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>7}
[2019-07-15T16:10:29,063][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2019-07-15T16:10:29,087][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-07-15T16:10:29,148][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2019-07-15T16:10:29,202][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"#timestamp"=>{"type"=>"date"}, "#version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2019-07-15T16:10:29,225][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been create for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[2019-07-15T16:10:29,229][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, :thread=>"#<Thread:0x74421f35 run>"}
[2019-07-15T16:10:30,202][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"main"}
[2019-07-15T16:10:30,408][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2019-07-15T16:10:30,416][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-07-15T16:10:30,755][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
My sampl.csv file looks like this:
id,name
------
1,john
2,doe
3,you
4,me
I am new to ELK. Any help appreciated. Also, I am using windows 10 as my OS. I successfully created an index using logstash withoug csv, but with csv it is not creating.
I want to view it in Kibana but since the index is not created, I cant see the index in Kibana.
You cannot use backslash in the path option of a file input. Use forward slash.
I have a log file looking like this
116.50.181.5 - - [18/May/2015:19:05:32 +0000] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/style2.css" "Mozilla/5.0 (X11; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0"
my logstash configuration is as below:
input {
file {
path => "C:\Users\PC\Documents\elk\Input\listening.txt"
start_position => "beginning"
}
}
filter {
grok {
match => {
"message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}'
}
}
date {
match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]
locale => en
}
geoip {
source => "clientip"
}
useragent {
source => "agent"
target => "useragent"
}
}
output {
elasticsearch
{
hosts => "http://localhost:9200"
index => "log"
}
}
everything works just fine, I have no errors in logstash , but the data does'nt appear in elasticsearch as expected.
C:\elk\logstash-7.1.1\bin>logstash -f logstashETL.conf
Sending Logstash logs to C:/elk/logstash-7.1.1/logs which is now configured via log4j2.properties
[2019-06-12T16:02:27,371][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-06-12T16:02:27,405][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.1.1"}
[2019-06-12T16:02:36,087][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-06-12T16:02:36,344][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-06-12T16:02:36,428][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>7}
[2019-06-12T16:02:36,428][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2019-06-12T16:02:36,469][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}
[2019-06-12T16:02:36,493][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2019-06-12T16:02:36,513][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, :thread=>"#<Thread:0x75642d2 run>"}
[2019-06-12T16:02:36,753][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"#timestamp"=>{"type"=>"date"}, "#version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2019-06-12T16:02:37,814][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"C:/elk/logstash-7.1.1/data/plugins/inputs/file/.sincedb_636c54fa423804cc695f80e1cb9d6ccd", :path=>["C:\\Users\\PC\\Documents\\elk\\Input\\listening.txt"]}
[2019-06-12T16:02:37,878][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"main"}
[2019-06-12T16:02:37,988][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-06-12T16:02:38,008][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2019-06-12T16:02:38,773][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
maybe there are somethings wrong or misssing in my code.
Add the below code in your input
start_position => "beginning"
sincedb_path => "/dev/null"
sincedb_path => "/dev/null" means it doesn't store sincedb files. These files are keeping byte ofset of where the logstash left on the file.
Then go to logstash/data/plugins/inputs/file directory. After that run below command at this directory
rm -r .sincedb*
Finally run your logstash pipeline. It should work.
I have used the zip files to start logstash, kibana and elasticsearch. I am ingesting a csv file from logstash to elastic search
input {
file {
path => "D:\tls202_part01\tls202_part01.csv"
start_position => "beginning"
}
}
filter {
csv {
separator => ","
columns => ["appln_id", "appln_title_lg", "appln_title"]
}
mutate {
convert => ["appln_id", "integer"]
convert => ["appln_title_lg", "string"]
convert => ["appln_title", "string"]
}
}
output {
elasticsearch {
hosts => "localhost"
index => "title"
}
stdout {
codec => rubydebug
}
}
this is my config file. When I search for index title it is not there and logstash logs are these:
Sending Logstash logs to D:/logstash-6.5.4/logs which is now configured via log4j2.properties
[2018-12-26T10:22:35,672][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-12-26T10:22:35,699][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.5.4"}
[2018-12-26T10:22:41,588][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-12-26T10:22:42,051][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2018-12-26T10:22:42,297][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2018-12-26T10:22:42,370][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-12-26T10:22:42,376][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-12-26T10:22:42,417][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost"]}
[2018-12-26T10:22:42,439][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-12-26T10:22:42,473][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"#timestamp"=>{"type"=>"date"}, "#version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-12-26T10:22:43,330][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"D:/logstash-6.5.4/data/plugins/inputs/file/.sincedb_bb5ff7ebd070422c5b611ac87e9e7087", :path=>["D:\\tls202_part01\\tls202_part01.csv"]}
[2018-12-26T10:22:43,390][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x389cc614 run>"}
[2018-12-26T10:22:43,499][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2018-12-26T10:22:43,532][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2018-12-26T10:22:43,842][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
The CSV file is large of 2GB csv data.
Also, kibana is showing no Elasticsearch data found for creating indexes.
It seems that logstash didn't found your file, change your path from backslash to forward slash and see if it works.
path => "D:/tls202_part01/tls202_part01.csv"