ElasticSearch + Logstash works, but does not displays any data - elasticsearch

I have an Oracle DB. Logstash retrieves data from Oracle and puts it to ElasticSearch. And everything looks fine, but no changes occur on the Logstash server, as if it doesn't know what to do.
logstash.conf:
input {
jdbc {
jdbc_driver_library => "C:\JBoss\wildfly\...\ojdbc7.jar"
jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
jdbc_connection_string => "jdbc:oracle:thin:#3d-ztemtis-ora.iba:1521/ORCL"
jdbc_user => "sample_user"
jdbc_password => "12345"
jdbc_validate_connection => true
# once a 2 minute
schedule => "2 * * * *"
statement => "SELECT * FROM table_one"
}
}
output {
elasticsearch {
hosts => "localhost:9200"
index => "tableone"
document_id => "%{uid}"
}
stdout{
codec => rubydebug
}
}
Logstash logs
D:\Workspace3\ElasticLogstash\logstash-6.5.1>bin\logstash -f logstash.conf
Sending Logstash logs to D:/Workspace3/ElasticLogstash/logstash-6.5.1/logs which is now configured via log4j2.properties
[2018-11-28T00:49:30,296][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-11-28T00:49:30,308][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.5.1"}
[2018-11-28T00:49:33,174][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-11-28T00:49:33,455][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2018-11-28T00:49:33,471][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-11-28T00:49:33,625][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2018-11-28T00:49:33,674][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-11-28T00:49:33,674][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-11-28T00:49:33,699][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2018-11-28T00:49:33,718][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-11-28T00:49:33,745][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"#timestamp"=>{"type"=>"date"}, "#version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-11-28T00:49:33,940][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x64e24d22 run>"}
[2018-11-28T00:49:33,971][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2018-11-28T00:49:34,217][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
ElasticSearch log
[2018-11-28T00:36:06,492][DEBUG][o.e.a.ActionModule ] [px9stLj] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2018-11-28T00:36:06,683][INFO ][o.e.d.DiscoveryModule ] [px9stLj] using discovery type [zen] and host providers [settings]
[2018-11-28T00:36:07,188][INFO ][o.e.n.Node ] [px9stLj] initialized
[2018-11-28T00:36:07,188][INFO ][o.e.n.Node ] [px9stLj] starting ...
[2018-11-28T00:36:07,387][INFO ][o.e.t.TransportService ] [px9stLj] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2018-11-28T00:36:10,500][INFO ][o.e.c.s.MasterService ] [px9stLj] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {px9stLj}{px9stLjKSkqdyzudpK1ZhA}{bkR2txqXTn-Eo1o7-2PqEA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=17058418688, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
[2018-11-28T00:36:10,500][INFO ][o.e.c.s.ClusterApplierService] [px9stLj] new_master {px9stLj}{px9stLjKSkqdyzudpK1ZhA}{bkR2txqXTn-Eo1o7-2PqEA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=17058418688, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {px9stLj}{px9stLjKSkqdyzudpK1ZhA}{bkR2txqXTn-Eo1o7-2PqEA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=17058418688, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-11-28T00:36:10,585][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [px9stLj] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}
[2018-11-28T00:36:10,585][INFO ][o.e.n.Node ] [px9stLj] started
[2018-11-28T00:36:10,921][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [px9stLj] Failed to clear cache for realms [[]]
[2018-11-28T00:36:10,962][INFO ][o.e.l.LicenseService ] [px9stLj] license [852e276a-f99f-4ce3-a5d6-86c7769ae24e] mode [basic] - valid
[2018-11-28T00:36:10,970][INFO ][o.e.g.GatewayService ] [px9stLj] recovered [3] indices into cluster_state
[2018-11-28T00:36:12,366][INFO ][o.e.c.r.a.AllocationService] [px9stLj] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[blog][0]] ...]).
As I said, the problem is - nothing is happens and no errors logged.
How can I know is this succesfully connected to Oracle?

Please see the schedule examples here:
https://discuss.elastic.co/t/how-to-run-the-schedule-every-five-minutes-in-logstash-5-0/66222
https://www.thegeekstuff.com/2011/07/cron-every-5-minutes/
I think your schedule section should look like this:
Every 2 minutes
schedule => "*/2 * * * *"

Related

Unable to ingest XML file into Elastic Search using Logstash XML filter

I have this XML file which I stored in D:\ in Window 10:
<?xml version="1.0" encoding="UTF-8"?>
<root>
<ChainId>7290027600007</ChainId>
<SubChainId>001</SubChainId>
<StoreId>001</StoreId>
<BikoretNo>9</BikoretNo>
<DllVerNo>8.0.1.3</DllVerNo>
</root>
I have installed Elastic Search and able to access it at http://localhost:9200/. I have installed Logstash and created logstash-xml.conf to ingest the above XML file. In logstash-xml.conf, the configuration is below:
input
{
file
{
path => "D:\data.xml"
start_position => "beginning"
sincedb_path => "NUL"
exclude => "*.gz"
type => "xml"
codec => multiline {
pattern => "<?xml "
negate => "true"
what => "previous"
}
}
}
filter {
xml{
source => "message"
store_xml => false
target => "root"
xpath => [
"/root/ChainId/text()", "ChainId",
"/root/SubChainId/text()", "SubChainId",
"/root/StoreId/text()", "StoreId",
"/root/BikoretNo/text()", "BikoretNo",
"/root/DllVerNo/text()", "DllVerNo"
]
}
mutate {
gsub => [ "message", "[\r\n]", "" ]
}
}
output{
elasticsearch{
hosts => ["http://localhost:9200/"]
index => "parse_xml"
}
stdout
{
codec => rubydebug
}
}
When I run this configuration in the command line, I see this below:
D:\logstash\bin>logstash -f logstash-xml.conf
"Using bundled JDK: ""
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/C:/Users/CHEEWE~1.NGA/AppData/Local/Temp/jruby-3748/jruby14189572270520245744jopenssl.jar) to field java.security.MessageDigest.provider
WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Sending Logstash logs to D:/logstash/logs which is now configured via log4j2.properties
[2020-12-05T09:27:21,716][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.10.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +indy +jit [mswin32-x86_64]"}
[2020-12-05T09:27:22,053][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-12-05T09:27:24,031][INFO ][org.reflections.Reflections] Reflections took 37 ms to scan 1 urls, producing 23 keys and 47 values
[2020-12-05T09:27:26,083][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2020-12-05T09:27:26,311][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2020-12-05T09:27:26,378][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7}
[2020-12-05T09:27:26,383][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2020-12-05T09:27:26,437][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200/"]}
[2020-12-05T09:27:26,487][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[2020-12-05T09:27:26,621][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"#timestamp"=>{"type"=>"date"}, "#version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2020-12-05T09:27:28,152][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["D:/logstash/bin/logstash-xml.conf"], :thread=>"#<Thread:0x65f57880 run>"}
[2020-12-05T09:27:29,176][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>1.02}
[2020-12-05T09:27:29,640][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-12-05T09:27:29,712][INFO ][filewatch.observingtail ][main][aca15cd3c6850472d105bd7b2b7a43da8ce8ec36a4b0b8c19830d898f1eb1109] START, creating Discoverer, Watch with file and sincedb collections
[2020-12-05T09:27:29,726][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-12-05T09:27:30,020][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
But, going back to ElasticSearch, I can logstash index created, but I can't see the XML data loaded.

Logstash not inserting records in ElasticSearch

I am trying to insert simple CSV values to ES. it is not going thru.
The conf file i am using is as follows:
input {
file {
path => "D:/ELK/Logs/rahul.csv"
start_position => "beginning"
sincedb_path => "NULL"
}
}
filter {
csv {
columns => [ "record_id", "duration", "src_bytes", "dest_bytes" ]
}
}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "rahul"
}
stdout {}
}
I checked the error log on Logstash\logs folder. following is there
[2020-04-17T17:07:34,598][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-04-17T17:07:34,662][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.6.2"}
[2020-04-17T17:07:35,419][INFO ][org.reflections.Reflections] Reflections took 25 ms to scan 1 urls, producing 20 keys and 40 values
[2020-04-17T17:07:36,217][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2020-04-17T17:07:36,328][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2020-04-17T17:07:36,359][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7}
[2020-04-17T17:07:36,362][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2020-04-17T17:07:36,393][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}
[2020-04-17T17:07:36,423][INFO ][logstash.outputs.elasticsearch][main] Using default mapping template
[2020-04-17T17:07:36,443][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been created for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[2020-04-17T17:07:36,447][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>6, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>750, "pipeline.sources"=>["D:/ELK/Logs/rahul.conf"], :thread=>"#<Thread:0x5e5be359 run>"}
[2020-04-17T17:07:36,467][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"#timestamp"=>{"type"=>"date"}, "#version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2020-04-17T17:07:37,167][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-04-17T17:07:37,188][INFO ][filewatch.observingtail ][main] START, creating Discoverer, Watch with file and sincedb collections
[2020-04-17T17:07:37,205][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-04-17T17:07:37,395][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
resolved by adding sincedb_path => "NULL" to the input plugin, like so:
input {
file {
path => "D:/ELK/Logs/rahul.csv"
start_position => "beginning"
sincedb_path => "NULL"
}
}

How to enter data from csv to elasticsearch using logstash?

I am trying to insert data into elasticsearch using logstash but getting stuck. My config file:
logstashCrime.conf
input {
file {
path => "C:\elk\sampl.csv"
start_position => "beginning"
sincedb_path => "nul"
}
}
filter {
csv {
separator => ","
columns => ["code","name"]
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "crime"
}
stdout {
codec => rubydebug
}
}
I am getting a response like this when I try to insert using logstash-7.2.0\bin\logstash -f c:\elk\logstashCrime.conf:
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to C:/elk/logstash-7.2.0/logs which is now configured via log4j2.properties
[2019-07-15T16:10:22,300][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-07-15T16:10:22,320][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.2.0"}
[2019-07-15T16:10:28,817][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-07-15T16:10:29,009][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-07-15T16:10:29,058][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>7}
[2019-07-15T16:10:29,063][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2019-07-15T16:10:29,087][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-07-15T16:10:29,148][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2019-07-15T16:10:29,202][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"#timestamp"=>{"type"=>"date"}, "#version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2019-07-15T16:10:29,225][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been create for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[2019-07-15T16:10:29,229][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, :thread=>"#<Thread:0x74421f35 run>"}
[2019-07-15T16:10:30,202][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"main"}
[2019-07-15T16:10:30,408][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2019-07-15T16:10:30,416][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-07-15T16:10:30,755][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
My sampl.csv file looks like this:
id,name
------
1,john
2,doe
3,you
4,me
I am new to ELK. Any help appreciated. Also, I am using windows 10 as my OS. I successfully created an index using logstash withoug csv, but with csv it is not creating.
I want to view it in Kibana but since the index is not created, I cant see the index in Kibana.
You cannot use backslash in the path option of a file input. Use forward slash.

logstash configuration pipeline

I have a log file looking like this
116.50.181.5 - - [18/May/2015:19:05:32 +0000] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/style2.css" "Mozilla/5.0 (X11; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0"
my logstash configuration is as below:
input {
file {
path => "C:\Users\PC\Documents\elk\Input\listening.txt"
start_position => "beginning"
}
}
filter {
grok {
match => {
"message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}'
}
}
date {
match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]
locale => en
}
geoip {
source => "clientip"
}
useragent {
source => "agent"
target => "useragent"
}
}
output {
elasticsearch
{
hosts => "http://localhost:9200"
index => "log"
}
}
everything works just fine, I have no errors in logstash , but the data does'nt appear in elasticsearch as expected.
C:\elk\logstash-7.1.1\bin>logstash -f logstashETL.conf
Sending Logstash logs to C:/elk/logstash-7.1.1/logs which is now configured via log4j2.properties
[2019-06-12T16:02:27,371][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-06-12T16:02:27,405][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.1.1"}
[2019-06-12T16:02:36,087][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-06-12T16:02:36,344][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-06-12T16:02:36,428][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>7}
[2019-06-12T16:02:36,428][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2019-06-12T16:02:36,469][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}
[2019-06-12T16:02:36,493][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2019-06-12T16:02:36,513][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, :thread=>"#<Thread:0x75642d2 run>"}
[2019-06-12T16:02:36,753][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"#timestamp"=>{"type"=>"date"}, "#version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2019-06-12T16:02:37,814][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"C:/elk/logstash-7.1.1/data/plugins/inputs/file/.sincedb_636c54fa423804cc695f80e1cb9d6ccd", :path=>["C:\\Users\\PC\\Documents\\elk\\Input\\listening.txt"]}
[2019-06-12T16:02:37,878][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"main"}
[2019-06-12T16:02:37,988][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-06-12T16:02:38,008][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2019-06-12T16:02:38,773][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
maybe there are somethings wrong or misssing in my code.
Add the below code in your input
start_position => "beginning"
sincedb_path => "/dev/null"
sincedb_path => "/dev/null" means it doesn't store sincedb files. These files are keeping byte ofset of where the logstash left on the file.
Then go to logstash/data/plugins/inputs/file directory. After that run below command at this directory
rm -r .sincedb*
Finally run your logstash pipeline. It should work.

Logstash not creating indexes in Windows 10

I have used the zip files to start logstash, kibana and elasticsearch. I am ingesting a csv file from logstash to elastic search
input {
file {
path => "D:\tls202_part01\tls202_part01.csv"
start_position => "beginning"
}
}
filter {
csv {
separator => ","
columns => ["appln_id", "appln_title_lg", "appln_title"]
}
mutate {
convert => ["appln_id", "integer"]
convert => ["appln_title_lg", "string"]
convert => ["appln_title", "string"]
}
}
output {
elasticsearch {
hosts => "localhost"
index => "title"
}
stdout {
codec => rubydebug
}
}
this is my config file. When I search for index title it is not there and logstash logs are these:
Sending Logstash logs to D:/logstash-6.5.4/logs which is now configured via log4j2.properties
[2018-12-26T10:22:35,672][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-12-26T10:22:35,699][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.5.4"}
[2018-12-26T10:22:41,588][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-12-26T10:22:42,051][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2018-12-26T10:22:42,297][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2018-12-26T10:22:42,370][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-12-26T10:22:42,376][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-12-26T10:22:42,417][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost"]}
[2018-12-26T10:22:42,439][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-12-26T10:22:42,473][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"#timestamp"=>{"type"=>"date"}, "#version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-12-26T10:22:43,330][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"D:/logstash-6.5.4/data/plugins/inputs/file/.sincedb_bb5ff7ebd070422c5b611ac87e9e7087", :path=>["D:\\tls202_part01\\tls202_part01.csv"]}
[2018-12-26T10:22:43,390][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x389cc614 run>"}
[2018-12-26T10:22:43,499][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2018-12-26T10:22:43,532][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2018-12-26T10:22:43,842][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
The CSV file is large of 2GB csv data.
Also, kibana is showing no Elasticsearch data found for creating indexes.
It seems that logstash didn't found your file, change your path from backslash to forward slash and see if it works.
path => "D:/tls202_part01/tls202_part01.csv"

Resources