pushing data from logtsash to elasticserach - elasticsearch

Question

i had stored my configuration file of logstash in the same folder in which logstash is installed.
while trying to push the data from logstash to elasticsearch it is showing that server is started but data is not pushed to the elastic serach. how we can validate whether data is being pushed to elastic search or not.
this is my logstash configuration file.
input{
file{
path =>"C:\Elastic\GOOG.csv"
start_position =>"beginning"
}
}
filter{
csv{
columns =>
["date_of_record","open","high","low","close","volume","adj_close"]
separator => ","
}
date {
match => ["date_of_record","yyyy-MM-dd"]
}
mutate {
convert => ["open","float"]
convert => ["high","float"]
convert => ["low","float"]
convert => ["close","float"]
convert => ["volume","integer"]
convert => ["adj_close","float"]
}
}
output{
elasticsearch {
hosts => ["localhost:9200"]
index => "CSVGOGO"
}
}
Logstash Logs are:
c:\Elastic>.\logstash-7.0.0\bin\logstash -f .\gogo.conf
Sending Logstash logs to c:/Elastic/logstash-7.0.0/logs which is now configured via log4j2.properties
[2019-10-12T20:13:24,602][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-10-12T20:13:24,831][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.0.0"}
[2019-10-12T20:14:42,358][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-10-12T20:14:43,392][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-10-12T20:14:43,868][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>7}
[2019-10-12T20:14:43,882][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2019-10-12T20:14:43,961][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-10-12T20:14:43,971][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2019-10-12T20:14:44,124][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, :thread=>"#<Thread:0x22517e24 run>"}
[2019-10-12T20:14:44,604][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1, "index.lifecycle.name"=>"logstash-policy", "index.lifecycle.rollover_alias"=>"logstash"}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"#timestamp"=>{"type"=>"date"}, "#version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2019-10-12T20:14:48,863][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"c:/Elastic/logstash-7.0.0/data/plugins/inputs/file/.sincedb_1eb0c3bd994c60a8564bc344e0f91452", :path=>["C:\\Elastic\\GOOG.csv"]}
[2019-10-12T20:14:48,976][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"main"}
[2019-10-12T20:14:49,319][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2019-10-12T20:14:49,331][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-10-12T20:14:52,244][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

Answer 1

The data will be pushed in ES only if the data flow happened through reader and processor correctly.
Input: Try to make sure that the file is correctly read by the input filter.
Filter: Try writing a ruby processor that prints what data if got from the input.
Output: Write output in the console too to make sure it's as per your expectation.
Also, you can start Logstash in debug mode to get more info.

Answer 2

For ELK stack- to test if data is pushed to ES and if you have installed kibana follow below process
Explanation->
1.optional- Add stdout in logstash pipeline to show what is going on.
stdout { codec => rubydebug }
2.mandatory- Add sincedb_path => "/dev/null" in input/ file pipeline.
Logstash has an interesting component or feature called sincedb. Logstash keeps track of where it was last reading a file before it crashed or stopped.
3.mandatory- index name should be in lowercase (csvgogo)
4.optional/mandatory- document_type => "csvfile" if you dont add then default will be 'logs'
So your logstash output pipeline may look like the following:-
input{
file{
path =>"C:\Elastic\GOOG.csv"
start_position =>"beginning"
sincedb_path => "/dev/null"
}
}
filter{
csv{
columns => ["date_of_record","open","high","low","close","volume","adj_close"]
separator => ","
}
date {
match => ["date_of_record","yyyy-MM-dd"]
}
mutate {
convert => ["open","float"]
convert => ["high","float"]
convert => ["low","float"]
convert => ["close","float"]
convert => ["volume","integer"]
convert => ["adj_close","float"]
}
}
output{
elasticsearch {
hosts => ["localhost:9200"]
index => "csvgogo"
document_type => "csvfile" #default 'logs'
}
}
1.try with kibana's dev tool('http://localhost:5601/app/kibana') option to run query-
GET /csvgogo/_search
{
"query": {
"match_all": {}
}
}
2.try with Chrome browser- 'http://localhost:9200/csvgogo/_search?pretty'
where 'csvgogo' is your ES index name.
it will show you the raw data on browser itself from elastic search.