I'm trying to import a csv file to create data on my elasticsearch server in order to test it.
but I'm blocked to importing data using config file
this is a command (on winodws) logstash -f file.config
this is my config file
input{
file {
path => "/E:/Formation/kibana/data/cars.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter{
csv{
separator => ","
columns => ["maker","model","mileage","manufacture_year","engine_displacement",
"engine_power","body_type","color_slug","stk_year","transimission","door_count",
"seat_count","fuel_type","date_created","date_last_seen","price_eur"]
}
mutate {
convert => ["mileage","integer"]
convert => ["price_eur","float"]
convert => ["engine_power","integer"]
convert => ["door_count","integer"]
convert => ["seat_count","integer"]
}
}
output {
elasticsearch {
hosts => "localhost"
index => "cars"
document_type=> "sold_cars"
}
stdout { }
}
and this is the error
UPDATE this is log after using mode --debug thanks for helping
16:49:29.252 [Ruby-0-Thread-11: E:/Formation/kibana/logstash-5.4.0/logstash-core/lib/logstash/pipeline.rb:532] DEBUG logstash.pipeline - Pushing flush onto pipeline
16:49:34.257 [Ruby-0-Thread-11: E:/Formation/kibana/logstash-5.4.0/logstash-core/lib/logstash/pipeline.rb:532] DEBUG logstash.pipeline - Pushing flush onto pipeline
16:49:39.257 [Ruby-0-Thread-11: E:/Formation/kibana/logstash-5.4.0/logstash-core/lib/logstash/pipeline.rb:532] DEBUG logstash.pipeline - Pushing flush onto pipeline
16:49:43.663 [[main]<file] DEBUG logstash.inputs.file - _globbed_files: /e/Formation/kibana/data/cars.csv: glob is: []
On Windows, you should use sincedb_path => "nul" instead of sincedb_path => "/dev/null", which is used on Linux-based operating systems.
Related
I'm newbie for using Logstash and Elasticsearch. I wanted to sync my MongoDB data into Elasticsearch using Logstash Plugin (logstash-input-mongodb).
In my mongodata.conf is
input {
uri => 'mongodb://127.0.0.1:27017/final?ssl=true'
placeholder_db_dir => '/opt/logstash-mongodb/'
placeholder_db_name => 'logstash_sqlite.db'
collection => 'twitter_stream'
batch_size => 5000
}
filter {
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
action => "index"
index => "twitter_stream"
hosts => ["localhost:9200"]
}
}
While I running bin/logstash -f /etc/logstash/conf.d/mongodata.conf --path.settings /etc/logstash/
The error was displayed like this
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2020-02-28T08:48:20,246][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-02-28T08:48:20,331][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.6.0"}
[2020-02-28T08:48:20,883][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \t\r\n], "#", "{" at line 2, column 13 (byte 21) after input {\n uri ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:47:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:55:in compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:17:in block in compile_sources'", "org/jruby/RubyArray.java:2580:in map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:14:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:161:in initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:27:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:326:in block in converge_state'"]}
[2020-02-28T08:48:21,114][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2020-02-28T08:48:25,969][INFO ][logstash.runner ] Logstash shut down.
Please help me, I don't have any idea about this.
Your configuration is wrong, you need to specify what type of input you are using.
Try to change your input to this one:
input {
mongodb {
uri => 'mongodb://127.0.0.1:27017/final?ssl=true'
placeholder_db_dir => '/opt/logstash-mongodb/'
placeholder_db_name => 'logstash_sqlite.db'
collection => 'twitter_stream'
batch_size => 5000
}
}
I am using logstash 6.2.4 with following yml settings -
pipeline.batch.size: 600
pipeline.workers: 1
dead_letter_queue.enable: true
The conf file used to run logstash application is -
input {
file {
path => "/home/administrator/Downloads/postgresql.log.2018-10-17-06"
start_position => "beginning"
}
}
filter {
grok {
match => { "message" => "%{DATESTAMP:timestamp} %{TZ}:%{IP:uip}\(%{NUMBER:num}\):%{WORD:dbuser}%{GREEDYDATA:msg}"}
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
id => 'es-1'
hosts => ["localhost:9200"]
timeout => 60
index => "dlq"
version => "%{[#metadata][version]}"
version_type => "external_gte"
}
}
The input is a normal log file which is formatted using grok filter.
Here the version is always a string rather than a integer and thus elasticsearch throws error 400 Bad Request.
On this error code - logstash should retry a finite number of times and then should push this request payload to dead_letter_queue file (as per the documentation - https://www.elastic.co/guide/en/logstash/current/dead-letter-queues.html), but it gets stuck in an infinite loop with mesaage -
[2018-10-23T12:11:42,475][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"localhost:9200/_bulk"}
Following are the contents of data/dead_letter_queue/main directory -
1.log (contains a single value "1")
Please assist if any configuration is missing leading to this situation.
I am new on ELK and I need your help.
I would like to get some information about the cpu, memory. Those informative are generated every 30 minutes.
My xml file
<?xml version="1.0" encoding="UTF-8"?>
<measData>
<measInfo Id="SensorProcessingCounters">
<measType p="1">SensorsProcessed</measType>
<measValue xxxxxxxxx >
<r p="1">81</r>
</measValue>
</measInfo>
</measData>
My logstash file.conf
input {
file {
path => "/home/test/Desktop/data/file.xml"
start_position => beginning
sincedb_path => "/dev/null"
codec => multiline
{
pattern => "<measData>|</measData>"
negate => true
what => "previous"
}
}
}
filter
{
xml {
store_xml => false
source => "message"
xpath =>
["//measInfo[#measInfoId="SensorProcessingCounters"]/measValue/r[#p='1']/text()", "SensorProcessingCounters"
]
}
mutate{
convert => {
"SensorProcessingCounters"=> "float"}
}
}
output{
elasticsearch
{
action => "index"
hosts => ["localhost:9200"]
index => "stock"
}
stdout{}
}
error message
[2018-07-12T11:16:19,253][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-07-12T11:16:19,973][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.3.1"}
[2018-07-12T11:16:20,649][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, ,, ] at line 20, column 27 (byte 432) after filter\r\n{\r\nxml {\r\nstore_xml => false\r\nsource => \"message\"\r\nxpath =>\r\n[\"//measInfo[#measInfoId=\"", :backtrace=>["/home/test/Desktop/logstash-6.3.1/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'", "/home/test/Desktop/logstash-6.3.1/logstash-core/lib/logstash/compiler.rb:50:in `compile_graph'", "/home/test/Desktop/logstash-6.3.1/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "/home/test/Desktop/logstash-6.3.1/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'", "/home/test/Desktop/logstash-6.3.1/logstash-core/lib/logstash/pipeline.rb:49:in `initialize'", "/home/test/Desktop/logstash-6.3.1/logstash-core/lib/logstash/pipeline.rb:167:in `initialize'", "/home/test/Desktop/logstash-6.3.1/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'", "/home/test/Desktop/logstash-6.3.1/logstash-core/lib/logstash/agent.rb:305:in `block in converge_state'"]}
[2018-07-12T11:16:21,024][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
Thank you
For this line:
["//measInfo[#measInfoId="SensorProcessingCounters"]/measValue/r[#p='1']/text()",
"SensorProcessingCounters"
I guess you should use single quotes:
["//measInfo[#measInfoId='SensorProcessingCounters']/measValue/r[#p='1']/text()",
"SensorProcessingCounters"
because quotes mismatch.
I'm very new to logstash and elasticsearch, I am trying to stash my first log to logstash in a way that I can (correct me if it is not the purpose) search it using elasticsearch....
I have a log that looks like this basically:
2016-12-18 10:16:55,404 - INFO - flowManager.py - loading metadata xml
So, I have created a config file test.conf that looks like this:
input {
file {
path => "/home/usr/tmp/logs/mylog.log"
type => "test-type"
id => "NEWTRY"
}
}
filter {
grok {
match => { "message" => "%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second} - %{LOGLEVEL:level} - %{WORD:scriptName}.%{WORD:scriptEND} - " }
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "ecommerce"
codec => line { format => "%{year}-%{month}-%{day} %{hour}:%{minute}:%{second} - %{level} - %{scriptName}.%{scriptEND} - \"%{message}\"" }
}
}
And then : ./bin/logstash -f test.conf
I do not see the log in elastic search when I go to: http://localhost:9200/ecommerce OR to http://localhost:9200/ecommerce/test-type/NEWTRY
Please tell me what am I doing wrong.... :\
Thanks,
Heather
I found a solution eventually-
I added both sincedb_path=>"/dev/null" (which from what I understood is for testing enviorment only) and start_position => "beginning" to the output file plugin and the file appeared both in elastic and in kibana
Thanks anyway for responding and trying to help!
The code mentioned is my logstash conf file . I provide my nginx access log file as input and output to elasticsearch .I also write the output to a text file which works fine .. But the output is never been written to elasticsearch.
input {
file {
path => "filepath"
start_position => "beginning"
}
}
output {
file {
path => "filepath"
}
elasticsearch {
host => localhost
port => "9200"
}
}
I also tried executing logstash binary from command line using -e option
input { stdin{ } output { elasticsearch { host => localhost } }
which works fine. I get the output written to elasticsearch.. But in the former case i dont . Help me solve this
I tried a few things, I have no idea why your case with just host works. If I try it, i get timeouts. This is the configuration that works for me:
elasticsearch {
protocol => "http"
host => "localhost"
port => "9200"
}
I tried with logstash 1.4.2 and elasticsearch 1.4.4