logstash not runs config - elasticsearch

I'm using filebeat on client side > logstash on serverside > elasticsearch on server side
filebeat on clientside works properly by sending file, but the configuration i've made on logstash returning
Fail
[WARN ] 2019-12-18 14:53:30.987 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[FATAL] 2019-12-18 14:53:31.341 [LogStash::Runner] runner - Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.
[ERROR] 2019-12-18 14:53:31.364 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
Here is my configfile
input {
beats {
port =>5044
}
}
filter {
grok {
match => { "message" =>"%{TIMESTAMP_ISO8601:timestamp}] %{WORD:test}\[%{NUMBER:nom}]\[%{DATA:tes}\] %{DATA:module_name}\: %{WORD:method}%{GREEDYDATA:log_message}" }
}
}
output {
elasticsearch
{
hosts => "127.0.0.1:9200"
index=>"test_log_pbx"
}
}
code to run my logstash config
/usr/share/logstash/bin/logstash -f logstash.conf
when i run configtest it returns
Thread.exclusive is deprecated, use Thread::Mutex
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2019-12-18 14:59:53.300 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-12-18 14:59:56.566 [LogStash::Runner] Reflections - Reflections took 139 ms to scan 1 urls, producing 20 keys and 40 values
Configuration OK
[INFO ] 2019-12-18 14:59:57.923 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
please help me i dont know whats wrong

A logstash instance already running, so you can not run another instance.If you made your logstash as service, you should stop the service. If you want to run multiple instances, you should modify pipelines.yml
If you want to learn more about pipelines.yml, I put link the below.
https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html

Related

Unable to read CSV file in logstash as its shutting down on its own

I have elasticsearch and Kibana are up and running and I want to read logs using logstash so for that I have passed csv file as an input in logstash.conf file but its not reading logs and shutting down automatically.
This is how I am running logstash command:
D:\logstash-8.1.0\bin>logstash -f "D:/logstash.conf"
logstash.conf
input{
file{
path => "D:/unicorn.csv"
start_position => beginning
}
}
output{
elasticsearch{
hosts => "localhost:9200"
index => "indexforlogstash"
}
stdout{}
}
Below are the terminal output:
"Using bundled JDK: ."
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and
will likely be removed in a future release.
Sending Logstash logs to D:/logstash-8.1.0/logs which is now configured via log4j2.properties
[2022-03-16T12:59:47,905][INFO ][logstash.runner ] Log4j configuration path used is:
D:\logstash-8.1.0\config\log4j2.properties
[2022-03-16T12:59:47,938][WARN ][logstash.runner ] The use of JAVA_HOME has been d
deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running
Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with
each specific version of Logstash, and generally provides best performance and reliability. If
you have compelling reasons for using your own JDK (organizational-specific compliance
requirements, for example), you can configure LS_JAVA_HOME to use that version instead.
[2022-03-16T12:59:47,942][INFO ][logstash.runner ] Starting Logstash
{"logstash.version"=>"8.1.0", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1
OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [mswin32-x86_64]"}
[2022-03-16T12:59:47,947][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -
Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -
XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -
Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true,
-XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -
Dlog4j2.isThreadContextMapInheritable=true, --add-opens=java.base/java.security=ALL-UNNAMED, --
add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, -
-add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-
UNNAMED]
[2022-03-16T12:59:48,058][INFO ][logstash.settings ] Creating directory
{:setting=>"path.queue", :path=>"D:/logstash-8.1.0/data/queue"}
[2022-03-16T12:59:48,104][INFO ][logstash.settings ] Creating directory
{:setting=>"path.dead_letter_queue", :path=>"D:/logstash-8.1.0/data/dead_letter_queue"}
[2022-03-16T12:59:48,285][WARN ][logstash.config.source.multilocal] Ignoring the
'pipelines.yml' file because modules or command line options are specified
[2022-03-16T12:59:48,347][INFO ][logstash.agent ] No persistent UUID file found.
Generating new UUID {:uuid=>"84410117-2fa7-499b-b55a-43a29192540e", :path=>"D:/logstash-
8.1.0/data/uuid"}
[2022-03-16T12:59:55,063][ERROR][logstash.config.sourceloader] No configuration found in the
configured sources.
[2022-03-16T12:59:55,424][INFO ][logstash.agent ] Successfully started Logstash API
endpoint {:port=>9600, :ssl_enabled=>false}
[2022-03-16T13:00:00,591][INFO ][logstash.runner ] Logstash shut down.
[2022-03-16T13:00:00,609][FATAL][org.logstash.Logstash ] Logstash stopped processing
because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby.jar:?]
at D_3a_.logstash_minus_8_dot_1_dot_0.lib.bootstrap.environment.<main>(D:\logstash-
8.1.0\lib\bootstrap\environment.rb:94) ~[?:?]
Someone let me know what I am doing wrong.
I believe the error is due to slash. in the config code
path => "D:\unicorn.csv"
also while executing
logstash -f "D:\logstash.conf"
I would also recommend to use this below command to check if there are any syntax errors in your code
logstash --config.test_and_exit -f "D:\logstash.conf"
Keep posted!!!Thanks!!!
In the output field you can add rubydebug to get to know the inputs whichever has been parsed logstash as an input . This will print the parsed data in command prompt
output{stdout{
codec => rubydebug
}
}
Error says that logstash is unable to locate configuration at location provided.
To solve the error follow below steps:
Modify file path in input as follows
input{
file{
path => "D:\\unicorn.csv"
start_position => beginning
}
}
Also run configuration with logstash as follows
D:\logstash-8.1.0\bin>logstash -f "D:\\logstash.conf"

I get error with Logstash: IllegalStateException

Here is my logstash.conf:
input {
mongodb{
uri => 'mongodb://localhost:27017/vnExpressCrawler?authSource=vnExpressCrawler'
placeholder_db_dir => '/opt/logstash-mongodb/'
placeholder_db_name => 'logstash_sqlite.db'
collection => 'articles'
batch_size => 5000
}
}
filter {
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
action => "index"
index => "mongo_log_data"
hosts => ["localhost:9200"]
}
}
I get error:
Error: Don't know how to handle Java::JavaLang::IllegalStateException for
PipelineAction::Create<main>
create at org/logstash/execution/ConvergeResultExt.java:135
add at org/logstash/execution/ConvergeResultExt.java:60
when I run logstash -f logstash.conf
Can someone help me solve this?
My Log:
Using JAVA_HOME defined java: C:\Program Files\Java\jdk1.8.0_251
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK
Sending Logstash logs to C:/Dev Programs/Logstash/logstash-7.13.2/logs which is now configured via log4j2.properties
[2021-06-21T14:21:04,709][INFO ][logstash.runner ] Log4j configuration path used is: C:\Dev Programs\Logstash\logstash-7.13.2\config\log4j2.properties
[2021-06-21T14:21:04,723][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.13.2", "jruby.version"=>"jruby 9.2.16.0 (2.5.7) 2021-03-03 f82228dc32 Java HotSpot(TM) 64-Bit Server VM 25.251-b08 on 1.8.0_251-b08 +indy +jit [mswin32-x86_64]"}
[2021-06-21T14:21:04,867][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-06-21T14:21:06,405][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2021-06-21T14:21:07,828][INFO ][org.reflections.Reflections] Reflections took 81 ms to scan 1 urls, producing 24 keys and 48 values
[2021-06-21T14:21:08,619][ERROR][logstash.plugins.registry] Unable to load plugin. {:type=>"input", :name=>"mongodb"}
[2021-06-21T14:21:08,635][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any input plugin named 'mongodb'. Are you sure this is correct? Trying to load the mongodb input plugin resulted in this error: Unable to load the requested plugin named mongodb of type input. The plugin is not installed.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:119)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:86)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "C_3a_.Dev_20_Programs.Logstash.logstash_minus_7_dot_13_dot_2.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(C:/Dev Programs/Logstash/logstash-7.13.2/logstash-core/lib/logstash/java_pipeline.rb:47)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "C_3a_.Dev_20_Programs.Logstash.logstash_minus_7_dot_13_dot_2.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(C:/Dev Programs/Logstash/logstash-7.13.2/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "C_3a_.Dev_20_Programs.Logstash.logstash_minus_7_dot_13_dot_2.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(C:/Dev Programs/Logstash/logstash-7.13.2/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "C_3a_.Dev_20_Programs.Logstash.logstash_minus_7_dot_13_dot_2.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(C:/Dev Programs/Logstash/logstash-7.13.2/logstash-core/lib/logstash/agent.rb:389)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.lang.Thread.run(Thread.java:748)"]}
warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true):
LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`
create at org/logstash/execution/ConvergeResultExt.java:135
add at org/logstash/execution/ConvergeResultExt.java:60
converge_state at C:/Dev Programs/Logstash/logstash-7.13.2/logstash-core/lib/logstash/agent.rb:402
[2021-06-21T14:21:08,649][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`"}
[2021-06-21T14:21:08,655][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:135:in `create'", "org/logstash/execution/ConvergeResultExt.java:60:in `add'", "C:/Dev Programs/Logstash/logstash-7.13.2/logstash-core/lib/logstash/agent.rb:402:in `block in converge_state'"]}
[2021-06-21T14:21:08,679][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.16.0.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.16.0.jar:?]
at C_3a_.Dev_20_Programs.Logstash.logstash_minus_7_dot_13_dot_2.lib.bootstrap.environment.<main>(C:\Dev Programs\Logstash\logstash-7.13.2\lib\bootstrap\environment.rb:89) ~[?:?]
The problem is stated in the error logs
Couldn't find any input plugin named 'mongodb'. Are you sure this is correct? Trying to load the mongodb input plugin resulted in this error: Unable to load the requested plugin named mongodb of type input. The plugin is not installed.
So you need to make sure to first properly install the mongodb input plugin before running Logstash.

how do i fix Docker ELK stack logstash import connection error

I'm having an issue trying to import a CSV file into my elastic search instance using log stash.
i'm using the pre-configured Docker ELK stack.
the error i recive when i run the command is as follows :
D:\data_sets\logstac config>docker cp CVEimport.conf 80:/
D:\data_sets\logstac config>docker exec -it 80 /bin/bash
bash-4.2$ logstash -f /CVEimport.conf --path.data ./storage/
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.9.0.jar) to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2020-04-03T12:35:08,901][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-04-03T12:35:08,921][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.6.0"}
[2020-04-03T12:35:10,957][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx#elasticsearch:9200/]}}
[2020-04-03T12:35:12,083][WARN ][logstash.licensechecker.licensereader] Restored connection to ES instance {:url=>"http://elastic:xxxxxx#elasticsearch:9200/"}
[2020-04-03T12:35:12,333][INFO ][logstash.licensechecker.licensereader] ES Output version determined {:es_version=>7}
[2020-04-03T12:35:12,349][WARN ][logstash.licensechecker.licensereader] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2020-04-03T12:35:13,001][INFO ][logstash.monitoring.internalpipelinesource] Monitoring License OK
[2020-04-03T12:35:13,007][INFO ][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline.
[2020-04-03T12:35:15,226][INFO ][org.reflections.Reflections] Reflections took 187 ms to scan 1 urls, producing 20 keys and 40 values
[2020-04-03T12:35:16,318][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx#127.0.0.1:9200/]}}
[2020-04-03T12:35:16,375][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elastic:xxxxxx#127.0.0.1:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elastic:xxxxxx#127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
it then continues trying to reconnect.
my config file is :
input {
file {
path => ["/CVEDB1.csv"]
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
columns=> ["Name","Status","Description","References","Phase","Votes","Comments"]
separator => ","
}
}
output {
elasticsearch {
hosts => "http://127.0.0.1:9200"
index => "cve"
user => "elastic"
password => "changeme"
}
stdout{}
}
Any information will be helpful as i am completely new to the ELK stack.
Do you have the Elasticsearch REST service running?
# are we green?
curl http://localhost:9200/_cluster/health?pretty
If you want to test logstash standalone, you can send the output to the console
# output to console
output {
stdout { codec => rubydebug }
}
Note. Username and password are not needed as long as you haven't secured your Elasticsearch cluster.
Note. When running docker
container(localhost) != host(localhost).
You can run Elasticsearch in host_mode
Utilize docker-compose and service discovery is established.
curl http://172.16.0.2/_cluster/health?pretty (docker0 subnet)

Loading logs in one machine into elasticsearch located setup in another machine using logstash

I have my logs and logstash running on the one EC2 machine (M1), so I read my logs placed on my local machine with this config:
input {
file{
path => "/path/to/logs/in/M1"
start_position => "beginning"
}
}
Now, we have elasticsearch running on a different EC2 machine (M2) and I need to transfer the logs from M1 to elasticsearch in M2 using logstash. I used the following output config:
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => "http://<M2 ip address>:9200"
index => "logstash-%{+YYYY.MM.dd}"
}
}
When I run the config file, I get the following error:
04:18:57.640 [[main]>worker0] WARN logstash.outputs.elasticsearch - UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}
04:18:57.646 [[main]>worker0] ERROR logstash.outputs.elasticsearch - Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>2}
04:18:59.682 [[main]>worker0] WARN logstash.outputs.elasticsearch - UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}
04:18:59.686 [[main]>worker0] ERROR logstash.outputs.elasticsearch - Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>4}
04:19:01.109 [Ruby-0-Thread-17: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:188] WARN logstash.outputs.elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x1d08c988 URL:http://10.60.40.120:9200>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://10.60.40.120:9200][Manticore::ConnectTimeout] connect timed out"}
04:19:02.111 [Ruby-0-Thread-17: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:188] INFO logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:url=>#<URI::HTTP:0x55444fcf URL:http://10.60.40.120:9200>, :healthcheck_path=>"/"}
I am new to logstash. Any help is appreciated.
UPDATE:
So I looked around in forumns and I got one solution which told me to update logstash output using the command:
sudo /usr/share/logstash/bin/logstash-plugin update logstash-output-elasticsearch
I also updated the logstash config file to include username and password:
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => ["<M2 ip address>"]
user => 'username'
password => 'changeme'
index => "logstash-%{+YYYY.MM.dd}"
manage_template => false
}
}
Now I'm getting a different error. Pleas help:
09:16:21.305 [[main]>worker0] WARN logstash.outputs.elasticsearch - Could not index event to Elasticsearch. {:status=>404, :action=>["index", {:_id=>nil, :_index=>"logstash-2017.04.17", :_type=>"Messagelog", :_routing=>nil}, 2017-04-17T10:06:11.348Z ip-10-60-40-201 No valid licenses found for COLL], :response=>{"index"=>{"_index"=>"logstash-2017.04.17", "_type"=>"Messagelog", "_id"=>nil, "status"=>404, "error"=>{"type"=>"index_not_found_exception", "reason"=>"no such index and [action.auto_create_index] ([.security,.monitoring*,.watches,.triggered_watches,.watcher-history*]) doesn't match", "index_uuid"=>"_na_", "index"=>"logstash-2017.04.17"}}}}
Thanks.
It looks like you have disable auto creation of index on elasticsearch. By default elasticsearch supports auto creation of indexes.
Remove
action.auto_create_index: -b*,+a*,-*
(whatever the pattern) in your elasticsearch.yml and you will be good.
Furthermore if you want to accept auto creation of indexes starting with l used the pattern +l*. That is by adding
action.auto_create_index: +l*
Read this for additional informations.

Error reading field 'topic_metadata': Error reading array of size 873589, only 41 bytes available

I have installed logstash version 5.2.2 by downloading zip file in a VM having fresh Ubuntu installed in it.
I have created a sample config file logstash-sample.conf with the following entry
input{
stdin{ }
}
output{
stdout{ }
}
And executing the command $bin/logstash -f logstash-simple.conf
it is running absolutely fine.
Now in the same Ubuntu machine, I installed kafka by following the exact same process mentioned
here https://www.digitalocean.com/community/tutorials/how-to-install-apache-kafka-on-ubuntu-14-04 and followed till step no 7.
Then I modified the logstash-sample.conf file to contain the following
input {
kafka{
bootstrap_servers => "localhost:9092"
topics => ["TutorialTopic"]
}
}
output {
stdout { codec => rubydebug }
}
And this time I am getting the following error,
sample#sample-VirtualBox:~/Downloads/logstash-5.2.2$ bin/logstash -f logstash-sample.conf
Sending Logstash's logs to /home/rs-switch/Downloads/logstash-5.2.2/logs which is now configured via log4j2.properties
[2017-03-07T00:26:25,629][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>125}
[2017-03-07T00:26:25,650][INFO ][logstash.pipeline ] Pipeline main started
[2017-03-07T00:26:26,039][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
log4j:WARN No appenders could be found for logger (org.apache.kafka.clients.consumer.ConsumerConfig).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "Ruby-0-Thread-14: /home/rs-switch/Downloads/logstash-5.2.2/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.6/lib/logstash/inputs/kafka.rb:229" org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'topic_metadata': Error reading array of size 873589, only 41 bytes available
at org.apache.kafka.common.protocol.types.Schema.read(org/apache/kafka/common/protocol/types/Schema.java:73)
at org.apache.kafka.clients.NetworkClient.parseResponse(org/apache/kafka/clients/NetworkClient.java:380)
at org.apache.kafka.clients.NetworkClient.handleCompletedReceives(org/apache/kafka/clients/NetworkClient.java:449)
at org.apache.kafka.clients.NetworkClient.poll(org/apache/kafka/clients/NetworkClient.java:269)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java:360)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java:224)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java:192)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java:163)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(org/apache/kafka/clients/consumer/internals/AbstractCoordinator.java:179)
at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(org/apache/kafka/clients/consumer/KafkaConsumer.java:974)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(org/apache/kafka/clients/consumer/KafkaConsumer.java:938)
at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
at RUBY.thread_runner(/home/rs-switch/Downloads/logstash-5.2.2/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.6/lib/logstash/inputs/kafka.rb:239)
at java.lang.Thread.run(java/lang/Thread.java:745)
[2017-03-07T00:26:28,742][WARN ][logstash.agent ] stopping pipeline {:id=>"main"}
Can anyone help me out resolving this issue? I am really stuck setting up ELK from last few weeks, but was not successful.
You most probably have a version conflict that is causing this issue. Check out the compatibility matrix in the Logstash Kafka input plugins documentation.
The link you mentioned for installing Kafka has you install version 0.8.2.1 which will not work with Kafka 0.10 clients. Kafka has version checking and backwards compatibility, but only if the broker is newer than the client, which is not the case here.
I'd recommend installing a current version of Kafka, there have been immense improvements since version 0.8 that you'd be missing out on if you tried downgrading Logstash instead.
Check out the Confluent Platform Quickstart for an easy way to get started.

Resources