I get error with Logstash: IllegalStateException - elasticsearch

Here is my logstash.conf:
input {
mongodb{
uri => 'mongodb://localhost:27017/vnExpressCrawler?authSource=vnExpressCrawler'
placeholder_db_dir => '/opt/logstash-mongodb/'
placeholder_db_name => 'logstash_sqlite.db'
collection => 'articles'
batch_size => 5000
}
}
filter {
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
action => "index"
index => "mongo_log_data"
hosts => ["localhost:9200"]
}
}
I get error:
Error: Don't know how to handle Java::JavaLang::IllegalStateException for
PipelineAction::Create<main>
create at org/logstash/execution/ConvergeResultExt.java:135
add at org/logstash/execution/ConvergeResultExt.java:60
when I run logstash -f logstash.conf
Can someone help me solve this?
My Log:
Using JAVA_HOME defined java: C:\Program Files\Java\jdk1.8.0_251
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK
Sending Logstash logs to C:/Dev Programs/Logstash/logstash-7.13.2/logs which is now configured via log4j2.properties
[2021-06-21T14:21:04,709][INFO ][logstash.runner ] Log4j configuration path used is: C:\Dev Programs\Logstash\logstash-7.13.2\config\log4j2.properties
[2021-06-21T14:21:04,723][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.13.2", "jruby.version"=>"jruby 9.2.16.0 (2.5.7) 2021-03-03 f82228dc32 Java HotSpot(TM) 64-Bit Server VM 25.251-b08 on 1.8.0_251-b08 +indy +jit [mswin32-x86_64]"}
[2021-06-21T14:21:04,867][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-06-21T14:21:06,405][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2021-06-21T14:21:07,828][INFO ][org.reflections.Reflections] Reflections took 81 ms to scan 1 urls, producing 24 keys and 48 values
[2021-06-21T14:21:08,619][ERROR][logstash.plugins.registry] Unable to load plugin. {:type=>"input", :name=>"mongodb"}
[2021-06-21T14:21:08,635][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any input plugin named 'mongodb'. Are you sure this is correct? Trying to load the mongodb input plugin resulted in this error: Unable to load the requested plugin named mongodb of type input. The plugin is not installed.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:119)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:86)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "C_3a_.Dev_20_Programs.Logstash.logstash_minus_7_dot_13_dot_2.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(C:/Dev Programs/Logstash/logstash-7.13.2/logstash-core/lib/logstash/java_pipeline.rb:47)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "C_3a_.Dev_20_Programs.Logstash.logstash_minus_7_dot_13_dot_2.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(C:/Dev Programs/Logstash/logstash-7.13.2/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "C_3a_.Dev_20_Programs.Logstash.logstash_minus_7_dot_13_dot_2.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(C:/Dev Programs/Logstash/logstash-7.13.2/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "C_3a_.Dev_20_Programs.Logstash.logstash_minus_7_dot_13_dot_2.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(C:/Dev Programs/Logstash/logstash-7.13.2/logstash-core/lib/logstash/agent.rb:389)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.lang.Thread.run(Thread.java:748)"]}
warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true):
LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`
create at org/logstash/execution/ConvergeResultExt.java:135
add at org/logstash/execution/ConvergeResultExt.java:60
converge_state at C:/Dev Programs/Logstash/logstash-7.13.2/logstash-core/lib/logstash/agent.rb:402
[2021-06-21T14:21:08,649][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`"}
[2021-06-21T14:21:08,655][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:135:in `create'", "org/logstash/execution/ConvergeResultExt.java:60:in `add'", "C:/Dev Programs/Logstash/logstash-7.13.2/logstash-core/lib/logstash/agent.rb:402:in `block in converge_state'"]}
[2021-06-21T14:21:08,679][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.16.0.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.16.0.jar:?]
at C_3a_.Dev_20_Programs.Logstash.logstash_minus_7_dot_13_dot_2.lib.bootstrap.environment.<main>(C:\Dev Programs\Logstash\logstash-7.13.2\lib\bootstrap\environment.rb:89) ~[?:?]

The problem is stated in the error logs
Couldn't find any input plugin named 'mongodb'. Are you sure this is correct? Trying to load the mongodb input plugin resulted in this error: Unable to load the requested plugin named mongodb of type input. The plugin is not installed.
So you need to make sure to first properly install the mongodb input plugin before running Logstash.

Related

Bad configuration in Logstash - Windows 11

I try execute input jdbc oracle with tracking column, but don't work, need other information? like file.conf, etc.?
D:\Elastic\logstash-8.6.0>.\bin\logstash.bat -f D:\Elastic\logstash-8.6.0\config\conf.d\bo_firmas_electronicas.conf
Using LS_JAVA_HOME defined java: C:\Program Files\Java\jdk-19
"WARNING: Logstash comes bundled with the recommended JDK(), but is overridden by the version defined in LS_JAVA_HOME. Consider clearing LS_JAVA_HOME to use the bundled JDK."
Sending Logstash logs to D:\Elastic\logstash-8.6.0\logs which is now configured via log4j2.properties
[2023-01-13T15:00:09,087][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ArgumentError) Cannot determine timezone from nil\n(secs:1673640004.484,utc~:\"2023-01-13 20:00:04.48399996757507324\",ltz~:nil)\n(etz:nil,tnz:\"PET\",tziv:\"2.0.5\",tzidv:\"1.2022.6\",rv:\"2.6.8\",rp:\"java\",win:true,rorv:nil,astz:nil,eov:\"1.2.7\",eotnz:\"???\",eotnfz:\"???\",eotlzn:\"???\",\ndebian:nil,centos:nil,osx:nil)\nTry setting `ENV['TZ'] = 'Continent/City'` in your script (see https://en.wikipedia.org/wiki/List_of_tz_database_time_zones)", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:181)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:846)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1229)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:131)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:329)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.RubyClass.newInstance(RubyClass.java:911)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:329)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:549)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:92)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:226)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:393)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:206)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:325)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:143)", "org.jruby.RubyProc.call(RubyProc.java:309)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:107)", "java.base/java.lang.Thread.run(Thread.java:1589)"]}
[2023-01-13T15:00:09,121][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?]
at D_3a_.Elastic.logstash_minus_8_dot_6_dot_0.lib.bootstrap.environment.<main>(D:\Elastic\logstash-8.6.0\lib\bootstrap\environment.rb:91) ~[?:?]
D:\Elastic\logstash-8.6.0>
D:\Elastic\logstash-8.6.0>.\bin\logstash.bat -f D:\Elastic\logstash-8.6.0\config\conf.d\bo_firmas_electronicas.conf
Using LS_JAVA_HOME defined java: C:\Program Files\Java\jdk-19
"WARNING: Logstash comes bundled with the recommended JDK(), but is overridden by the version defined in LS_JAVA_HOME. Consider clearing LS_JAVA_HOME to use the bundled JDK."
Sending Logstash logs to D:\Elastic\logstash-8.6.0\logs which is now configured via log4j2.properties
[2023-01-13T15:02:00,714][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ArgumentError) Cannot determine timezone from nil\n(secs:1673640115.819,utc~:\"2023-01-13 20:01:55.8190000057220459\",ltz~:nil)\n(etz:nil,tnz:\"PET\",tziv:\"2.0.5\",tzidv:\"1.2022.6\",rv:\"2.6.8\",rp:\"java\",win:true,rorv:nil,astz:nil,eov:\"1.2.7\",eotnz:\"???\",eotnfz:\"???\",eotlzn:\"???\",\ndebian:nil,centos:nil,osx:nil)\nTry setting `ENV['TZ'] = 'Continent/City'` in your script (see https://en.wikipedia.org/wiki/List_of_tz_database_time_zones)", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:181)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:846)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1229)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:131)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:329)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.RubyClass.newInstance(RubyClass.java:911)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:329)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:549)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:92)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:226)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:393)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:206)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:325)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:143)", "org.jruby.RubyProc.call(RubyProc.java:309)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:107)", "java.base/java.lang.Thread.run(Thread.java:1589)"]}
[2023-01-13T15:02:00,748][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?]
at D_3a_.Elastic.logstash_minus_8_dot_6_dot_0.lib.bootstrap.environment.<main>(D:\Elastic\logstash-8.6.0\lib\bootstrap\environment.rb:91) ~[?:?]`
According to your logs, your problem is related with timestamp issue :
Unable to configure plugins: (ArgumentError) Cannot determine timezone from nil\n(secs:1673640004.484,utc~:\"2023-01-13 20:00:04.48399996757507324\",ltz~:nil)\n(etz:nil,tnz:\"PET\",tziv:\"2.0.5\",tzidv:\"1.2022.6\",rv:\"2.6.8\",rp:\"java\",win:true,rorv:nil,astz:nil,eov:\"1.2.7\",eotnz:\"???\",eotnfz:\"???\",eotlzn:\"???\",\ndebian:nil,centos:nil,osx:nil)\nTry setting `ENV['TZ'] = 'Continent/City'` in your script (see https://en.wikipedia.org/wiki/List_of_tz_database_time_zones)
And, you can easily find similar questions are here:
https://discuss.elastic.co/t/i-get-error-unable-to-configure-plugins-argumenterror-cannot-determine-timezone-from-nil/319531
Logstash 8.0 #timestamp with nanoseconds precision
https://discuss.elastic.co/t/nanoseconds-with-logstash-and-elasticsearch/282492
So, I think this answer you need to put inside the your pipeline too:
https://stackoverflow.com/a/70804574/721600

Unable to read CSV file in logstash as its shutting down on its own

I have elasticsearch and Kibana are up and running and I want to read logs using logstash so for that I have passed csv file as an input in logstash.conf file but its not reading logs and shutting down automatically.
This is how I am running logstash command:
D:\logstash-8.1.0\bin>logstash -f "D:/logstash.conf"
logstash.conf
input{
file{
path => "D:/unicorn.csv"
start_position => beginning
}
}
output{
elasticsearch{
hosts => "localhost:9200"
index => "indexforlogstash"
}
stdout{}
}
Below are the terminal output:
"Using bundled JDK: ."
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and
will likely be removed in a future release.
Sending Logstash logs to D:/logstash-8.1.0/logs which is now configured via log4j2.properties
[2022-03-16T12:59:47,905][INFO ][logstash.runner ] Log4j configuration path used is:
D:\logstash-8.1.0\config\log4j2.properties
[2022-03-16T12:59:47,938][WARN ][logstash.runner ] The use of JAVA_HOME has been d
deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running
Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with
each specific version of Logstash, and generally provides best performance and reliability. If
you have compelling reasons for using your own JDK (organizational-specific compliance
requirements, for example), you can configure LS_JAVA_HOME to use that version instead.
[2022-03-16T12:59:47,942][INFO ][logstash.runner ] Starting Logstash
{"logstash.version"=>"8.1.0", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1
OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [mswin32-x86_64]"}
[2022-03-16T12:59:47,947][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -
Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -
XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -
Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true,
-XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -
Dlog4j2.isThreadContextMapInheritable=true, --add-opens=java.base/java.security=ALL-UNNAMED, --
add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, -
-add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-
UNNAMED]
[2022-03-16T12:59:48,058][INFO ][logstash.settings ] Creating directory
{:setting=>"path.queue", :path=>"D:/logstash-8.1.0/data/queue"}
[2022-03-16T12:59:48,104][INFO ][logstash.settings ] Creating directory
{:setting=>"path.dead_letter_queue", :path=>"D:/logstash-8.1.0/data/dead_letter_queue"}
[2022-03-16T12:59:48,285][WARN ][logstash.config.source.multilocal] Ignoring the
'pipelines.yml' file because modules or command line options are specified
[2022-03-16T12:59:48,347][INFO ][logstash.agent ] No persistent UUID file found.
Generating new UUID {:uuid=>"84410117-2fa7-499b-b55a-43a29192540e", :path=>"D:/logstash-
8.1.0/data/uuid"}
[2022-03-16T12:59:55,063][ERROR][logstash.config.sourceloader] No configuration found in the
configured sources.
[2022-03-16T12:59:55,424][INFO ][logstash.agent ] Successfully started Logstash API
endpoint {:port=>9600, :ssl_enabled=>false}
[2022-03-16T13:00:00,591][INFO ][logstash.runner ] Logstash shut down.
[2022-03-16T13:00:00,609][FATAL][org.logstash.Logstash ] Logstash stopped processing
because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby.jar:?]
at D_3a_.logstash_minus_8_dot_1_dot_0.lib.bootstrap.environment.<main>(D:\logstash-
8.1.0\lib\bootstrap\environment.rb:94) ~[?:?]
Someone let me know what I am doing wrong.
I believe the error is due to slash. in the config code
path => "D:\unicorn.csv"
also while executing
logstash -f "D:\logstash.conf"
I would also recommend to use this below command to check if there are any syntax errors in your code
logstash --config.test_and_exit -f "D:\logstash.conf"
Keep posted!!!Thanks!!!
In the output field you can add rubydebug to get to know the inputs whichever has been parsed logstash as an input . This will print the parsed data in command prompt
output{stdout{
codec => rubydebug
}
}
Error says that logstash is unable to locate configuration at location provided.
To solve the error follow below steps:
Modify file path in input as follows
input{
file{
path => "D:\\unicorn.csv"
start_position => beginning
}
}
Also run configuration with logstash as follows
D:\logstash-8.1.0\bin>logstash -f "D:\\logstash.conf"

logstash not runs config

I'm using filebeat on client side > logstash on serverside > elasticsearch on server side
filebeat on clientside works properly by sending file, but the configuration i've made on logstash returning
Fail
[WARN ] 2019-12-18 14:53:30.987 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[FATAL] 2019-12-18 14:53:31.341 [LogStash::Runner] runner - Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.
[ERROR] 2019-12-18 14:53:31.364 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
Here is my configfile
input {
beats {
port =>5044
}
}
filter {
grok {
match => { "message" =>"%{TIMESTAMP_ISO8601:timestamp}] %{WORD:test}\[%{NUMBER:nom}]\[%{DATA:tes}\] %{DATA:module_name}\: %{WORD:method}%{GREEDYDATA:log_message}" }
}
}
output {
elasticsearch
{
hosts => "127.0.0.1:9200"
index=>"test_log_pbx"
}
}
code to run my logstash config
/usr/share/logstash/bin/logstash -f logstash.conf
when i run configtest it returns
Thread.exclusive is deprecated, use Thread::Mutex
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2019-12-18 14:59:53.300 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-12-18 14:59:56.566 [LogStash::Runner] Reflections - Reflections took 139 ms to scan 1 urls, producing 20 keys and 40 values
Configuration OK
[INFO ] 2019-12-18 14:59:57.923 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
please help me i dont know whats wrong
A logstash instance already running, so you can not run another instance.If you made your logstash as service, you should stop the service. If you want to run multiple instances, you should modify pipelines.yml
If you want to learn more about pipelines.yml, I put link the below.
https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html

W7 Logstash JRUBY Error

I am new to the entire ELK Stack, and I am trying to set up Logstash. I followed all of the instructions (unzipping, setting up config file, starting Logstash). My setup is Windows 7, and my java version is 1.8.0_51.
When I run the following command (pipeline.conf is my config file):
C:\Elastic\logstash-6.2.2\bin>logstash -f pipeline.conf
I am getting the following error:
[ERROR] 2018-03-15 12:30:05.101 [main] Logstash -
java.lang.IllegalStateException:
org.jruby.exceptions.RaiseException:
(LoadError) Could not load FFI Provider:
(NotImplementedError) FFI not available:
com.kenai.jffi.Foreign.getVersion()I
See http://jira.codehaus.org/browse/JRUBY-4583
Here is what my config file:
input {
stdin {
}
}
output {
stdout {
codec => rubydebug
}
}
Any help would be appreciated. http://jira.codehaus.org/browse/JRUBY-4583 doesn't seem like a valid site. I have tried my exact process on a different machine, and Logstash works. I have been trying to look for a solution for about 2 days now. HELP PLS
Issue Resolved on the Elastic Discussion site:
https://discuss.elastic.co/t/windows-7-logstash-jruby-error/124152

Error reading field 'topic_metadata': Error reading array of size 873589, only 41 bytes available

I have installed logstash version 5.2.2 by downloading zip file in a VM having fresh Ubuntu installed in it.
I have created a sample config file logstash-sample.conf with the following entry
input{
stdin{ }
}
output{
stdout{ }
}
And executing the command $bin/logstash -f logstash-simple.conf
it is running absolutely fine.
Now in the same Ubuntu machine, I installed kafka by following the exact same process mentioned
here https://www.digitalocean.com/community/tutorials/how-to-install-apache-kafka-on-ubuntu-14-04 and followed till step no 7.
Then I modified the logstash-sample.conf file to contain the following
input {
kafka{
bootstrap_servers => "localhost:9092"
topics => ["TutorialTopic"]
}
}
output {
stdout { codec => rubydebug }
}
And this time I am getting the following error,
sample#sample-VirtualBox:~/Downloads/logstash-5.2.2$ bin/logstash -f logstash-sample.conf
Sending Logstash's logs to /home/rs-switch/Downloads/logstash-5.2.2/logs which is now configured via log4j2.properties
[2017-03-07T00:26:25,629][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>125}
[2017-03-07T00:26:25,650][INFO ][logstash.pipeline ] Pipeline main started
[2017-03-07T00:26:26,039][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
log4j:WARN No appenders could be found for logger (org.apache.kafka.clients.consumer.ConsumerConfig).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "Ruby-0-Thread-14: /home/rs-switch/Downloads/logstash-5.2.2/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.6/lib/logstash/inputs/kafka.rb:229" org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'topic_metadata': Error reading array of size 873589, only 41 bytes available
at org.apache.kafka.common.protocol.types.Schema.read(org/apache/kafka/common/protocol/types/Schema.java:73)
at org.apache.kafka.clients.NetworkClient.parseResponse(org/apache/kafka/clients/NetworkClient.java:380)
at org.apache.kafka.clients.NetworkClient.handleCompletedReceives(org/apache/kafka/clients/NetworkClient.java:449)
at org.apache.kafka.clients.NetworkClient.poll(org/apache/kafka/clients/NetworkClient.java:269)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java:360)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java:224)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java:192)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java:163)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(org/apache/kafka/clients/consumer/internals/AbstractCoordinator.java:179)
at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(org/apache/kafka/clients/consumer/KafkaConsumer.java:974)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(org/apache/kafka/clients/consumer/KafkaConsumer.java:938)
at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
at RUBY.thread_runner(/home/rs-switch/Downloads/logstash-5.2.2/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.6/lib/logstash/inputs/kafka.rb:239)
at java.lang.Thread.run(java/lang/Thread.java:745)
[2017-03-07T00:26:28,742][WARN ][logstash.agent ] stopping pipeline {:id=>"main"}
Can anyone help me out resolving this issue? I am really stuck setting up ELK from last few weeks, but was not successful.
You most probably have a version conflict that is causing this issue. Check out the compatibility matrix in the Logstash Kafka input plugins documentation.
The link you mentioned for installing Kafka has you install version 0.8.2.1 which will not work with Kafka 0.10 clients. Kafka has version checking and backwards compatibility, but only if the broker is newer than the client, which is not the case here.
I'd recommend installing a current version of Kafka, there have been immense improvements since version 0.8 that you'd be missing out on if you tried downgrading Logstash instead.
Check out the Confluent Platform Quickstart for an easy way to get started.

Resources