Bad configuration in Logstash - Windows 11 - elasticsearch
I try execute input jdbc oracle with tracking column, but don't work, need other information? like file.conf, etc.?
D:\Elastic\logstash-8.6.0>.\bin\logstash.bat -f D:\Elastic\logstash-8.6.0\config\conf.d\bo_firmas_electronicas.conf
Using LS_JAVA_HOME defined java: C:\Program Files\Java\jdk-19
"WARNING: Logstash comes bundled with the recommended JDK(), but is overridden by the version defined in LS_JAVA_HOME. Consider clearing LS_JAVA_HOME to use the bundled JDK."
Sending Logstash logs to D:\Elastic\logstash-8.6.0\logs which is now configured via log4j2.properties
[2023-01-13T15:00:09,087][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ArgumentError) Cannot determine timezone from nil\n(secs:1673640004.484,utc~:\"2023-01-13 20:00:04.48399996757507324\",ltz~:nil)\n(etz:nil,tnz:\"PET\",tziv:\"2.0.5\",tzidv:\"1.2022.6\",rv:\"2.6.8\",rp:\"java\",win:true,rorv:nil,astz:nil,eov:\"1.2.7\",eotnz:\"???\",eotnfz:\"???\",eotlzn:\"???\",\ndebian:nil,centos:nil,osx:nil)\nTry setting `ENV['TZ'] = 'Continent/City'` in your script (see https://en.wikipedia.org/wiki/List_of_tz_database_time_zones)", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:181)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:846)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1229)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:131)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:329)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.RubyClass.newInstance(RubyClass.java:911)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:329)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:549)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:92)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:226)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:393)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:206)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:325)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:143)", "org.jruby.RubyProc.call(RubyProc.java:309)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:107)", "java.base/java.lang.Thread.run(Thread.java:1589)"]}
[2023-01-13T15:00:09,121][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?]
at D_3a_.Elastic.logstash_minus_8_dot_6_dot_0.lib.bootstrap.environment.<main>(D:\Elastic\logstash-8.6.0\lib\bootstrap\environment.rb:91) ~[?:?]
D:\Elastic\logstash-8.6.0>
D:\Elastic\logstash-8.6.0>.\bin\logstash.bat -f D:\Elastic\logstash-8.6.0\config\conf.d\bo_firmas_electronicas.conf
Using LS_JAVA_HOME defined java: C:\Program Files\Java\jdk-19
"WARNING: Logstash comes bundled with the recommended JDK(), but is overridden by the version defined in LS_JAVA_HOME. Consider clearing LS_JAVA_HOME to use the bundled JDK."
Sending Logstash logs to D:\Elastic\logstash-8.6.0\logs which is now configured via log4j2.properties
[2023-01-13T15:02:00,714][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ArgumentError) Cannot determine timezone from nil\n(secs:1673640115.819,utc~:\"2023-01-13 20:01:55.8190000057220459\",ltz~:nil)\n(etz:nil,tnz:\"PET\",tziv:\"2.0.5\",tzidv:\"1.2022.6\",rv:\"2.6.8\",rp:\"java\",win:true,rorv:nil,astz:nil,eov:\"1.2.7\",eotnz:\"???\",eotnfz:\"???\",eotlzn:\"???\",\ndebian:nil,centos:nil,osx:nil)\nTry setting `ENV['TZ'] = 'Continent/City'` in your script (see https://en.wikipedia.org/wiki/List_of_tz_database_time_zones)", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:181)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:846)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1229)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:131)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:329)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.RubyClass.newInstance(RubyClass.java:911)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:329)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:549)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:92)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:226)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:393)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:206)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:325)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:143)", "org.jruby.RubyProc.call(RubyProc.java:309)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:107)", "java.base/java.lang.Thread.run(Thread.java:1589)"]}
[2023-01-13T15:02:00,748][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?]
at D_3a_.Elastic.logstash_minus_8_dot_6_dot_0.lib.bootstrap.environment.<main>(D:\Elastic\logstash-8.6.0\lib\bootstrap\environment.rb:91) ~[?:?]`
According to your logs, your problem is related with timestamp issue :
Unable to configure plugins: (ArgumentError) Cannot determine timezone from nil\n(secs:1673640004.484,utc~:\"2023-01-13 20:00:04.48399996757507324\",ltz~:nil)\n(etz:nil,tnz:\"PET\",tziv:\"2.0.5\",tzidv:\"1.2022.6\",rv:\"2.6.8\",rp:\"java\",win:true,rorv:nil,astz:nil,eov:\"1.2.7\",eotnz:\"???\",eotnfz:\"???\",eotlzn:\"???\",\ndebian:nil,centos:nil,osx:nil)\nTry setting `ENV['TZ'] = 'Continent/City'` in your script (see https://en.wikipedia.org/wiki/List_of_tz_database_time_zones)
And, you can easily find similar questions are here:
https://discuss.elastic.co/t/i-get-error-unable-to-configure-plugins-argumenterror-cannot-determine-timezone-from-nil/319531
Logstash 8.0 #timestamp with nanoseconds precision
https://discuss.elastic.co/t/nanoseconds-with-logstash-and-elasticsearch/282492
So, I think this answer you need to put inside the your pipeline too:
https://stackoverflow.com/a/70804574/721600
Related
Unable to read CSV file in logstash as its shutting down on its own
I have elasticsearch and Kibana are up and running and I want to read logs using logstash so for that I have passed csv file as an input in logstash.conf file but its not reading logs and shutting down automatically. This is how I am running logstash command: D:\logstash-8.1.0\bin>logstash -f "D:/logstash.conf" logstash.conf input{ file{ path => "D:/unicorn.csv" start_position => beginning } } output{ elasticsearch{ hosts => "localhost:9200" index => "indexforlogstash" } stdout{} } Below are the terminal output: "Using bundled JDK: ." OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. Sending Logstash logs to D:/logstash-8.1.0/logs which is now configured via log4j2.properties [2022-03-16T12:59:47,905][INFO ][logstash.runner ] Log4j configuration path used is: D:\logstash-8.1.0\config\log4j2.properties [2022-03-16T12:59:47,938][WARN ][logstash.runner ] The use of JAVA_HOME has been d deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead. [2022-03-16T12:59:47,942][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.1.0", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [mswin32-x86_64]"} [2022-03-16T12:59:47,947][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, - Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, - XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, - Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, - Dlog4j2.isThreadContextMapInheritable=true, --add-opens=java.base/java.security=ALL-UNNAMED, -- add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, - -add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL- UNNAMED] [2022-03-16T12:59:48,058][INFO ][logstash.settings ] Creating directory {:setting=>"path.queue", :path=>"D:/logstash-8.1.0/data/queue"} [2022-03-16T12:59:48,104][INFO ][logstash.settings ] Creating directory {:setting=>"path.dead_letter_queue", :path=>"D:/logstash-8.1.0/data/dead_letter_queue"} [2022-03-16T12:59:48,285][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2022-03-16T12:59:48,347][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"84410117-2fa7-499b-b55a-43a29192540e", :path=>"D:/logstash- 8.1.0/data/uuid"} [2022-03-16T12:59:55,063][ERROR][logstash.config.sourceloader] No configuration found in the configured sources. [2022-03-16T12:59:55,424][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false} [2022-03-16T13:00:00,591][INFO ][logstash.runner ] Logstash shut down. [2022-03-16T13:00:00,609][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit org.jruby.exceptions.SystemExit: (SystemExit) exit at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby.jar:?] at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby.jar:?] at D_3a_.logstash_minus_8_dot_1_dot_0.lib.bootstrap.environment.<main>(D:\logstash- 8.1.0\lib\bootstrap\environment.rb:94) ~[?:?] Someone let me know what I am doing wrong.
I believe the error is due to slash. in the config code path => "D:\unicorn.csv" also while executing logstash -f "D:\logstash.conf" I would also recommend to use this below command to check if there are any syntax errors in your code logstash --config.test_and_exit -f "D:\logstash.conf" Keep posted!!!Thanks!!!
In the output field you can add rubydebug to get to know the inputs whichever has been parsed logstash as an input . This will print the parsed data in command prompt output{stdout{ codec => rubydebug } }
Error says that logstash is unable to locate configuration at location provided. To solve the error follow below steps: Modify file path in input as follows input{ file{ path => "D:\\unicorn.csv" start_position => beginning } } Also run configuration with logstash as follows D:\logstash-8.1.0\bin>logstash -f "D:\\logstash.conf"
I get error with Logstash: IllegalStateException
Here is my logstash.conf: input { mongodb{ uri => 'mongodb://localhost:27017/vnExpressCrawler?authSource=vnExpressCrawler' placeholder_db_dir => '/opt/logstash-mongodb/' placeholder_db_name => 'logstash_sqlite.db' collection => 'articles' batch_size => 5000 } } filter { } output { stdout { codec => rubydebug } elasticsearch { action => "index" index => "mongo_log_data" hosts => ["localhost:9200"] } } I get error: Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main> create at org/logstash/execution/ConvergeResultExt.java:135 add at org/logstash/execution/ConvergeResultExt.java:60 when I run logstash -f logstash.conf Can someone help me solve this? My Log: Using JAVA_HOME defined java: C:\Program Files\Java\jdk1.8.0_251 WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK Sending Logstash logs to C:/Dev Programs/Logstash/logstash-7.13.2/logs which is now configured via log4j2.properties [2021-06-21T14:21:04,709][INFO ][logstash.runner ] Log4j configuration path used is: C:\Dev Programs\Logstash\logstash-7.13.2\config\log4j2.properties [2021-06-21T14:21:04,723][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.13.2", "jruby.version"=>"jruby 9.2.16.0 (2.5.7) 2021-03-03 f82228dc32 Java HotSpot(TM) 64-Bit Server VM 25.251-b08 on 1.8.0_251-b08 +indy +jit [mswin32-x86_64]"} [2021-06-21T14:21:04,867][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2021-06-21T14:21:06,405][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} [2021-06-21T14:21:07,828][INFO ][org.reflections.Reflections] Reflections took 81 ms to scan 1 urls, producing 24 keys and 48 values [2021-06-21T14:21:08,619][ERROR][logstash.plugins.registry] Unable to load plugin. {:type=>"input", :name=>"mongodb"} [2021-06-21T14:21:08,635][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any input plugin named 'mongodb'. Are you sure this is correct? Trying to load the mongodb input plugin resulted in this error: Unable to load the requested plugin named mongodb of type input. The plugin is not installed.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:119)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:86)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "C_3a_.Dev_20_Programs.Logstash.logstash_minus_7_dot_13_dot_2.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(C:/Dev Programs/Logstash/logstash-7.13.2/logstash-core/lib/logstash/java_pipeline.rb:47)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "C_3a_.Dev_20_Programs.Logstash.logstash_minus_7_dot_13_dot_2.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(C:/Dev Programs/Logstash/logstash-7.13.2/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "C_3a_.Dev_20_Programs.Logstash.logstash_minus_7_dot_13_dot_2.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(C:/Dev Programs/Logstash/logstash-7.13.2/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "C_3a_.Dev_20_Programs.Logstash.logstash_minus_7_dot_13_dot_2.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(C:/Dev Programs/Logstash/logstash-7.13.2/logstash-core/lib/logstash/agent.rb:389)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.lang.Thread.run(Thread.java:748)"]} warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true): LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>` create at org/logstash/execution/ConvergeResultExt.java:135 add at org/logstash/execution/ConvergeResultExt.java:60 converge_state at C:/Dev Programs/Logstash/logstash-7.13.2/logstash-core/lib/logstash/agent.rb:402 [2021-06-21T14:21:08,649][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`"} [2021-06-21T14:21:08,655][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:135:in `create'", "org/logstash/execution/ConvergeResultExt.java:60:in `add'", "C:/Dev Programs/Logstash/logstash-7.13.2/logstash-core/lib/logstash/agent.rb:402:in `block in converge_state'"]} [2021-06-21T14:21:08,679][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit org.jruby.exceptions.SystemExit: (SystemExit) exit at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.16.0.jar:?] at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.16.0.jar:?] at C_3a_.Dev_20_Programs.Logstash.logstash_minus_7_dot_13_dot_2.lib.bootstrap.environment.<main>(C:\Dev Programs\Logstash\logstash-7.13.2\lib\bootstrap\environment.rb:89) ~[?:?]
The problem is stated in the error logs Couldn't find any input plugin named 'mongodb'. Are you sure this is correct? Trying to load the mongodb input plugin resulted in this error: Unable to load the requested plugin named mongodb of type input. The plugin is not installed. So you need to make sure to first properly install the mongodb input plugin before running Logstash.
logstash not runs config
I'm using filebeat on client side > logstash on serverside > elasticsearch on server side filebeat on clientside works properly by sending file, but the configuration i've made on logstash returning Fail [WARN ] 2019-12-18 14:53:30.987 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [FATAL] 2019-12-18 14:53:31.341 [LogStash::Runner] runner - Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting. [ERROR] 2019-12-18 14:53:31.364 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit Here is my configfile input { beats { port =>5044 } } filter { grok { match => { "message" =>"%{TIMESTAMP_ISO8601:timestamp}] %{WORD:test}\[%{NUMBER:nom}]\[%{DATA:tes}\] %{DATA:module_name}\: %{WORD:method}%{GREEDYDATA:log_message}" } } } output { elasticsearch { hosts => "127.0.0.1:9200" index=>"test_log_pbx" } } code to run my logstash config /usr/share/logstash/bin/logstash -f logstash.conf when i run configtest it returns Thread.exclusive is deprecated, use Thread::Mutex WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN ] 2019-12-18 14:59:53.300 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO ] 2019-12-18 14:59:56.566 [LogStash::Runner] Reflections - Reflections took 139 ms to scan 1 urls, producing 20 keys and 40 values Configuration OK [INFO ] 2019-12-18 14:59:57.923 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash please help me i dont know whats wrong
A logstash instance already running, so you can not run another instance.If you made your logstash as service, you should stop the service. If you want to run multiple instances, you should modify pipelines.yml If you want to learn more about pipelines.yml, I put link the below. https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html
W7 Logstash JRUBY Error
I am new to the entire ELK Stack, and I am trying to set up Logstash. I followed all of the instructions (unzipping, setting up config file, starting Logstash). My setup is Windows 7, and my java version is 1.8.0_51. When I run the following command (pipeline.conf is my config file): C:\Elastic\logstash-6.2.2\bin>logstash -f pipeline.conf I am getting the following error: [ERROR] 2018-03-15 12:30:05.101 [main] Logstash - java.lang.IllegalStateException: org.jruby.exceptions.RaiseException: (LoadError) Could not load FFI Provider: (NotImplementedError) FFI not available: com.kenai.jffi.Foreign.getVersion()I See http://jira.codehaus.org/browse/JRUBY-4583 Here is what my config file: input { stdin { } } output { stdout { codec => rubydebug } } Any help would be appreciated. http://jira.codehaus.org/browse/JRUBY-4583 doesn't seem like a valid site. I have tried my exact process on a different machine, and Logstash works. I have been trying to look for a solution for about 2 days now. HELP PLS
Issue Resolved on the Elastic Discussion site: https://discuss.elastic.co/t/windows-7-logstash-jruby-error/124152
Error reading field 'topic_metadata': Error reading array of size 873589, only 41 bytes available
I have installed logstash version 5.2.2 by downloading zip file in a VM having fresh Ubuntu installed in it. I have created a sample config file logstash-sample.conf with the following entry input{ stdin{ } } output{ stdout{ } } And executing the command $bin/logstash -f logstash-simple.conf it is running absolutely fine. Now in the same Ubuntu machine, I installed kafka by following the exact same process mentioned here https://www.digitalocean.com/community/tutorials/how-to-install-apache-kafka-on-ubuntu-14-04 and followed till step no 7. Then I modified the logstash-sample.conf file to contain the following input { kafka{ bootstrap_servers => "localhost:9092" topics => ["TutorialTopic"] } } output { stdout { codec => rubydebug } } And this time I am getting the following error, sample#sample-VirtualBox:~/Downloads/logstash-5.2.2$ bin/logstash -f logstash-sample.conf Sending Logstash's logs to /home/rs-switch/Downloads/logstash-5.2.2/logs which is now configured via log4j2.properties [2017-03-07T00:26:25,629][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>125} [2017-03-07T00:26:25,650][INFO ][logstash.pipeline ] Pipeline main started [2017-03-07T00:26:26,039][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} log4j:WARN No appenders could be found for logger (org.apache.kafka.clients.consumer.ConsumerConfig). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Exception in thread "Ruby-0-Thread-14: /home/rs-switch/Downloads/logstash-5.2.2/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.6/lib/logstash/inputs/kafka.rb:229" org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'topic_metadata': Error reading array of size 873589, only 41 bytes available at org.apache.kafka.common.protocol.types.Schema.read(org/apache/kafka/common/protocol/types/Schema.java:73) at org.apache.kafka.clients.NetworkClient.parseResponse(org/apache/kafka/clients/NetworkClient.java:380) at org.apache.kafka.clients.NetworkClient.handleCompletedReceives(org/apache/kafka/clients/NetworkClient.java:449) at org.apache.kafka.clients.NetworkClient.poll(org/apache/kafka/clients/NetworkClient.java:269) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java:360) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java:224) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java:192) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java:163) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(org/apache/kafka/clients/consumer/internals/AbstractCoordinator.java:179) at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(org/apache/kafka/clients/consumer/KafkaConsumer.java:974) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(org/apache/kafka/clients/consumer/KafkaConsumer.java:938) at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498) at RUBY.thread_runner(/home/rs-switch/Downloads/logstash-5.2.2/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.6/lib/logstash/inputs/kafka.rb:239) at java.lang.Thread.run(java/lang/Thread.java:745) [2017-03-07T00:26:28,742][WARN ][logstash.agent ] stopping pipeline {:id=>"main"} Can anyone help me out resolving this issue? I am really stuck setting up ELK from last few weeks, but was not successful.
You most probably have a version conflict that is causing this issue. Check out the compatibility matrix in the Logstash Kafka input plugins documentation. The link you mentioned for installing Kafka has you install version 0.8.2.1 which will not work with Kafka 0.10 clients. Kafka has version checking and backwards compatibility, but only if the broker is newer than the client, which is not the case here. I'd recommend installing a current version of Kafka, there have been immense improvements since version 0.8 that you'd be missing out on if you tried downgrading Logstash instead. Check out the Confluent Platform Quickstart for an easy way to get started.