ELK: start logstash fails, illegalstateexception - elasticsearch

Since my server could not access internet. I downloaded filebeat, logstash, elasticsearch and kibana's rpm format files.
I use command rpm -Uvh xxx.rpm to install filebeat logstash elasticsearch and kibana. Then I modified configuration. For logstash, I updated pipelines.yml and copy it to /etc/logstash/conf.d directory.
The piplelines.yml is as follows:
input {
beats {
port => "5044"
}
}
filter {
grok {
match => {
"message" => [
"%{TIMESTAMP_ISO8601:timestamp} \[%{DATA:threadname}\] - %{LOGLEVEL:loglevel} - app-info-exception-info - params:%{SPACE}\{\"%{DATA:jsondata}\"\} %{DATA:excentionname}: %{DATA:exceptiondetail}\n(?m)%{GREEDYDATA:extralines}",
"(?<timestamp>[\d\-\s\:]+)\s\[(?<threadname>[\d\.\w\s\(\)\-]+)\]\s-\s(?<loglevel>[\w]+)\s+-\s(?<appinfo>app-info)\s-\s(?<systemmsg>[\w\d\:\{\}\,\-\(\)\s\"]+)",
"(?<timestamp>[\d\-\s\:]+)\s\[(?<threadname>[\d\.\w\s\(\)\-]+)\]\s-\s(?<loglevel>[\w]+)\s+-\s+(?<systemmsg>[\s\.\w\-\'\:\d\[\]\/]+)"
]
}
}
}
output {
#stdout { codec => rubydebug }
elasticsearch {
hosts => [ "localhost:9200" ]
index => "ykt"
}
}
From /var/log/, I could see filebeat, elasticsearch are fine. But in logstash's logstash-plain.log, there are error messages:
[2018-06-20T20:21:54,138][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2018-06-20T20:22:07,642][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2018-06-20T20:22:21,299][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2018-06-20T20:22:34,572][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2018-06-20T20:22:48,019][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
After updation yml/configuration, I do
systemctl daemon-reload
systemctl restart logstash
I do not why there is illegalStateException. Any help?

Related

Unable to run the logstash conf in CMD (SystemExit)

I am completely new to ELK and trying to work on very basic configuration.
However,When trying to run my logstash-Testing.conf in CMD (D:\Loggers\logstash-8.1.3\bin>.\logstash.bat -f ..\config\logstash-Testing.conf), I am facing error and logstash process is stopped . Could someone plz prove a solution and the reason for the error
[2022-04-27T20:21:19,146][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby.jar:?]
at D_3a_.Loggers.logstash_minus_8_dot_1_dot_3.lib.bootstrap.environment.<main>(D:\Loggers\logstash-8.1.3\lib\bootstrap\environment.rb:94) ~[?:?]
D:\Loggers\logstash-8.1.3\bin>
Below is my conf file for reference:
input {
file{
type=>"syslog"
path=>"C:/Users/ragav/Downloads/eStockCompany/src/logs/application.log"
}
}
output {
stdout { codec => rubydebug }
elasticsearch{
hosts =>["http://localhost:9200"]
index => "CompanyAppLog"
}
}

Logstash error: Pipeline error {:pipeline_id=>"main", :exception=>#<Grok::PatternError: pattern %{JOBNAME:project} not defined>,

your textLogstash keeps crashing and I'm not sure what the issue is
Full log:
[2022-03-30T18:21:34,633][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-03-30T18:21:37,520][INFO ][org.reflections.Reflections] Reflections took 167 ms to scan 1 urls, producing 119 keys and 417 values
[2022-03-30T18:21:39,677][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://104.154.51.160:9200"]}
[2022-03-30T18:21:40,456][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://104.154.51.160:9200/]}}
[2022-03-30T18:21:40,912][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://104.154.51.160:9200/"}
[2022-03-30T18:21:40,944][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (7.17.1) {:es_version=>7}
[2022-03-30T18:21:40,957][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[2022-03-30T18:21:41,089][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. data_stream => auto resolved to false
[2022-03-30T18:21:41,094][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. data_stream => auto resolved to false
[2022-03-30T18:21:41,209][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[2022-03-30T18:21:41,559][ERROR][logstash.javapipeline ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<Grok::PatternError: pattern %{JOBNAME:project} not defined>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:123:in block in compile'", "org/jruby/RubyKernel.java:1442:in loop'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:93:in compile'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.4.1/lib/logstash/filters/grok.rb:282:in block in register'", "org/jruby/RubyArray.java:1821:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.4.1/lib/logstash/filters/grok.rb:276:in block in register'", "org/jruby/RubyHash.java:1415:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.4.1/lib/logstash/filters/grok.rb:271:in register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:232:in block in register_plugins'", "org/jruby/RubyArray.java:1821:in each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:231:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:590:in maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:244:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:189:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:141:in block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/jenkins.conf"], :thread=>"#<Thread:0x7b168333 run>"}
[2022-03-30T18:21:41,570][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2022-03-30T18:21:41,597][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2022-03-30T18:21:41,780][INFO ][logstash.runner ] Logstash shut down.
[2022-03-30T18:21:41,801][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]
My conf file
`input {
beats {
port => 5000
type => jenkins
}
}
filter {
if [type] == "jenkins" {
grok {
patterns_dir => ["/etc/logstash/patterns"]
match => {
"message" => "%{TIMESTAMP_ISO8601:createdAt}%{SPACE}[id=%{INT:buildId}]%{SPACE}%{LOGLEVEL:level}%{SPACE}%{JAVACLASS:class}%{DATA:state}:%{SPACE}%{JOBNAME:project} #%{NUMBER:buildNumber} %{DATA:execution}: %{WORD:status}"
}
}
}
}
output {
if [type] == "jenkins" {
elasticsearch {
hosts => 'elasticsearch server ip goes in here'
index => 'jenkins-%{+YYYY.MM.dd}'
}
}
} `
I was thinking it wasn't seeing my conf file but then this line was in the logs referring to it:
"pipeline.sources"=>["/etc/logstash/conf.d/jenkins.conf"], :thread=>"#<Thread:0x7b168333 run>"}
I'm confused please help!

Logstash sync mongo data to elasticsearch

I'm newbie for using Logstash and Elasticsearch. I wanted to sync my MongoDB data into Elasticsearch using Logstash Plugin (logstash-input-mongodb).
In my mongodata.conf is
input {
uri => 'mongodb://127.0.0.1:27017/final?ssl=true'
placeholder_db_dir => '/opt/logstash-mongodb/'
placeholder_db_name => 'logstash_sqlite.db'
collection => 'twitter_stream'
batch_size => 5000
}
filter {
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
action => "index"
index => "twitter_stream"
hosts => ["localhost:9200"]
}
}
While I running bin/logstash -f /etc/logstash/conf.d/mongodata.conf --path.settings /etc/logstash/
The error was displayed like this
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2020-02-28T08:48:20,246][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-02-28T08:48:20,331][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.6.0"}
[2020-02-28T08:48:20,883][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \t\r\n], "#", "{" at line 2, column 13 (byte 21) after input {\n uri ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:47:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:55:in compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:17:in block in compile_sources'", "org/jruby/RubyArray.java:2580:in map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:14:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:161:in initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:27:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:326:in block in converge_state'"]}
[2020-02-28T08:48:21,114][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2020-02-28T08:48:25,969][INFO ][logstash.runner ] Logstash shut down.
Please help me, I don't have any idea about this.
Your configuration is wrong, you need to specify what type of input you are using.
Try to change your input to this one:
input {
mongodb {
uri => 'mongodb://127.0.0.1:27017/final?ssl=true'
placeholder_db_dir => '/opt/logstash-mongodb/'
placeholder_db_name => 'logstash_sqlite.db'
collection => 'twitter_stream'
batch_size => 5000
}
}

Error pushing logs to elasticsearch using kafka as input for logstash

Below is my logstash config file logstash.conf. Lotstash reads logs from Kafka topic 'my_topic' and outputs it to elasticsearch index 'es-index'.
input {
kafka {
bootstrap_servers =>"kafka1.xxx:9092,kafka2.xxx:9092,kafka3.xxx:9092"
topics => ["my_topic"]
codec => "json"
group_id => "logstashgroup"
}
}
output {
elasticsearch {
hosts => ["es1.myhost:9200","es2.myhost:9200","es3.myhost:9200"]
user => "user123"
password => "password"
index => "es-index"
}
}
filter {
json {
source => "message"
skip_on_invalid_json => true
}
}
It worked fine for a few months but recently it started throwing the following errors:
[ERROR][logstash.outputs.elasticsearch] An unknown error occurred sending a bulk request to Elasticsearch. We will retry indefinitely {:error_message=>"Could not read from stream: Corrupt GZIP trailer", :error_class=>"Manticore::StreamClosedException"
and,
[FATAL][logstash.runner ] An unexpected error occurred! {:error=>org.apache.kafka.common.KafkaException: Received exception when fetching the next record from my_topic-1. If needed, please seek past the record to continue consumption., :backtrace=>["org.apache.kafka.clients.consumer.internals.Fetcher$PartitionRecords.fetchRecords(org/apache/kafka/clients/consumer/internals/Fetcher.java:1469)", "org.apache.kafka.clients.consumer.internals.Fetcher$PartitionRecords.access$1600(org/apache/kafka/clients/consumer/internals/Fetcher.java:1328)",
then,
[ERROR][logstash.javapipeline ] A plugin had an unrecoverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::Kafka codec=><LogStash::Codecs::JSON id=>"json_9a7a9d96-d7be-4292-a3de-67f797d22ab5"
then,
Error: Received exception when fetching the next record from my_topic-1. If needed, please seek past the record to continue consumption.
Exception: Java::OrgApacheKafkaCommon::KafkaException
and then logstash stops
[ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
I'm not sure what's the problem. I couldn't find anything relevant to solve the problem. Any help will be greatly appreciated.

LogStash setup LoadError

I'm trying to set up LogStash and I'm following this tutorial exactly. But when I run command
bin/logstash -e 'input { stdin { } } output { stdout {} }'
it gives me the following error:
warning: --1.9 ignored
LoadError: no such file to load -- bundler
require at org/jruby/RubyKernel.java:940
require at C:/jruby-9.0.0.0/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54
setup! at C:/Users/ryan.dai/Desktop/logstash-1.5.3/lib/bootstrap/bundler.rb:43
<top> at c:/Users/ryan.dai/Desktop/logstash-1.5.3/lib/bootstrap/environment.rb:46
I tried jruby -S gem install bundler as suggested from someone else but it doesn't work. Totally new to Ruby, what is happening and what should I do?
You can fallow the below URL for installing entire ELK Setup.
Here you need to pass the file(log) as a path to the input of the logstash configuration.
input {
file {
path => "/tmp/access_log"
start_position => "beginning"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
output {
elasticsearch { host => localhost }
stdout { codec => rubydebug }
}
ELK Setup Installtion
Commands for running with CMD Prompt:
logstash -f logstash.conf for running logstash
logstash --configtest -f logstash.conf for configuration test
logstash --debug -f logstash.conf for debug the logstash configuration
Logstash configuration Examples

Resources