Logstash getting syntax errors after upgrading to 7.13.3 - ruby

So my company has me upgrading our Logstash version for our repository to 7.13.3 from 6.6.2.
After fixing some of the other errors with the upgrade, it seems the last piece is to change the ruby syntax in the config file.
However, I am not too familiar with the language and not sure why the syntax no longer works.
Here is an example of one of the syntax errors we get from the file.
[2021-07-21T16:10:22,524][ERROR][logstash.javapipeline ][main] Pipeline error {
:pipeline_id=>"main",
:exception=>#<RegexpError: unmatched range specifier in char-class: /(?<ucd_environment_name1>(?<=release_ucd_environment_name:)[\w-.]*)/m>,
:backtrace=>[
"org/jruby/RubyRegexp.java:965:in `initialize'",
"/Users/808451090/Desktop/app/logstash-7.13.3/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:127:in `compile'",
"/Users/808451090/Desktop/app/logstash-7.13.3/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.4.0/lib/logstash/filters/grok.rb:282:in `block in register'",
"org/jruby/RubyArray.java:1809:in `each'",
"/Users/808451090/Desktop/app/logstash-7.13.3/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.4.0/lib/logstash/filters/grok.rb:276:in `block in register'",
"org/jruby/RubyHash.java:1415:in `each'",
"/Users/808451090/Desktop/app/logstash-7.13.3/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.4.0/lib/logstash/filters/grok.rb:271:in `register'",
"org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in `register'",
"/Users/808451090/Desktop/app/logstash-7.13.3/logstash-core/lib/logstash/java_pipeline.rb:228:in `block in register_plugins'",
"org/jruby/RubyArray.java:1809:in `each'",
"/Users/808451090/Desktop/app/logstash-7.13.3/logstash-core/lib/logstash/java_pipeline.rb:227:in `register_plugins'",
"/Users/808451090/Desktop/app/logstash-7.13.3/logstash-core/lib/logstash/java_pipeline.rb:586:in `maybe_setup_out_plugins'",
"/Users/808451090/Desktop/app/logstash-7.13.3/logstash-core/lib/logstash/java_pipeline.rb:240:in `start_workers'",
"/Users/808451090/Desktop/app/logstash-7.13.3/logstash-core/lib/logstash/java_pipeline.rb:185:in `run'",
"/Users/808451090/Desktop/app/logstash-7.13.3/logstash-core/lib/logstash/java_pipeline.rb:137:in `block in start'"
],
"pipeline.sources"=>["/Users/808451090/Desktop/app/logstash-7.13.3/devops-jenkins/jenkins.conf"],
:thread=>"#<Thread:0x4fbac51b run>"
}
The line this error references:
grok {
match => { "message_string" => "(?<ucd_environment_name1>(?<=release_ucd_environment_name:)[\w-.]*)" }
}
There are other lines where this error occurs, but they all have similar syntax to this line so I'm sure I can apply the same change to those ones too.
If anyone can point me to how I can change this line to check for the same expression that this Logstash version accepts syntactically.

The error is
unmatched range specifier in char-class
In character definitions, you can define ranges of characters as e.g. [a-z]. When using a literal dash character there, you have to be careful to either escape it or to make sure it unambiguously defines a single character rather than a range.
In your example, you can just escape the dash in your regex to make sure the dash is regarded as a single possible character:
grok {
match => { "message_string" => "(?<ucd_environment_name1>(?<=release_ucd_environment_name:)[\w\-.]*)" }
}

Related

Logstash: TypeError: no implicit conversion of nil into string

I'm new to ELK and trying to input 3rd party REST API to Logstash to feed data into it. I'm getting TypeError: no implicit conversion of nil into string.
I'm using http_input_poller plugin to feed data. Here is configuration for it.
input {
http_poller {
urls => {
test1 => "https://example.com/api/now/table/sys_user?sysparm_limit=1"
}
request_timeout => 60
# Supports "cron", "every", "at" and "in" schedules by rufus scheduler
schedule => { cron => "* * * * * UTC"}
codec => "json"
# A hash of request metadata info (timing, response headers, etc.) will be sent here
metadata_target => "http_poller_metadata"
}
}
output {
stdout {
codec => rubydebug
}
}
I'm using this command to run
sudo ./logstash -f logstash_http_poller.conf --path.settings /etc/logstash/ --path.data
I'm getting this error message
masteradmin#usr:/usr/share/logstash/bin$ sudo ./logstash -f logstash_http_poller.conf --path.settings /etc/logstash/ --path.data
[FATAL] 2020-03-18 07:07:07.377 [main] runner - An unexpected error occurred! {:error=>#<TypeError: no implicit conversion of nil into String>, :backtrace=>["org/jruby/RubyFileTest.java:96:in `directory?'", "org/jruby/RubyFileTest.java:88:in `directory?'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:510:in `block in value'", "org/jruby/RubyKernel.java:1906:in `tap'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:509:in `value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:97:in `get_value'", "/usr/share/logstash/logstash-core/lib/logstash/environment.rb:94:in `block in LogStash'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:167:in `block in post_process'", "org/jruby/RubyArray.java:1814:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:166:in `post_process'", "/usr/share/logstash/logstash-core/lib/logstash/util/settings_helper.rb:26:in `post_process'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:246:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:242:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in `<main>'"]}
[ERROR] 2020-03-18 07:07:07.386 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
Need help. Thanks
After some analysis of the error message I strongly believe that this error occurres due to the fact that no value for the --path.data argument was specified.
TypeError: no implicit conversion of nil into String> could indicate that nil (the non-existent path) can not be converted to a string (what the path/argument value has to be)
RubyFileTest.java:96:in ``directory?'" indicates that the ruby class expected a valid directory
So please specify a value for the argument in your run command and let me know if that solved the issue.
EDIT:
The path.data argument is not mandatory, so you can remove it. The default location would then be the data-folder inside the directory you extracted the binaries. (See https://www.elastic.co/guide/en/logstash/current/dir-layout.html for reference)

Logstash: :message=>"Failed to flush outgoing items"

I have set up a 3 node ES instance (elasticsearch-1.4.2). It works properly when the logstash is run from 2 nodes. However an exception as shown below is thrown on running the logstash on 3rd node. Could you please help?
{:timestamp=>"2015-11-30T06:50:58.873000-0700",
:message=>"Failed to flush outgoing items",
:outgoing_count=>1,
:exception=>#<Errno::EBADF: Bad file descriptor - Bad file descriptor>,
:backtrace=>[
"org/jruby/RubyIO.java:2097:in `close'",
"//scratch/LOGSTASH/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/connection.rb:173:in `connect'",
"org/jruby/RubyArray.java:1613:in `each'",
"//scratch/LOGSTASH/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/connection.rb:139:in `connect'",
"//scratch/LOGSTASH/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/agent.rb:406:in `connect'",
"org/jruby/RubyProc.java:271:in `call'",
"//scratch/LOGSTASH/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/pool.rb:48:in `fetch'",
"//scratch/LOGSTASH/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/agent.rb:403:in `connect'",
"//scratch/LOGSTASH/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/agent.rb:319:in `execute'",
"//scratch/LOGSTASH/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/ftw-0.0.39/lib/ftw/agent.rb:217:in `post!'",
"//scratch/LOGSTASH/logstash-1.4.2/lib/logstash/outputs/elasticsearch/protocol.rb:106:in `bulk_ftw'",
"//scratch/LOGSTASH/logstash-1.4.2/lib/logstash/outputs/elasticsearch/protocol.rb:80:in `bulk'",
"//scratch/LOGSTASH/logstash-1.4.2/lib/logstash/outputs/elasticsearch.rb:315:in `flush'",
"//scratch/LOGSTASH/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:219:in `buffer_flush'",
"org/jruby/RubyHash.java:1339:in `each'",
"//scratch/LOGSTASH/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:216:in `buffer_flush'",
"//scratch/LOGSTASH/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:193:in `buffer_flush'",
"//scratch/LOGSTASH/logstash-1.4.2/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:159:in `buffer_receive'",
"//scratch/LOGSTASH/logstash-1.4.2/lib/logstash/outputs/elasticsearch.rb:311:in `receive'",
"//scratch/LOGSTASH/logstash-1.4.2/lib/logstash/outputs/base.rb:86:in `handle'",
"(eval):148:in `initialize'",
"org/jruby/RubyProc.java:271:in `call'",
"//scratch/LOGSTASH/logstash-1.4.2/lib/logstash/pipeline.rb:266:in `output'",
"//scratch/LOGSTASH/logstash-1.4.2/lib/logstash/pipeline.rb:225:in `outputworker'",
"//scratch/LOGSTASH/logstash-1.4.2/lib/logstash/pipeline.rb:152:in `start_outputs'"],
:level=>:warn}
I could solve this by passing the max open file parameter in the command line while starting ES as >>
bin/elasticsearch -Des.max-open-files=true -f

MongoDB Ruby driver and mongoS

I am trying to use the MongoDB Ruby driver to pull information from a cluster via mongoS. Here is what I've done.
#mongo_client = Mongo::Connection.new('mongoshost', 27320)
#db = #mongo_client.db("thedatabase")
#auth = #db.authenticate("username", "password")
if(#mongo_client)
print "Successfully connected to mongos\n"
else
print "Connection failed"
end
if(#auth == true)
print "Auth successful\n"
else
print "Auth failed"
end
collection = #db.collection("thecollection")
puts collection.find.to_a
When I run this, everything works up until the point that it tries to print the documents in the collection. Here's what my output looks like:
C:\Code\whatever>ruby getdata.rb
Successfully connected to mongos
Auth successful
C:/Ruby200/lib/ruby/gems/2.0.0/gems/bson-1.9.2/lib/bson/bson_c.rb:24:in `deseria
lize': time must be positive (ArgumentError)
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/bson-1.9.2/lib/bson/bson_c.rb:2
4:in `deserialize'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/mongo-1.9.2/lib/mongo/networkin
g.rb:223:in `read_documents'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/mongo-1.9.2/lib/mongo/networkin
g.rb:169:in `receive'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/mongo-1.9.2/lib/mongo/networkin
g.rb:133:in `receive_message'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/mongo-1.9.2/lib/mongo/cursor.rb
:497:in `block in send_initial_query'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/mongo-1.9.2/lib/mongo/util/logg
ing.rb:55:in `block in instrument'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/mongo-1.9.2/lib/mongo/util/logg
ing.rb:20:in `instrument'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/mongo-1.9.2/lib/mongo/util/logg
ing.rb:54:in `instrument'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/mongo-1.9.2/lib/mongo/cursor.rb
:493:in `send_initial_query'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/mongo-1.9.2/lib/mongo/cursor.rb
:478:in `refresh'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/mongo-1.9.2/lib/mongo/cursor.rb
:124:in `next'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/mongo-1.9.2/lib/mongo/cursor.rb
:290:in `each'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/mongo-1.9.2/lib/mongo/cursor.rb
:314:in `to_a'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/mongo-1.9.2/lib/mongo/cursor.rb
:314:in `to_a'
from getdata.rb:27:in `<main>'
The "time must be positive" error had a few search results, but nothing that helped me figure out what the issue is. One of the fields in these documents does hold a negative number for a date, but I'm not sure why that's an issue since it's reading it and not writing (For example, one of the fields looks like so: {"md" : Date(-62135596800000)})
Is this an issue with the driver, is my code bad, or do I need to figure this out in another manner? I'm pretty new to ruby so any help would be appreciated!
Your data is bad as you have already noted. The problem is that the driver is trying to expand the date value ( which is really just a number internally to mongo within a BSON timestamp field ) into a real DateTime object.
When it tries to do this on read, the number is invalid an the inflation fails. If you fix the date value the problem will be corrected.
Note: If you do this in the shell use the ISOate helper to put a correct value in. If you use code then use a DateTime object. Don't use strings as you will mess your data up further.

Why I cannot set the value of this hash key?

task = {:project=>1000,
:order=>0,
:partial_image=>nil,
:options=>{
:height=>50,
:width=>50,
:start_row=>1,
:start_column=>1,
:end_row=>50,
:end_column=>50,
:scene=>0}}
project = redis.hget('active_projects', task[:project])
=>
{:name=>"Pov",
:tasks=>
{0=>
{:project=>1000,
:order=>0,
:partial_image=>nil,
:options=>
{:height=>50,
:width=>50,
:start_row=>1,
:start_column=>1,
:end_row=>50,
:end_column=>50,
:scene=>"blabla"
}
}
},
:id=>1000,
:image=>"",
:options=>
{:height=>100,
:width=>50,
:scene=>"blabla"
}
}
task[:partial_image] = 'blablabla'
project[:tasks][task[:order]] = task # this is line 37
Failure/Error: completed_task = DPovray::Task.perform(task)
TypeError:
can't convert Symbol into Integer
# ./lib/jobs/job.rb:37:in `[]'
# ./lib/jobs/job.rb:37:in `block in perform'
# ./lib/jobs/job.rb:35:in `perform'
# ./spec/task_spec.rb:22:in `block (4 levels) in <top (required)>'
The code is in https://github.com/Nerian/DPovray
The test that fails can be run with rspec spec/task_spec.rb
Actually project = redis.hget('active_projects', task[:project]) is returning a string, not a ruby hash. So that is why it fails.
I am playing with https://github.com/nateware/redis-objects to see if I can do what I want to do.
Also, instead of doing:
Redis.new.hset('active_projects', active_project[:id], active_project)
You can do:
Redis.new.hset('active_projects', active_project[:id], Marshal.dump(active_project))
And it just works, thanks to hash marshaling.
Nonetheless, I do not consider this a good solution. I don't like to use Marshaling as it is much difficult do debug by looking at the database.
Also I just got a:
incompatible marshal file format (can't be read)
format version 4.8 required; 123.58 given
So let us discover a different approach...
edit:
Now I am playing with JSON.dump and JSON.parse. They seem a better approach.
Edit:
I ended up encapsulating this hash into a real object. So I have Project class and a Task class. In each one I define the methods to_json and self.json_create(o) to that they can be convert to and from JSON.
It works quite well.

How to gsub an unicode 0083 with ruby?

I have loaded a string from a html.file, and I have writen it to a yaml file with the plugin ya2yaml:
- title: 'What a wonderful day!'
body: ... # main contents here
and I will load the .yml file by YAML::parse_file method.
but "\n" in the string will cause load problems, so I tried to gsub all "\n" to "", but there is still problems, a char '0083'(I see it in the terminal) still breaks the line, and cause loading problem:
in `load': syntax error on line 32, col 6: ` </strong><br>ok ' (ArgumentError)
from /home/croplio/.rvm/rubies/ruby-1.9.2-preview3/lib/ruby/1.9.1/syck.rb:178:in `parse'
from /home/croplio/.rvm/rubies/ruby-1.9.2-preview3/lib/ruby/1.9.1/syck.rb:203:in `block in parse_file'
from /home/croplio/.rvm/rubies/ruby-1.9.2-preview3/lib/ruby/1.9.1/syck.rb:202:in `open'
So what's wrong with the yaml or the char 0083?
or how can I avoid this problem?
0083 is a unicode character 'NO BREAK HERE'.
I don't know YAML::pars, but maybe you can switch it to use unicodes or use pure ascii codes.

Resources