I'm using Logstash 1.5.2
trying to insert datas in MongoDB with logstash-output-mongodb
It works well only when my parsed logs doesn't contain arrays, when the JSON contains array it generates an exception.
Failed to send event to MongoDB
undefined methodbson_type
NoMethodError: undefined method `error_code' for #NoMethodError:0x39794292
#cancelled=false>, :exception=>#<NoMethodError: undefined methodbson_type' for "2015-08-24T12:26:19.664Z":LogStash::Timestamp>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.1-java/lib/bson/hash.rb:44:in to_bson'", "org/jruby/RubyHash.java:1341:ineach'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.1-java/lib/bson/hash.rb:43:in to_bson'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.1-java/lib/bson/encodable.rb:57:inencode_with_placeholder_and_null'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.1-java/lib/bson/hash.rb:42:in to_bson'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.1-java/lib/bson/array.rb:49:into_bson'", "org/jruby/RubyArray.java:1613:in each'", "org/jruby/RubyEnumerable.java:978:ineach_with_index'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.1-java/lib/bson/array.rb:46:in to_bson'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.1-java/lib/bson/encodable.rb:57:inencode_with_placeholder_and_null'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.1-java/lib/bson/array.rb:45:in to_bson'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.1-java/lib/bson/hash.rb:46:into_bson'", "org/jruby/RubyHash.java:1341:in each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.1-java/lib/bson/hash.rb:43:into_bson'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.1-java/lib/bson/encodable.rb:57:in encode_with_placeholder_and_null'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.1-java/lib/bson/hash.rb:42:into_bson'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/protocol/serializers.rb:155:in serialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/protocol/message.rb:153:inserialize_fields'", "org/jruby/RubyArray.java:1613:in each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/protocol/message.rb:141:inserialize_fields'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/protocol/message.rb:70:in serialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/server/connection.rb:123:inwrite'", "org/jruby/RubyArray.java:1613:in each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/server/connection.rb:122:inwrite'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/server/connectable.rb:66:in dispatch'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/loggable.rb:44:inlog'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/loggable.rb:67:in log_debug'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/server/connectable.rb:65:indispatch'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/operation/executable.rb:35:in execute'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/server/connection_pool.rb:99:inwith_connection'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/server/context.rb:63:in with_connection'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/operation/executable.rb:34:inexecute'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/operation/write/insert.rb:72:in execute_write_command'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/operation/write/insert.rb:62:inexecute'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/collection.rb:190:in insert_many'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/collection.rb:175:ininsert_one'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-mongodb-0.1.4/lib/logstash/outputs/mongodb.rb:56:in receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.2.2-java/lib/logstash/outputs/base.rb:88:inhandle'", "(eval):513:in output_func'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.2.2-java/lib/logstash/pipeline.rb:243:inoutputworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.2.2-java/lib/logstash/pipeline.rb:165:in start_outputs'"], :level=>:warn}
NoMethodError: undefined methoderror_code' for #
receive at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-mongodb-0.1.4/lib/logstash/outputs/mongodb.rb:60
handle at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.2.2-java/lib/logstash/outputs/base.rb:88
output_func at (eval):513
outputworker at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.2.2-java/lib/logstash/pipelinestrong text.rb:243
start_outputs at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.2.2-java/lib/logstash/pipeline.rb:165
`
Any idea to solve this issue please ?
Thanks !
I have this same error when logstash mongoldb output configuration has
isodate => true
I have no arrays in my data. Just Nginx access-log data. When I output my data with
stdout { codec => rubydebug }
#timestamp looks like this
{
"#timestamp" => "2015-09-16T18:49:11.347Z",
}
Main issues is that when I output this to mongodb, the data is inserted as string with quotes escaped like this:
"#timestamp": "\"2015-09-16T18:14:06.178Z\"",
I'm using logstash 1.5.4 and mongodb 3.0.6, and the error I'm getting is
:exception=>#<NoMethodError: undefined method `bson_type' for "2015-09-16T18:33:52.474Z":LogStash::Timestamp>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.4-java/lib/bson/hash.rb:44:in `to_bson'", [...]
I haven't found any solution for this yet.
Related
I'm new to ELK and trying to input 3rd party REST API to Logstash to feed data into it. I'm getting TypeError: no implicit conversion of nil into string.
I'm using http_input_poller plugin to feed data. Here is configuration for it.
input {
http_poller {
urls => {
test1 => "https://example.com/api/now/table/sys_user?sysparm_limit=1"
}
request_timeout => 60
# Supports "cron", "every", "at" and "in" schedules by rufus scheduler
schedule => { cron => "* * * * * UTC"}
codec => "json"
# A hash of request metadata info (timing, response headers, etc.) will be sent here
metadata_target => "http_poller_metadata"
}
}
output {
stdout {
codec => rubydebug
}
}
I'm using this command to run
sudo ./logstash -f logstash_http_poller.conf --path.settings /etc/logstash/ --path.data
I'm getting this error message
masteradmin#usr:/usr/share/logstash/bin$ sudo ./logstash -f logstash_http_poller.conf --path.settings /etc/logstash/ --path.data
[FATAL] 2020-03-18 07:07:07.377 [main] runner - An unexpected error occurred! {:error=>#<TypeError: no implicit conversion of nil into String>, :backtrace=>["org/jruby/RubyFileTest.java:96:in `directory?'", "org/jruby/RubyFileTest.java:88:in `directory?'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:510:in `block in value'", "org/jruby/RubyKernel.java:1906:in `tap'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:509:in `value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:97:in `get_value'", "/usr/share/logstash/logstash-core/lib/logstash/environment.rb:94:in `block in LogStash'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:167:in `block in post_process'", "org/jruby/RubyArray.java:1814:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:166:in `post_process'", "/usr/share/logstash/logstash-core/lib/logstash/util/settings_helper.rb:26:in `post_process'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:246:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:242:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in `<main>'"]}
[ERROR] 2020-03-18 07:07:07.386 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
Need help. Thanks
After some analysis of the error message I strongly believe that this error occurres due to the fact that no value for the --path.data argument was specified.
TypeError: no implicit conversion of nil into String> could indicate that nil (the non-existent path) can not be converted to a string (what the path/argument value has to be)
RubyFileTest.java:96:in ``directory?'" indicates that the ruby class expected a valid directory
So please specify a value for the argument in your run command and let me know if that solved the issue.
EDIT:
The path.data argument is not mandatory, so you can remove it. The default location would then be the data-folder inside the directory you extracted the binaries. (See https://www.elastic.co/guide/en/logstash/current/dir-layout.html for reference)
I'm writing Ruby script to communicate with Youtube Streaming API to create live stream and live broadcast. So far, I have been able to write code to insert new stream and it works but I'm facing problem while inserting broadcast. I'm getting the following error:
/usr/local/rvm/gems/ruby-2.3.0/gems/google-api-client-0.8.0/lib/google/api_client.rb:652:in `block (2 levels) in execute!': Scheduled start time is required (Google::APIClient::ClientError)
from /usr/local/rvm/gems/ruby-2.3.0/gems/retriable-1.4.1/lib/retriable/retry.rb:27:in `perform'
from /usr/local/rvm/gems/ruby-2.3.0/gems/retriable-1.4.1/lib/retriable.rb:15:in `retriable'
from /usr/local/rvm/gems/ruby-2.3.0/gems/google-api-client-0.8.0/lib/google/api_client.rb:635:in `block in execute!'
from /usr/local/rvm/gems/ruby-2.3.0/gems/retriable-1.4.1/lib/retriable/retry.rb:27:in `perform'
from /usr/local/rvm/gems/ruby-2.3.0/gems/retriable-1.4.1/lib/retriable.rb:15:in `retriable'
from /usr/local/rvm/gems/ruby-2.3.0/gems/google-api-client-0.8.0/lib/google/api_client.rb:626:in `execute!'
from ruby_script.rb:44:in `insert_broadcast'
from ruby_script.rb:98:in `<main>'
So, its complaining that I haven't mentioned the scheduled start time. Here's the relevant code:
# Create a liveBroadcast resource and set its title, scheduled start time,
# scheduled end time, and privacy status.
def insert_broadcast(client, youtube, options)
insert_broadcast_response = client.execute!(
api_method: youtube.live_broadcasts.insert,
parameters: {
part: 'snippet,status'
},
body_object: {
snippet: {
title: options[:broadcast_title],
scheduledStartTime: options[:state_time],
scheduledEndTime: options[:end_time]
},
status: {
privacyStatus: options[:privacy_status]
}
}
)
p "Broadcast: #{insert_broadcast_response.data.id}"
return insert_broadcast_response.id
end
options = {
stream_title: 'stream',
broadcast_title: 'dumdum',
start_time: '2018-01-30T00:00:00.000Z',
end_time: '2018-01-30T00:01:00.000Z',
privacy_status: 'private'
}
The format of date time I have borrowed from this code sample which is in Python. I tried with start_time: Time.now hoping Ruby date time object might work but it didn't. How do I solve this?
Edit: Here's the complete script.
Fix the typo as per #Simple-Lime's answer, and if its still doesn't work try the following:
You need to format your time correctly:
Time.now output is 2017-07-27 07:35:38 +0000 which is wrong.
You need to change it to datetime format of ISO-8601 standard with decimal notation of the timezone with Time#iso8601:
require 'time'
# example time format: '2014-01-30T00:00:00.000Z'
Time.now.getutc.iso8601(3)
# => "2017-07-27T07:30:33.742Z"
In your code you have scheduledStartTime: options[:state_time], and are passing it into the method as options[:start_time]
I would like to get help with the code in ruby filter.
logstash version is 5.0.0 alpha4.
I am testing the code in ruby filter as below but I am getting _rubyexception.
ruby {
code => "
event['newfield'] = 'test'
"
}
ruby filter is defined inside filter { }.
The logstash.log shows as below,
:timestamp=>"2016-08-03T15:26:47.291000+0900", :message=>"Ruby exception occurred: undefined method[]=' for 2016-08-03T06:26:46.829Z test %{message}:LogStash::Event", :level=>:error}
I cant find the reason why ruby filter is unable to use event object.
I appreciate if I could get some help to cope this issue.
Thanks,
Yu
There is a new event API in Logstash 5.0.0 and now you need to set fields on the event as follows:
ruby {
code => "
event.set('newfield', 'test')
"
}
For get the value from field name:
ruby {
code => "
value = (event.get('field_name'))
"
}
this is my 10th day using puppet so sorry if there is not enough information.
I installed module 'statsd' from: https://forge.puppet.com/jdowning/statsd on my puppet-master and used it like in readme file.
class { 'statsd':
backends => ['./backends/graphite'],
graphiteHost => "graphite-relay.dev.sizmdx.com",
graphite_globalPrefix => 'statsite',
graphite_legacyNamespace => false,
stackdriver_sendTimerPercentiles => false,
configfile => '/etc/statsd/localConfig.js'
}
I keep getting this error:
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Error from DataBinding 'hiera' while looking up 'statsd::librato_skipInternalMetrics': undefined method `to_sym' for []:Array on node beadmin-dev0-lior-cm-3f4gxw0f.sizmek
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
We do have 'Heira' in the puppet project (altough I'm not familiar too much with that)
here the Heira.yaml:
:backends:
- yaml
:hierarchy:
- client/%{clientcert}
- roles/%{role}
- environment/%{env}
- global
:yaml:
:datadir: /opt/puppet/hiera
I google'd the error without any findings, any suggestions ?
if need more information please let me know.
thanks in advance
task = {:project=>1000,
:order=>0,
:partial_image=>nil,
:options=>{
:height=>50,
:width=>50,
:start_row=>1,
:start_column=>1,
:end_row=>50,
:end_column=>50,
:scene=>0}}
project = redis.hget('active_projects', task[:project])
=>
{:name=>"Pov",
:tasks=>
{0=>
{:project=>1000,
:order=>0,
:partial_image=>nil,
:options=>
{:height=>50,
:width=>50,
:start_row=>1,
:start_column=>1,
:end_row=>50,
:end_column=>50,
:scene=>"blabla"
}
}
},
:id=>1000,
:image=>"",
:options=>
{:height=>100,
:width=>50,
:scene=>"blabla"
}
}
task[:partial_image] = 'blablabla'
project[:tasks][task[:order]] = task # this is line 37
Failure/Error: completed_task = DPovray::Task.perform(task)
TypeError:
can't convert Symbol into Integer
# ./lib/jobs/job.rb:37:in `[]'
# ./lib/jobs/job.rb:37:in `block in perform'
# ./lib/jobs/job.rb:35:in `perform'
# ./spec/task_spec.rb:22:in `block (4 levels) in <top (required)>'
The code is in https://github.com/Nerian/DPovray
The test that fails can be run with rspec spec/task_spec.rb
Actually project = redis.hget('active_projects', task[:project]) is returning a string, not a ruby hash. So that is why it fails.
I am playing with https://github.com/nateware/redis-objects to see if I can do what I want to do.
Also, instead of doing:
Redis.new.hset('active_projects', active_project[:id], active_project)
You can do:
Redis.new.hset('active_projects', active_project[:id], Marshal.dump(active_project))
And it just works, thanks to hash marshaling.
Nonetheless, I do not consider this a good solution. I don't like to use Marshaling as it is much difficult do debug by looking at the database.
Also I just got a:
incompatible marshal file format (can't be read)
format version 4.8 required; 123.58 given
So let us discover a different approach...
edit:
Now I am playing with JSON.dump and JSON.parse. They seem a better approach.
Edit:
I ended up encapsulating this hash into a real object. So I have Project class and a Task class. In each one I define the methods to_json and self.json_create(o) to that they can be convert to and from JSON.
It works quite well.