Logstash stuck when starting up - elasticsearch

What's wrong with the following logstash configuration?
input {
file {
type => "access_log"
# Wildcards work, here :)
path => [ "/root/isaac/my_logs/access_logs/gw_access_log*"]
start_position => "beginning"
}
}
output {
stdout { debug => true }
elasticsearch { embedded => true }
}
When running the above configuration, logstash is stuck on startup as follows:
[root#myvm logstash]# java -jar logstash-1.3.3-flatjar.jar agent -f logstash-complex.conf
Using milestone 2 input plugin 'file'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.3.3/plugin-milestones {:level=>:warn}
More importantly what are the ways to debug the issue?
I already checked that the file i am putting in the path do exist.

That isn't stuck, that's running.
you get this:
Using milestone 2 input plugin 'file'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.3.3/plugin-milestones {:level=>:warn}
Once logstash has started successfully
If you add -- web onto the end of your command then you should be able to see some output in Kibana web interface

If you aren't seeing messages appear in the console, first I would check that new entries are definitely being written to the file(s) that you're trying to tail. Since you're using the stdout output you should see the messages written to the console at the same time as they're going into the embedded Elasticsearch.
What I would suggest is you simplify your config by removing the elasticsearch output - this should speed up the startup time (it can take a minute or two for the embedded elasticsearch instance to start up) and focus on getting messages onto the console output first.
If you do want more verbose debug output from Logstash you can start the program with -v, -vv or -vvv for progressively more detailed debug information. E.g.:
java -jar logstash-1.3.3-flatjar.jar agent -f logstash-complex.conf -vvv
Fair warning that -vvv does produce a LOT of debug information, so start with -v and work your way up.

Related

W7 Logstash JRUBY Error

I am new to the entire ELK Stack, and I am trying to set up Logstash. I followed all of the instructions (unzipping, setting up config file, starting Logstash). My setup is Windows 7, and my java version is 1.8.0_51.
When I run the following command (pipeline.conf is my config file):
C:\Elastic\logstash-6.2.2\bin>logstash -f pipeline.conf
I am getting the following error:
[ERROR] 2018-03-15 12:30:05.101 [main] Logstash -
java.lang.IllegalStateException:
org.jruby.exceptions.RaiseException:
(LoadError) Could not load FFI Provider:
(NotImplementedError) FFI not available:
com.kenai.jffi.Foreign.getVersion()I
See http://jira.codehaus.org/browse/JRUBY-4583
Here is what my config file:
input {
stdin {
}
}
output {
stdout {
codec => rubydebug
}
}
Any help would be appreciated. http://jira.codehaus.org/browse/JRUBY-4583 doesn't seem like a valid site. I have tried my exact process on a different machine, and Logstash works. I have been trying to look for a solution for about 2 days now. HELP PLS
Issue Resolved on the Elastic Discussion site:
https://discuss.elastic.co/t/windows-7-logstash-jruby-error/124152

i'm unable to get http-get response in OpenDayLight karaf console

as i'm new to this OpenDayLight, the following is my requirement
after installation of southbound plugin OVSDB in karaf console, i'm unable to get the http response from network-topology
installation is completely done by providing the following command in karaf console
feature:install odl-ovsdb-southbound-impl-ui
but after this installation,if i run the follow command
GET http://<controller-ip>:8181/restconf/config/network-topology:network-topology/topology/ovsdb:1/
or
GET http://<controller-ip>:8181/restconf/operational/network-topology:network-topology/topology/ovsdb:1
the result body should be:
{
"topology": [
{
"topology-id": "ovsdb:1"
}
]
}
but the result of my console is showing only
INFO
please give me any idea to get resultant output.
what ODL version are you using?
what tool are you using to issue the REST call?
I just tried with a very recent distribution (but I've known this to work
for a long time on much older versions as well):
opendaylight-user#root>feature:install odl-ovsdb-southbound-impl-ui
opendaylight-user#root>
then:
$ curl -u "admin:admin" http://127.0.0.1:8181/restconf/config/network-topology:network-topology/topology/ovsdb:1
which gave me:
{"topology":[{"topology-id":"ovsdb:1"}]}

Logstash 5 Alpha4 to elasticsearch5 Alpha4 communication error

Elasticsearch 5 is secured with xpack security and hooked with ldap which is working fine. Even user has admin right in role_mapping.
Logstash 5 configuration is as below
output {
elasticsearch {
hosts => ['localhost:9200']
user => 'gaurav#gmail.com'
password => 'pwd'
}
}
Getting below error and because of which logstash is not able to pass data to elasticsearch.
{:timestamp=>"2016-07-14T16:32:29.592000+0530",
:message=>"Encountered an unexpected error submitting a bulk request! Will retry.",
:error_message=>"undefined method code' for #",
:class=>"NoMethodError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-4.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:217:insafe_bulk'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-4.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:105:in submit'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-4.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:72:inretrying_submit'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-4.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:23:in multi_receive'", "org/jruby/RubyArray.java:1653:ineach_slice'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-4.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:22:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:136:inthreadsafe_multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_
I think I may have figured it out. I am using the Logstash 5.1.1-alpine docker image. As far as I can tell, it comes with the elasticsearch-output plugin v4.5.0, which seems to have this bug. Forcing an update of that plugin to the latest (6.2) has fixed this issue.
My Dockerfile now
FROM logstash:5.1.1-alpine
RUN $LOGSTASH_PATH/logstash-plugin install --version 6.2.0 logstash-output-elasticsearch
With the updated plugin, I no longer see this error.

Logstash Config File for IIS

I have recently installed the ELK stack on a Windows server (following this: https://community.ulyaoth.net/threads/how-to-install-logstash-on-a-windows-server-with-kibana-in-iis.17/)
I can get the IIS logs from the server into Logstash and into Elasticsearch, but I can't get the same logs from another server.
Here is my logstash config file from my second server;
input {
file {
type => "IISLog"
path => "C:/inetpub/logs/LogFiles/W3SVC*/*.log"
}
}
filter {
mutate {
add_field => [ "hostip", "%{host}" ]
}
dns {
reverse => [ "host" ]
action => replace
}
}
output {
elasticsearch {
host => "ELK01v"
port => "9301"
}
}
but there is nothing showing in Kibana
In the stderr.log for Logstash I can see the following;
Exception in thread ">output" org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [30s]
at org.elasticsearch.action.support.master.TransportMasterNodeOperationAction$3.onTimeout(org/elasticsearch/action/support/master/TransportMasterNodeOperationAction.java:180)
at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(org/elasticsearch/cluster/service/InternalClusterService.java:492)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java/util/concurrent/ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java/util/concurrent/ThreadPoolExecutor.java:615)
at java.lang.Thread.run(java/lang/Thread.java:745)
and this from the stdout.log;
{:timestamp=>"2014-08-22T15:04:55.775000+0100", :message=>"Using milestone 2 input plugin 'file'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.4.2/plugin-milestones", :level=>:warn}
{:timestamp=>"2014-08-22T15:04:55.853000+0100", :message=>"Using milestone 2 filter plugin 'dns'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.4.2/plugin-milestones", :level=>:warn}
log4j, [2014-08-22T15:05:34.215] WARN: org.elasticsearch.discovery: [logstash-WEB01v-3460-4038] waited for 30s and no initial state was set by the discovery
log4j, [2014-08-22T15:09:06.334] WARN: org.elasticsearch.transport: [logstash-WEB01v-3460-4038] Transport response handler not found of id [240]
I've confirmed that I can telnet to ELK01v on port 9301, but I can't think what else could be causing these errors. Is there anyone with ELK knowledge that could help at all?
Thanks
This is an indication that it's trying to join your cluster but wasn't able to for some reason (for example a firewall -- there is communication in both directions when it joins the cluster). The easist solution is to add protocol => http to your elasticsearch ouput. This will work since you've already verified the firewall is open in that direction.

How can I disable console messages when running maven commands?

I'm in the process of executing Maven commands to run tests in the console (MacOSX). Recently, development efforts have produced extraneous messages in the console (info, debug, warning, etc.) I'd like to know how to remove messages like this:
INFO c.c.m.s.c.p.ApplicationProperties - Loading application properties from: app-config/shiro.properties
I've used this code to remove messages from the dbunit tests:
ch.qos.logback.classic.Logger Logger = (ch.qos.logback.classic.Logger)LoggerFactory.getLogger("org.dbunit");
Logger.setLevel(Level. ERROR);
However, I'm unsure how to disable these additional (often verbose and irritating) messages from showing up on the console so that I can see the output more easily. Additional messages appear as above and these:
DEBUG c.c.m.s.c.f.dao.AbstractDBDAO - Adding filters to the Main Search query.
WARN c.c.m.s.c.p.JNDIConfigurationProperties - Unable to find JNDI value for name: xxxxx
INFO c.c.m.a.t.d.DatabaseTestFixture - * executing sql: xxxxx
The successful answer was:
SOLUTION: Solution to issue IS adding a 'logback-test.xml' file to the root of my test folder. I used the default contents (as instructed by the documentation - thanks #Charlie). Once file exists there, FIXED!

Resources