I am trying to setup ELK on Ubuntu 18.04 and I only have port 80 for now to test Elasticsearch dashboard so I modified the elasticsearch.yml as below
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: x.x.x.x
#
# Set a custom port for HTTP:
#
http.port: 80
#
# For more information, consult the network module documentation.
#
But in logstash logs it says
[2019-05-10T08:46:01,216][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://x.x.x.x:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://x.x.x.x:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
I think it is trying to find elasticsearch on 9200 ..
Any help on this will be appreciated
You need to edit the output port of logstash, you can look it up here.
input { stdin { } }
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
output {
elasticsearch { hosts => ["localhost:9200"] } <-- change this
stdout { codec => rubydebug }
}
Change http.port into elasticsearch.yml file from config directory
Related
I am trying to configure logstash and filebeat running in kubernetes to connect and push logs from kubernetes cluster to my deployment in the elastic cloud.
I have configured the logstash.yaml file with host, username and password, please find the config below:
apiVersion: v1
kind: ConfigMap
metadata:
name: logstash-config
namespace: ns-elastic
data:
logstash.conf: |-
input {
beats {
port => "9600"
}
}
filter {
fingerprint {
source => "message"
target => "[#metadata][fingerprint]"
method => "MURMUR3"
}
# Container logs are received with variable named index_prefix
# Since it is in json format, we can decode it via json filter plugin.
if [index_prefix] == "store-logs" {
if [message] =~ /^\{.*\}$/ {
json {
source => "message"
skip_on_invalid_json => true
}
}
}
if [index_prefix] == "ingress-" {
if [message] =~ /^\{.*\}$/ {
json {
source => "message"
skip_on_invalid_json => true
}
}
}
# do not expose index_prefix field to kibana
mutate {
# #metadata is not exposed outside of Logstash by default.
add_field => { "[#metadata][index_prefix]" => "%{index_prefix}-%{+YYYY.MM.dd}" }
# since we added index_prefix to metadata, we no longer need ["index_prefix"] field.
remove_field => ["index_prefix"]
}
}
output {
# You can uncomment this line to investigate the generated events by the logstash.
stdout { codec => rubydebug }
elasticsearch {
hosts => "https://******.es.*****.azure.elastic-cloud.com:9243"
user => "username"
password => "*****************"
document_id => "%{[#metadata][fingerprint]}"
# The events will be stored in elasticsearch under previously defined index_prefix value.
index => "%{[#metadata][index_prefix]}"
}
}
However, the logstash restarts with the below error:
[2022-06-19T17:32:31,943][INFO ][org.logstash.beats.Server][main][3cdfe6dec21f50e50e275d7a0c7a3d34d8ead0610c72e80ef9c735c2ef53beb9] Starting server on port: 9600
[2022-06-19T17:32:38,154][ERROR][logstash.javapipeline ][main][3cdfe6dec21f50e50e275d7a0c7a3d34d8ead0610c72e80ef9c735c2ef53beb9] A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:main
Plugin: <LogStash::Inputs::Beats port=>9600, id=>"3cdfe6dec21f50e50e275d7a0c7a3d34d8ead0610c72e80ef9c735c2ef53beb9", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_4b2c91f6-9a6f-4e5e-9a96-5b42e20cd0d9", enable_metric=>true, charset=>"UTF-8">, host=>"0.0.0.0", ssl=>false, add_hostname=>false, ssl_verify_mode=>"none", ssl_peer_metadata=>false, include_codec_tag=>true, ssl_handshake_timeout=>10000, tls_min_version=>1, tls_max_version=>1.3, cipher_suites=>["TLS_AES_128_GCM_SHA256", "TLS_AES_256_GCM_SHA384", "TLS_CHACHA20_POLY1305_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256", "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"], client_inactivity_timeout=>60, executor_threads=>1>
Error: Address already in use
Exception: Java::JavaNet::BindException
Stack: sun.nio.ch.Net.bind0(Native Method)
sun.nio.ch.Net.bind(sun/nio/ch/Net.java:459)
sun.nio.ch.Net.bind(sun/nio/ch/Net.java:448)
sun.nio.ch.ServerSocketChannelImpl.bind(sun/nio/ch/ServerSocketChannelImpl.java:227)
io.netty.channel.socket.nio.NioServerSocketChannel.doBind(io/netty/channel/socket/nio/NioServerSocketChannel.java:134)
io.netty.channel.AbstractChannel$AbstractUnsafe.bind(io/netty/channel/AbstractChannel.java:562)
io.netty.channel.DefaultChannelPipeline$HeadContext.bind(io/netty/channel/DefaultChannelPipeline.java:1334)
io.netty.channel.AbstractChannelHandlerContext.invokeBind(io/netty/channel/AbstractChannelHandlerContext.java:506)
io.netty.channel.AbstractChannelHandlerContext.bind(io/netty/channel/AbstractChannelHandlerContext.java:491)
io.netty.channel.DefaultChannelPipeline.bind(io/netty/channel/DefaultChannelPipeline.java:973)
io.netty.channel.AbstractChannel.bind(io/netty/channel/AbstractChannel.java:260)
io.netty.bootstrap.AbstractBootstrap$2.run(io/netty/bootstrap/AbstractBootstrap.java:356)
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(io/netty/util/concurrent/AbstractEventExecutor.java:164)
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(io/netty/util/concurrent/SingleThreadEventExecutor.java:472)
io.netty.channel.nio.NioEventLoop.run(io/netty/channel/nio/NioEventLoop.java:500)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(io/netty/util/concurrent/SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(io/netty/util/internal/ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(io/netty/util/concurrent/FastThreadLocalRunnable.java:30)
java.lang.Thread.run(java/lang/Thread.java:829)
Can anyone please help me understand what I am doing incorrectly here? My endgoal is to push logs from my kubernetes cluster to my deployment of elasticsearch service on Elastic Cloud. Please assist as I am unable to get enough resources on this.
The error we see in your logs says:
Error: Address already in use
Exception: Java::JavaNet::BindException
This means there is already a process that binds on port TCP/9600.
You could use netstat -plant to inspect services listening on your host. Could be another instance of logstash that was not properly shut down.
I run logstash(192.168.56.100) and elasticsearch(192.168.56.100) on same host, but elasticsearch just create index and not receiving data. And I try run logstash(192.168.56.101) on different host, and elasticsearch(192.168.56.100) can receiving data. this my elasticsearch and logstash config
example data : 12345~78904
input {
file {
path => "/tmp/tmp.log"
}
}
filter {
grok {
match => ["message" , '%{INT:shopID}~%{INT:userID}']
}
}
output {
elasticsearch {
hosts => "127.0.0.1:9200"
index => "shop"
}
}
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
http.port: 9200
elasticsearch version 5.1.1 -
logstash version 2.4.0
Do you have any errors in logstash log? Try instead 0.0.0.0 set in elasticsearch.yml _local_ or 127.0.0.1. https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html#network-interface-values
I just built an ELK server on Windows so I'm new to the process. I've read through the docs but am having trouble parsing out my IIS advanced logs, especially x-forwarded-for data as we're behind a load balancer..
My advanced logging is set up to output the data like this:
$date, $time, $s-ip, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $X-Forwarded-For, $csUser-Agent, $cs-Referer, $sc-status, $sc-substatus, $sc-win32-status, $time-taken
I set up my logstash.conf like this:
input {
tcp {
host => "localhost"
type => "iis"
port => 5044
}
}
filter {
if [type] == "iis" {
grok {
match => {"message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{IPORHOST:site} %{URIPATH:page} %{NOTSPACE:query_string} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:client_host} %{NOTSPACE:useragent} %{NOTSPACE:referer} %{GREEDYDATA:response} %{NUMBER:httpStatusCode:int} %{NUMBER:scSubstatus:int} %{NUMBER:scwin32status:int} %{NUMBER:timeTakenMS:int}"}
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "iis"
document_type => "main"
}
}
I don't think this is correct as I'm not getting data. I've scoured the docs but am still having issues and am not sure if there are other steps I need to take, like mapping the fields.
I'm currently using filebeat from one server to push data to my ELK server. I'm not sure if this is the best way as well (maybe nxlog?). We don't want to install logstash on the client machines.
Can someone lend me a hand? It would be GREATLY appreciated!!
Thanks,
George
Since you are using Filebeat then you need to use the beats input and not the tcp input. See the documentation on how to setup Logstash for Beats.
Essentially you need to replace your tcp input with:
input {
beats {
port => 5044
}
}
And inside your Filebeat configuration file, set the document_type to iis so that your filter condition will match.
filebeat:
prospectors:
- paths:
- 'C:\path\to\your\iis\logs\*.log'
document_type: iis
I'm Unable to load index to elasticsearch using logstash. The follwing are my logstash.conf settings. To me config settings seems fine. Please help if I'm missing something.
Assume that Logstash & elastic search services are running fine.
input {
file {
type => "IISLog"
path => "C:/inetpub/logs/LogFiles/W3SVC1/u_ex140930.log"
start_postition => "beginning"
}
}
output {
stdout { debug => true debug_format => "ruby"}
elasticsearch_http {
host => "localhost"
port => 9200
protocol => "http"
index => "iislogs2"
}
}
You can start with checking the following:
Check the logstash log file for errors.
Run the following command:telnet localhost 9200 and verify you are able to connect.
Check elasticsearch log files for errors.
I am trying to connect Logstash with Elasticsearch but cannot get it working.
Here is my logstash conf:
input {
stdin {
type => "stdin-type"
}
file {
type => "syslog-ng"
# Wildcards work, here :)
path => [ "/var/log/*.log", "/var/log/messages", "/var/log/syslog" ]
}
}
output {
stdout { }
elasticsearch{
type => "all"
embedded => false
host => "192.168.0.23"
port => "9300"
cluster => "logstash-cluster"
node_name => "logstash"
}
}
And I only changed these details in my elasticsearch.yml
cluster.name: logstash-cluster
node.name: "logstash"
node.master: false
network.bind_host: 192.168.0.23
network.publish_host: 192.168.0.23
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["localhost"]
With these configurations I could not make Logstash connect to ES. Can someone please suggest where I am going wrong?
First, I suggest matching your "type" attributes up.
In your input you have 2 different types, and in your output you have a type that doesn't exists in any of your inputs.
For testing, change your output to:
output {
stdout { }
elasticsearch{
type => "stdin-type"
embedded => false
host => "192.168.0.23"
port => "9300"
cluster => "logstash-cluster"
node_name => "logstash"
}
}
Then,have you created an index on your ES instance?
From the guides I've used, and my own experience (others may have another way that works) I've always used an index so that when I push something into ES, I can use the ES API and quickly check if the data has gone in or not.
Another suggestion would be to simply run your Logstash forwarder and indexer with debug flags to see what is going on behind the scenes.
Can you connect to your ES instance on 127.0.0.1? Also, try to experiment with the port and host. As a rather new user of the Logstash system, I found that my understanding at the start went against the reality of the setup. Sometimes the host IP isn't what you think it is, as well as the port. If you are willing to check your network and identify listening ports and IPs, then you can sort this out, otherwise do some intelligent trial and error.
I highly recommend this guide as a comprehensive starting point. Both points I've mentioned are (in)directly touched upon in the guide. While the guide has a slightly more complex starting point, the ideas and concepts are thorough.
I could not make Logstash connect to ES
This happened to me when my logstash and elasticsearch versions were out of sync
from the docs:
VERSION NOTE: Your Elasticsearch cluster must be running Elasticsearch
1.1.1. If you use any other version of Elasticsearch, you should set protocol => http in this plugin.
Setting protocol => http explicitly as outlined above fixed it for me.
As Adam said, the thing was the protocol setting, so only for testing I did:
logstash -e 'input { stdin { } } output { elasticsearch { host => localhost protocol => "http" port => "9200" } }'
And that seems to be working on OSX. Issue here.
Following is tested on
elasticsearch:5.4.0
and
logstash:5.4.0
(I have use docker container on OpenStack)
For Elasticsearch :
/usr/share/elasticsearch/config/elasticsearch.yml should look like as follows -
cluster.name: "docker-cluster"
network.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1
No change in any other files of /usr/share/elasticsearch/config/ is required
Run Elasticsearch using simple command -
sudo docker run --name elasticsearch -p 9200:9200 docker.elastic.co/elasticsearch/elasticsearch:5.4.0
For Logstash :
/usr/share/logstash/config/logstash.yml should look like as follows -
http.host: "0.0.0.0"
path.config: /usr/share/logstash/pipeline
# http://111.*.*.11:9200 is the IP & Port of Elasticsearch's server
xpack.monitoring.elasticsearch.url: http://111.*.*.11:9200
# "elastic" is the user name of Elasticsearch's account
xpack.monitoring.elasticsearch.username: elastic
# "changeme" is the password of Elasticsearch's "elastic" user
xpack.monitoring.elasticsearch.password: changeme
No change in any other files of /usr/share/logstash/config/ is required
/usr/share/logstash/pipeline/logstash.conf should look like as follows -
input {
file {
path => "/usr/share/logstash/test_i.log"
}
}
output {
elasticsearch {
# http://111.*.*.11:9200 is the IP & Port of Elasticsearch's server
hosts => ["http://111.*.*.11:9200"]
# "elastic" is the user name of Elasticsearch's account
user => "elastic"
# "changeme" is the password of Elasticsearch's "elastic" user
password => "changeme"
}
}
Run Logstash using simple command -
sudo docker run --name logstash --expose 25826 -p 25826:25826 docker.elastic.co/logstash/logstash:5.4.0 --debug
NOTE : Need not to do any configuration before running Docker containers. At first run the container using simple commands as mentioned above. Then go to corresponding dir, make the required changes, save it, exit container & restart the container, changes will be reflected.
I have had the same error message, and it took me a while to discover in the TRACE log of elasticsearch's discovery process that the ip address logstash was using was incorrect.
I had several ip addresses and logstash used the wrong one. After that, everything went okay.
First,you don't need to create an index in ES.
Because,you don`t need to create "index" in elasticsearch;when the logstash assign the index,the index will be created automatically.
By the way,if you did not set the index value,it will be set as default value as "logstash-%{+YYYY.MM.dd}"
(you can check this in logstash offcial guide)~
Second,you may not keep your "elastic type" the same type as your "input type";you can also write your output like this:
output {
stdout { }
elasticsearch{
embedded => false
host => "192.168.0.23"
port => "9300"
index => "a_new_index"
cluster => "logstash-cluster"
node_name => "logstash"
document_type =>"my-own-type"
}
}
With the "document_type",you can save your logs into the any type you want~
Finally,if you don`t want to assign the "document_type";it will be set the same with your "input type"
Or even you forget to assign type in "all of the configuration file";the type will be set as default value as logs~
OK,enjoy it~
I have a two node cluster of elastisearch, and only one for logstatsh.
This config works for me:
Node elk1:
#/etc/elasticsearch/elasticsearch.yml
script.disable_dynamic: true
cluster.name: elk-fibra
node.name: "elk1"
node.master: true
node.data: true
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["elk1.lab.fibra"]
root#elk1:
#/etc/logstash/conf.d/30-lumberjack-output.conf
output {
elasticsearch { host => localhost protocol => "http" port => "9200" }
stdout { codec => rubydebug }
}
Node elk2:
#/etc/elasticsearch/elasticsearch.yml
script.disable_dynamic: true
cluster.name: elk-fibra
node.name: "elk2"
node.master: false
node.data: true
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["elk1.lab.fibra"]
input => Logstash
output => ElasticSearch
input{
http {
port => 5044
response_headers => {
"Access-Control-Allow-Origin" => "*"
"Content-Type" => "text/plain"
"Access-Control-Allow-Headers" => "Origin, X-Requested-With, Content-Type, Accept"
}
}
}
output{
elasticsearch {
hosts => ["http://elasticsearch:9200"]
index => "logstash-%{+YYYY.MM.dd}"
user => elastic
password => ****
}