error INFO No non-zero metrics in the last 30s message in filebeat - elasticsearch

I 'm newbie in ELK and and I'm getting issues while running logstash. I ran logstash as define in structure step by step as I do for file beat but
But when run filebeat and logstash, Its show logstash successfully runs at port 9600. In filebeat it gives like this
INFO No non-zero metrics in the last 30s
Logstash is not getting input from file beat. Please help.
My problem is as the same as this article and did what it said but noting change .
the filebeat.yml is :
filebeat.prospectors:
- input_type: log
paths:
- /usr/share/tomcat/log_app/news/*.log
output.logstash:
hosts: ["10.0.20.163:5000"]
and I ran this command sudo ./filebeat -e -c filebeat.yml -d "publish"
the logstash config file is :
input {
beats {
port => "5000"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
index => "%{[#metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[#metadata][type]}"
}
}
then ran the commands
1)bin/logstash -f first-pipeline.conf --config.test_and_exit - this gave Ok
2)bin/logstash -f first-pipeline.conf --config.reload.automatic -This started the logstash on port 9600
I couldn't proceeds after this since filebeat gives the INFO
INFO No non-zero metrics in the last 30s
and I use
elastic search : 5.5.1
kibana : 5.5.1
logstash : 5.5.1
file beat : 5.5.1

If you want to resend your data, you can try to delete filebeat's registry file, and when you restart filebeat, it will send the data again.
File location depends on your platform. See https://www.elastic.co/guide/en/beats/filebeat/5.3/migration-registry-file.html
Registry file location can also be defined in your filebeat.yml:
filebeat.registry_file: registry
https://www.elastic.co/guide/en/beats/filebeat/current/configuration-global-options.html

Everytime you stop the filebeat. It will start reading the data from the tail of file. And because the sample file which you are using are not getting frequent data. It's not able to fetch and send it to elastic search.
Edit your log file. Add few more redundant data and then try it. It should work.
This error which you have mentioned is because FIlebeat is not able to get any updated data in that file.

Related

Source of host Variable in Logstash

I'm using ELK (Kibana, ElasticSearch and Logstash are running as Docker containers) and LogstashTcpSocketAppender in Spring boot app to forward data to logstash.
Logstash config is very simple:
input {
tcp {
port => 4560
codec => json_lines
}
}
output {
elasticsearch {
hosts => [ "elasticsearch:9200" ]
}
}
The issue is that in Kibana I see host field with "gateway" value - "host: gateway".
What I don't understand is HOW is this field populated and added to logstash-* index in Kibana as:
I do not set any host variable in logback config and can clearly see it's not going out of there.
This might be set by logstash itself. But I couldn't find a concrete reference in logback documentation of how this field is being populated. And what does "gateway" really mean?
This is very confusive to me.
Could anyone please explain.
Thanks in advance.

Shipping logs to my local windows ELK/Logstash file from remote centos7 using filebeats

I have ELK all this three components configured on my local windows machine up and running.
I have a logfile on a remote centos7 machine available which I want to ship from there to my local windows with the help of Filebeat. How I can achieve that?
I have installed filebeat on centos with the help of rpm.
in configuration file I have made following changes
commented output.elasticsearch
and uncommented output.logstash (which Ip of my windows machine shall I give overe here? How to get that ip)
AND
**filebeat.inputs:
type: log
enabled: true
paths:
path to my log file**
The flow of the data is:
Agent => logstash > elasticsearch
Agent could be beat family, and you are using filebeat that is okay.
You have to configure all these stuff.
on filebeat you have to configure filebeat.yml
on logstash you have to configure logstash.conf
on elasticsearch you have to configure elasticsearch.yml
I personally will start with logstash.conf
input
{
beats
{
port =>5044
}
}
filter
{
#I assume you just want the log to run into elasticsearch
}
output
{
elasticsearch
{
hosts => "(YOUR_ELASTICSEARCH_IP):9200"
index=>"insertindexname"
}
stdout
{
codec => rubydebug
}
}
this is a very minimal working configuration. this means, Logstash will listen for input from filebeat in port 5044. Filter is needed when you want parse the data. Output is where you want to store the data. We are using elasticsearch output plugin since you want to store it to elasticsearch. stdout is super helpful for debugging your configuration file. add this you will regret nothing. This will print all the messages that sent to elasticsearch
filebeat.yml
filebeat.inputs:
- type: log
paths:
- /directory/of/your/file/file.log
output.logstash:
hosts: ["YOUR_LOGSTASH_IP:5044"]
this is a very minimal working filebeat.yml. paths is where you want logstash to harvest the file.
When you done configuring the file, start elasticsearch then logstash then filebeat.
Let me know any difficulties

filebeat configuration to send logfile to ELK which is installed in cloudfoundry

I've been working to install ELK stack in CloudFoundry and sending log files from other local server by using filebeat.
I have successfully installed ELK in CloudFoundry and able to see sample messages.
Now I am trying to send log files from local server by using filebeat. Can you suggest how to configure filebeat to send log files from local server to Logstash in CloudFoundry?
You'll need to configure the Logstash output in Filebeat for this, specifying the host & port for target logstash:
#----------------------------- Logstash output --------------------------------
output.logstash:
hosts: ["127.0.0.1:5044"]
On the logstash side, you'll need to add a beats input to the config:
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "%{[#metadata][beat]}-%{[#metadata][version]}"
}
}
Read the complete documentation here.

Unable to connect Filebeat to logstash for logging using ELK

Hi I've been working on a automated logging using elastic stack. I have filebeat that is reading logs from the path and output is set to logstash over the port 5044. The logstash config has an input listening to 5044 and output pushing to localhost:9200. The issue is I can't get it to work, I have no idea what's happening. Below are the files:
My filebeat.yml path: etc/filebeat/filebeat.yml
#=========================== Filebeat prospectors =============================
filebeat.prospectors:
# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.
- input_type: log
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /mnt/vol1/autosuggest/logs/*.log
#- c:\programdata\elasticsearch\logs\*
<other commented stuff>
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["10.10.XX.XX:5044"]
# Optional SSL. By default is off.
<other commented stuff>
My logstash.yml path: etc/logstash/logstash.yml
<other commented stuff>
path.data: /var/lib/logstash
<other commented stuff>
path.config: /etc/logstash/conf.d
<other commented stuff>
# ------------ Metrics Settings --------------
#
# Bind address for the metrics REST endpoint
#
http.host: "10.10.XX.XX"
#
# Bind port for the metrics REST endpoint, this option also accept a range
# (9600-9700) and logstash will pick up the first available ports.
#
# http.port: 9600-9700
<other commented stuff>
path.logs: /var/log/logstash
<other commented stuff>
My logpipeline30aug.config file path: /usr/share/logstash
input {
beats {
port => 5044
}
}
filter {
grok {
match => { "message" => "\A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{WORD:var0}%{SPACE}%{NOTSPACE}%{SPACE}(?<searchinfo>[^#]*)#(?<username>[^#]*)#(?<searchQuery>[^#]*)#(?<latitude>[^#]*)#(?<longitude>[^#]*)#(?<client_ip>[^#]*)#(?<responseTime>[^#]*)" }
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "logstash30aug2017"
document_type => "log"
}
}
Please Note: Elasticsearch, logstash, filebeat are all installed on the same machine with ip: 10.10.XX.XX and I've checked the firewall, it's not the issue for sure.
I checked that logstash, filebeat services are all running. Filebeat is able to push the data to elasticsearch when configured so and logstash is able to push the data to elasticsearch when configured so.
Maybe it's how I am executing the process is the issue..
I do a bin/logstash -f logpipeline30aug.config in /usr/share/logstash to start it and then I do a /etc/init.d/filebeat start from the root directory.
Please Note: Formatting may be effected due to stackoverflow formatting issue
Can someone please help? I've been trying everything since 3 days now, I've gone through the documentations as well
Your filebeat.yml looks invalid.
The output section lacks an indentation:
output.logstash:
hosts: ["10.10.XX.XX:5044"]
In general, check the correctness of the config files to ensure they're ok.
For instance, for filebeat, you can run:
filebeat -c /etc/filebeat/filebeat.yml -configtest
If you have any errors it explains what is that error so you can fix it.
You can use a similar approach for other ELK services as well

How to configure FileBeat and Logstash to add XML Files in Elasticsearch?

I'm a beginner here. My own problem is to configure FileBeat and Logstash to add XML Files in Elasticsearch on CentOS 7.
I have already install the last version of filebeat, logstash, elasticsearch and Kibana, with the plug-in "elasticsearch-head" in standalone to see inside elasticsearch. And to test my installation, i have successfully add simple log file from CentOS system (/var/log/messages), and see it inside elasticsearch-head plug-in (6 index and 26 shards):
This is a viex of my elasticsearch-head plug-in
And now, next step is to add log from XML file. After reading the documentation, i have configure filebeat and logstash. All services are running, and i try the command "touch /mes/AddOf.xml" to try to active an filebeat event, and forward log to logstash (AddOf.xml is my log file).
My XML data structure is like this for one log event :
<log4j:event logger="ServiceLogger" timestamp="1494973209812" level="INFO" thread="QueueWorker_1_38a0fec5-7c7f-46f5-a87a-9134fff1b493">
<log4j:message>Traitement du fichier \\ifs-app-01\Interfaces_MES\AddOf\ITF_MES_01_01_d2bef200-3a85-11e7-1ab5-9a50967946c3.xml</log4j:message>
<log4j:properties>
<log4j:data name="log4net:HostName" value="MES-01" />
<log4j:data name="log4jmachinename" value="MES-01" />
<log4j:data name="log4net:Identity" value="" />
<log4j:data name="log4net:UserName" value="SOFRADIR\svc_mes_sf" />
<log4j:data name="LogName" value="UpdateOperationOf" />
<log4j:data name="log4japp" value="MES_SynchroService.exe" />
</log4j:properties>
<log4j:locationInfo class="MES_SynchroService.Core.FileManager" method="TraiteFichier" file="C:\src\MES_PROD\MES_SynchroService\Core\FileManager.cs" line="47" />
</log4j:event>
My filebeat configuration like this (/etc/filebeat/filebeat.yml):
filebeat.prospectors:
# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.
- input_type: log
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /mes/*.xml
document_type: message
### Multiline options
# Mutiline can be used for log messages spanning multiple lines. This is common
# for Java Stack Traces or C-Line Continuation
# The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
multiline.pattern: ^<log4j:event
# Defines if the pattern set under pattern should be negated or not. Default is false.
multiline.negate: true
# Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
# that was (not) matched before or after or as long as a pattern is not matched based on negate.
# Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
multiline.match: after
#================================ Outputs =====================================
# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["localhost:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
My input logstash configuration (/etc/logstash/conf.d/01-beats-input.conf) :
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
My filter logstash configuration (/etc/logstash/conf.d/01-beats-filter.conf) :
filter
{
xml
{
source => "message"
xpath =>
[
"/log4j:event/log4j:message/text()", "messageMES"
]
store_xml => true
target => "doc"
}
}
My output logstash configuration (/etc/logstash/conf.d/01-beats-output.conf) :
output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "mes_log"
document_type => "%{[#metadata][type]}"
}
}
But when i try the command "touch /mes/AddOf.xml", or add manually an event log in AddOf.xml, i don't see a new index with events log from XML file in elasticsearch.
I have see documentation for XML plug-in for logstash (here), but i don't now if i need to install something ? Or maybe I'm not doing the right thing for filebeat to send the logs to logstash ?
I'm very involved and motivated to learn about ELK stack. Thank you in advance for your expertise and help. I would be grateful ! :)
In your filebeat config, the regex for multiline.pattern probably should be in single-quotes:
multiline.pattern: '^<log4j:event'

Resources