I have my ELK setup like below:
HOST1: Component(which generates log) + Logstash (To send logs to redis)
HOST2: Redis + Elasticsearch + Logstash ( To parse data based on grok and send it to elasticsearch on same setup)
HOST3: Redis + Elasticsearch + Logstash ( To parse data based on grok and send it to elasticsearch on same setup)
HOST4: nginx + Kibana 4
Now when I send one error log line from logstash to redis, I get double entry in Kibana 4. Like below:
Plus I didnt get any email alert from logstash, although it is configured to send alert when severity == "Erro".
this is part of logstash conf file:
output {
elasticsearch { host => ["<ELK IP>"] port => "9200" protocol => "http" }
if [severity] =~ /Erro/
{
email {
from => "someone#somedomain.com"
subject => "Error Alert"
to => "someone#somedomain.com"
via => "smtp"
htmlbody => "<h2>Error Alert1</h2><br/><br/><div
align='center'>%{message}</div>"
options => [
"smtpIporHost", "smtp.office365.com",
"port", "587",
"domain", "smtp.office365.com",
"userName", "someone#somedomain.com",
"password", "somepasswd",
"authenticationType", "login",
"starttls", "true"
]
}
}
stdout { codec => rubydebug }
}
I am using following custom grok pattern to parse log line:
ABTIMESTAMP %{YEAR}%{MONTHNUM2}%{MONTHDAY} %{USERNAME}
ABLOGLEVEL (Note|Erro|Fatl|Warn|Urgt)
ABLOG %{ABTIMESTAMP:timestamp} %{HOST:hostname} %{WORD:servername} %{INT:pid} %{INT:lwp} %{INT:thread} %{ABLOGLEVEL:severity};%{USERNAME:event}\(%{NUMBER:msgcat}/%{NUMBER:msgnum}\)%{GREEDYDATA:greedydata}
Any help here as, how to get each email alert for every error log line?
Thanks in advance!
resolved it... Actually I was having multiple conf files in logstash/conf.d folder. I removed all unnecessary files and only kept my conf file and now its working. :). Thank you Val for your help
Related
I am new to elasticsearch. I tried to configure elastisearch, Kibana , logstash with MQTT plugin. I supposed to send logs to elasticseach through logstash MQTT plugin. I installed them on Mac locally, but when starting logstash, it throws following error.
[2021-11-12T17:26:37,976][ERROR][logstash.outputs.elasticsearch][logstash_pipeline] Unable to get license information {:url=>"http://localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '400' contacting Elasticsearch at URL 'http://localhost:9200/_license'"}
my logstash configuration file islike:
input {
mqtt {
host => "localhost"
port => 1883
topic => "test"
qos => 0
certificate_path => "/Users/john/logstash-7.10.2/logstash/m2mqtt_ca.crt"
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "%{[#metadata][beat]}-%{[#metadata][version]}-%{+YYYY.MM.dd}"
#user => "elastic"
#password => "changeme"
}
}
can anybody tell, what was the issue?
I have a so far running ELK installation that I want to use to analyse log files from differenct sources:
nginx-logs
auth-logs
and so on...
I am using filebeat to collect content from logfiles and sending it to logstash with this filebeat.yml:
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/*.log
- /var/nginx/example_com/logs/
output.logstash:
hosts: ["localhost:5044"]
In logstash I alread configured a grok-section, but only for nginx-logs. This was the only working tutorial I found. So this config receives content from filebeat, filters is (that's what grok is for?) and sends it to elasticsearch.
input {
beats {
port => 5044
}
}
filter {
grok {
patterns_dir => "/etc/logstash/patterns"
match => { "message" => "%{NGINXACCESS}" }
}
}
output {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "%{[#metadata][beat]}-%{[#metadata][version]}-%{+YYYY.MM.dd}"
document_type => "%{[#metadata][type]}"
}
}
That's the content of the one nginx-pattern file I am referencing:
NGUSERNAME [a-zA-Z\.\#\-\+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:clientip} (?:-|(%{WORD}.%{WORD})) %{USER:ident} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{QS:forwarder}
But I have trouble understanding how to manage different log-data sources. Because now Kibana only displays log content from /var/log, but there is no log data from my particular nginx folder.
What is it, that I am doing wrong here?
Since you are running filebeat, you already have a module available, that process nginx logs filebeat nginx module
This way, you will not need logstash to process the logs, and you only have to point the output directly to elasticsearch.
But, since you are processing multiple paths with different logs, and because elastic stack don't allow to have multiple output forms (logstash + elasticserach), you can set logstash to only process logs that do not come from nginx. This way, and using the module (that comes with sample dashboards) , your logs will do:
Filebeat -> Logstash (from input plugin to output plugin - without any filtering) -> Elasticsearch
If you really want to process the logs on your own, you are in a good path to finish. But right now, all your logs are being process by the grok pattern. So maybe the problem is with your pattern, that processes logs from nginx, and not from nginx in the same way. You can filter the logs in the filter plugin, with something like this:
#if you are using the module
filter {
if [fileset][module] == "nginx" {
}
}
if not, please check different available examples at logstash docs
Another thing you can try, it's add this to you filter. This way, if the grok fails,you will see the log in kibana, but, with the "_grok_parse_error_nginx_error" failure tag.
grok {
patterns_dir => "/etc/logstash/patterns"
match => { "message" => "%{NGINXACCESS}" }
tag_on_failure => [ "_grok_parse_error_nginx_error" ]
}
Official logstash elastic cloud module
Official doc for starting with
My logstash.yml looks like:
cloud.id: "Test:testkey"
cloud.auth: "elastic:password"
With 2 spaces in front and no space at end, within ""
This is all I have in logstash.yml and nothing else,
And I am getting:
[2018-08-29T12:33:52,112][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://myserverurl:12345/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'https://myserverurl:12345/'"}
And the my_config_file_name.conf looks like:
input{jdbc{...jdbc here... This works, as I see data in windows console}}
output {
stdout { codec => json_lines }
elasticsearch {
hosts => ["myserverurl:12345"]
index => "my_index"
# document_id => "%{brand}"
}
What I am doing is hitting bin/logstash on windows cmd,
It loads data from database that I have configured in input of conf file and then shows me error, I want to index my data from MySQL to elasticsearch on Cloud, I took 14 days trial and created a test index, for learning purpose as I later have to deploy it.
My Pipeline looks like:
- pipeline.id: my_id
path.config: "./config/conf_file_name.conf"
pipeline.workers: 1
If logs won't include senistive data, I can also provide them.
Basically I wan't to sync (schedule check) my MYSQL data with ElasticSearch on cloud i.e. AWS
The output shall be:
elasticsearch {
hosts => ["https://yourhost:yourport/"]
user => "elastic"
password => "password"
# protocol => https
# port => "yourport"
index => "test_index"
# document_id => "%{table_id}"
# - represent comments
as stated at: Configuring logstash with elastic cloud docs
The document provided while deploying app does not provide config for jdbc, jdbc as well need user and password even if defined in settings file i.e. logstash.yml
Also if you created your API key in the web UI you will not be able to get the values needed to configure Logstash. You must to use the devtool console found at /app/dev_tools#/console with something like this:
POST /_security/api_key
{
"name": "logstash"
}
of which the output is something like:
{
"id": "<id value>",
"name": "logstash",
"api_key": "<api key>",
"encoded": "<encoded api key>"
}
And in your logstash pipeline config you use the values like this:
output {
elasticsearch {
cloud_id => "<cloud id>"
api_key => "<id value>:<api key>"
data_stream => true
ssl => true
}
stdout { codec => rubydebug }
}
Note the combined "api_key" value separated by ":". Also, you can find the "cloud id" under your "Deployments" menu option.
I add the same issue in my dev environment. After scour hours on google, I understood by default, when you install Logstash, X-Pack is installed. In the doc https://www.elastic.co/guide/en/logstash/current/setup-xpack.html it is stated that
Blockquote
X-Pack is an Elastic Stack extension that provides security, alerting, monitoring, machine learning, pipeline management, and many other capabilities
Blockquote
As I don't need x-pack to run in my dev while I am streaming Elasticsearch, I had to disable it by setting ilm_enabled to false in the output of my indexation file configuration.
output {
elasticsearch {
hosts => [.. ]
ilm_enabled => false
}
}
The link bellow may help
https://discuss.opendistrocommunity.dev/t/logstash-oss-with-non-removable-x-pack/655/3
i have 3 node setup
10.x.x.1 - application and filebeat
10.x.x.2 - machine for parsing and logstash
10.x.x.3 - having centralized logstash node from where we need to push messages into Elastic Search
in 10.x.x.2 when i set the output codec to stdout , i can see the messages coming from 10.x.x.1.
Now, i need to forward all the json messages from 10.x.x.2 to 10.x.x.3 . I tried using TCP. But the messages are not gettting sent.
10.x.x.2 logstash conf file
input {
beats {
port => 5045
}
}
output{
#stdout { codec => rubydebug }
tcp{
host => "10.x.x.3"
port => 3389
}
10.x.x.3 logstash conf file
input{
tcp{
host => "10.x.x.3"
port => 3389
#mode => "server"
#codec => "json"
}
}
output{
stdout{ codec => rubydebug }
}
is there any plugin which can send json data from one logstash to another logstash server
Your config should work.
But you have to be carreful with the "codec" properties.
Try first to set it to "line" on the output AND the input plugins of the two logstash.
And see if log are incoming.
With the codec set to "line" you will have logicly no problem to forward the logs.
Then work on the "json" properties.
Do not forget that you can activate the debug mode of logstash with the argument --debug and you can log with the arguments : -l logFileName
When you start to work with the codec json look for "_jsonparsefailure" tags, which could explain why it do not transfert logs between the two logstash.
I'm Unable to load index to elasticsearch using logstash. The follwing are my logstash.conf settings. To me config settings seems fine. Please help if I'm missing something.
Assume that Logstash & elastic search services are running fine.
input {
file {
type => "IISLog"
path => "C:/inetpub/logs/LogFiles/W3SVC1/u_ex140930.log"
start_postition => "beginning"
}
}
output {
stdout { debug => true debug_format => "ruby"}
elasticsearch_http {
host => "localhost"
port => 9200
protocol => "http"
index => "iislogs2"
}
}
You can start with checking the following:
Check the logstash log file for errors.
Run the following command:telnet localhost 9200 and verify you are able to connect.
Check elasticsearch log files for errors.