i have 2 filebeats running on different machine(Ex: A & B) and filebeat is also configured to send logstash machine but logstash is not receiving the data even though I have added the private IP of those machines.
so i have 2 logstash .conf file to receive from 2 different filebeats.
My conf file looks like this ->
A.conf
input{
beat{
port:""
host:"private-ip-m1"
}
}
B.conf
input{
beat{
port:""
host:"private-ip-m2"
}
}
after I run logstash it is unable to connect. it says Error: Cannot assign requested address.
Can anyone tell me is there any other way to this ?
Related
I am trying to send tomcat logs to ELK. I am using Filebeat to scan the files.
My log file name would be "project_err.DD-MM-YYYY". In filebeat configuration, I am giving the file name as foldername\project_err*
But filebeat is ignoring the files. Is this configuration correct?
I tried running HDFS on k8s using this project. I got journal node and zookeeper Up and running but if I see namenode it is not going to Error state
If I look after the logs I see it as hostname could not be resolved. Then I tried seeing the /etc/hosts entry of the file on the boot up script, IP and hostname for name is not set
I'm not using any customized image just using the same helm charts as it was.
What should I need to do If I want name node to be up and running
Helm Chart Link: https://github.com/apache-spark-on-k8s/kubernetes-HDFS/tree/master/charts
I have my logstash configuration in my ubuntu server which reads data from the postgres database and send the data to elastic search. I have configured a schedule at each 15 minutes the logstash will look the postgres table, if there is any change in the table it sends the data to elastic search.
But each time the logstash is also sending the logs to syslog which I does not need. Because of logstash my syslog file consumes more memory.
So how to stop logstash to send its logs to syslog. Is there is any configuration in logstash.yml to avoid sending logs to syslog.
I referred many sites in online in which they said to remove below line from the configuration.
stdout { codec => rubydebug }
But I don't have this line.
In my output I just send my data to elastic search which I brought from AWS.
Is there is a way to stop logstash to sending its logs to syslog?
disable the rootLogger.appendRef.console in log4j
The logfiles that logstash itself produces are created through log4j, one stream goes by default to the console. Syslog will write to consolelogs to the syslog file itself. In the Ubuntu version of logstash this is configured in the file name/etc/logstash/log4j2.properties
In the default configuration there is a line that starts with
rootLogger.appenderRef.console
If you add a # in front of the line and restart logstash. The logfiles that logstash creates will stop going to syslog
service logstash restart
The other rootLogger that uses the RollingFileAppender should still write logmessages from logstash itself (so not the messages that are being processed by your pipeline) to
/var/log/logstash/logstash-plain.log
It's easy to confuse the logfiles that logstash creates with the messages that you process, especially if they get mixed by the logstash-output-stdout or logstash-output-syslog plugins. This is not applicable to you because you use the logstash-output-elasticsearch plugin that writes to elasticsearch.
The log4j.properties file gets skipped if you run logstash from the commandline, in Ubuntu. It's a nice way of testing your pipeline in a terminal, you can run multiple logstash instances in parallel (e.g. the service and a commandline test pipeline)
/usr/share/logstash/bin/logstash -f your_pipeline.conf
To avoid write to syslog, check your pipelines and log4j.properties files.
In your pipelines files, remove all occurences of this :
stdout { codec => rubydebug }
And in your log4j.properties files comment this line :
rootLogger.appenderRef.console
I am trying to aggregate all the container logs for an application to a single file for better debugging the spark application.
I configured yarn log-aggregation as suggested by https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-debugging.html. From the docs, "Log aggregation (Hadoop 2.x) compiles logs from all containers for an individual application into a single file. " -> this sounds like what I need.
I created an emr cluster with:
[
{
"Classification": "yarn-site",
"Properties": {
"yarn.log-aggregation-enable": "true",
"yarn.log-aggregation.retain-seconds": "-1",
"yarn.nodemanager.remote-app-log-dir": "s3:\/\/mybucket\/emr_log"
}
}
]
I verify the above configuration is in the yarn-site.xml on the new cluster. But submitting jobs to the cluster, I am still seeing multiple container logs when looking at the s3 log directory.
I expected to see one file under s3://mybucket/emr_log/cluster_id/containers/application_id/stderr.gz
But I still found several container directories under the application_id directories:
Ex:
s3://mybucket/emr_log/cluster_id/containers/application_id/container_XXXXXXXX_0001_01_000001/stderr.gz
s3://mybucket/emr_log/cluster_id/containers/application_id/container_XXXXXXXX_0001_01_000002/stderr.gz
s3://mybucket/emr_log/cluster_id/containers/application_id/container_XXXXXXXX_0001_01_000003/stderr.gz
Am I missing anything? Is it possible to have one single container log for one application?
I have processed a file using logstash and pushed it to elasticsearch it work. However, I had to make some changes to the logstash conf file and need to process the log file again. I deleted the index on es and restarted the logstash. I dont see the data in elasticsearch, it looks like the file is not being processed.
1. I am using logstash version 2.3.2
2. I deleted _sincedb file, restarted logstash, no log
3. I checked the conf file syntax via --configcheck and it is ok.
Any ideas what I am missing here?
I dont see any index created, no data in es. I tried these steps multiple times.
Logstash is smart enough to remember until which line it has already processed each file you've given him and stores that cursor in a sincedb file.
So, in addition to the path setting, you need to specify two more parameters in your file input that will make sure that the file is re-processed on each run:
file {
path => "/path/to/file"
start_position => "beginning"
sincedb_path => "/dev/null"
}