I have filebeats 5.x ship logs to logstash.
How do I reset the “file pointer” in filebeat
This is a similar problem to
How to force Logstash to reparse a file?
https://discuss.elastic.co/t/how-do-i-reset-the-file-pointer-in-filebeats/49440
I cleaned all elasticsearch's data, delete the /var/lib/filebeat/registry. but filebeat is only shipping the new line.
change the registry_file is invalid, the file's offset saved to new file (delete file is the same problem)
filebeat.registry_file: registry
Stop filbeat service.
Rename the register file - usually found in /var/lib/filebeat/registry
Start filbeat service.
sudo service filbeat stop
mv /var/lib/filebeat/registry /var/lib/filebeat/registry.old
sudo service filbeat start
The Filebeat agent stores all of its state in the registry file. The location of the registry file should be set inside of your configuration file using the filebeat.registry_file configuration option.
I recommend specifying an absolute path in this option so that you know exactly where the file will be located. If you use a relative path then the value is interpreted relative to the ${path.data} directory. On Linux installations, when started as a service or started using the filebeat.sh wrapper, path.data is set to /var/lib/filebeat.
After deleting this registry file, Filebeat will begin reading all files from the beginning (unless you have configured a prospector with tail_files: true.
If you continue to have problems, I recommend looking at the Filebeat log file which will contain a line stating where the registry file is located. For example:
2017/01/18 18:51:31.418587 registrar.go:85: INFO Registry file set to: /var/lib/filebeat/registry
As already mentioned here, stopping the filebeat service, deleting the registry file(s) and restarting the service is correct.
I just wanted to add for Windows users, if you haven't specified a unique location for the filebeat.registry_file, it will likely default to ${path.data}/registry which is somewhat confusingly the C:\ProgramData\filebeat directory as mentioned by the folks at Elastic.
In my case I had to show hidden files before it was displayed.
Related
I've installed Filebeats in my machine, and I was wondering in which location should the configuration file "filebeat.yml" should stay, once I've found 2 diretories for Elastic
C:\ProgramData\Elastic\Beats\filebeat
-> [I can find also filebeat yml examples here][1]
C:\Program Files\Elastic\Beats\8.1.2\filebeat
Can someone help ?
[1]: https://i.stack.imgur.com/8xqgU.png
The goal is to have a .yml file in a location that the filebeat program can access. Either one would work just fine. All you would do is point the running filebeat to the desired filebeat.yml file. For example, on Linux, if I create a new .yml file called example.yml, I would run it by doing ./filebeat -c /example.yml.
The same should be the case for Windows.
I've succesfully been able to set up Elasticsearch, Kibana etc and when I run: 'sudo systemctl status elasticsearch' it is all running fine.
However, when I execute 'sudo systemctl status logstash' this is the output:
It fails to start logstash, I've read numerous articles online saying it's something to do with path or config perhaps but I've had no luck finding a correct working solution.
I have JDK downloaded and followed the guide on the logstash documentation site so I'm unsure to as why logstash is not being allowed to run.
This is the output when I try to find out the logstash version.
The error message is
No configuration found in the configured sources
This means that you don't have any pipeline configuration in /etc/logstash/conf.d that Logstash can run, so it stops.
run logstash, logstash will read pipelines.yml to find your conf location
Logstash will find your .conf file from pipelines.yml. By default it will looking at /etc/logstash/conf.d/ as pipelines.yml show.
You have to move your configuration file to the path so logstash could find it.
or you want to run with specified file with it will ignore the pipeline.yml so logstash will directly go into your .conf
/usr/share/logstash/bin/logstash -f yourconf.conf
I will suggest you to do 1. but 2 is good for debugging your configuration file.
When I set the path.logs in the elasticsearch.yml I get the behaviour, that some logs are in the defined folder, but some stuff is also always created in the elasticsearch root folder.
So in the elasticsearch root folder in logs I find the pid file gc stderr and stdout file...
When I remove the folder it´s always created on startup.
How can I prevent ES from splitting up in two folders?
The path.logs in elasticsearch.yml change the path for the elasticsearch logs only.
Logs related to the jvm like the gc logs are set in the jvm.options file and the PID file location is set when starting up elasticsearch using the option -p.
If you installed elasticearch using a package manager like yum or apt, you will need to edit the systemd elasticsearch.service and change the PID_DIR variable.
If you are starting elasticsearch using the command line you will need to pass the PID file location using the option -p, something like -p /path/to/elasticsearch.pid
I'm attempting to set the share_jobs setting described HERE to false.
The documentation HERE says to visit http://myserver:port/dump_config to find the location of the HUE config. For me it gives /var/run/cloudera-scm-agent/process/73-hue-HUE_SERVER/.
As you may guess from it being in /var/run/ the directory is recreated every time on boot so changes to the hue.ini are not saved and do not seem to affect the config listed at http://myserver:port/dump_config.
I ran find / -name hue.ini to see if there was another hue.ini to change. It returned:
/var/run/cloudera-scm-agent/process/73-hue-HUE_SERVER/hue.ini
/var/run/cloudera-scm-agent/process/69-hue-BEESWAX_SERVER/hue.ini
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/share/hue/desktop/conf/hue.ini
/opt/cloudera/parcels/CDH-4.7.0-1.cdh4.7.0.p0.40/etc/hue/hue.ini
I've changed the config in each of these files to no avail. Where exactly is the hue.ini that I need to change?
I've also been looking around the Cloudera Manager for the jobbrowser section in the configuration but I have not been able to find it.
hue.ini file inside the server configuration directory (/var/run/cloudera-scm-agent/process/73-hue-HUE_SERVER/) should be modified that too not directly, should use CM instead. Location of the configuration directory will be changed during restart or applying new configurations.
Usually important service configurations will be listed in the cloudera manager configuration section. Configuration properties which are unimportant or new may not be listed, in that case these properties are supplied in the safety_val sections.
Go to CM Hue service configuration section search for hue_safety_valve_server.ini, then add the following
[jobbrowser]
share_jobs=false
Save and restart. Now if you go and check the /var/run/cloudera-scm-agent/process/XXX-hue-HUE_SERVER/hue.ini file you can see the change. What ever contents you are giving in safety_val will get appended to hue.ini file.
I am working on setting up a load balancing cluster on windows server 2012 and have a shared drive where I want the configuration files for Apache to exist at. This way each member of the LB can load the exact same config files. How do I change where the config file is located independently of where the ServerRoot is?
Start the Apache process with the -d parameter and give your alternative ServerRoot as an argument, though I'd imagine it would be a much better idea for you to use some mechanism to sync the files locally to each server.
Also read http://httpd.apache.org/docs/2.4/mod/core.html#mutex, as it's advised if you're running from a networked file system.
If you just want to specify the main config file, start the process with the -f parameter and the path to the config file as an argument.