I am using Logstash with ElasticSearch to analyze and store data out of my apache logs. In my setup logstash is taking input from a file stdin.log.
I want to create a script which automatically insert latest logs into stdin.log when ever logstash have reached at the end of stdin.log. So my question is that is there a way to find whether logstash has reached to eof or not? Can I use sincedb file for this purpose?
I have achieved my goal by comparing size of file with offset provided in sincedb file.
currentPosition = tail -1 .sincedb | awk '{printf $4}'
yields current offset of file logstash's file pointer in logfile. While
fileSize = stat -c '%s' stdin.log
yields total size in bytes. So comparing it
if[[ $currentPosition = $fileSize ]]; then #Proceed
You can look inside the sincedb file to get the inodes and current offsets.
Another option is lsof -oo10 -p $LOGSTASHPID and examine the OFFSET column for the file in question
Related
I had deleted an index. It contained logs from a certain file. Even after deletion of that index why doesn't logstash/elasticsearch read the same log file while creating a new index? And who does the role of reading the logs- ES or LS?
Logstash reads your logs and puts them into elasticsearch. There is something called a sincedb that Logstash uses to keep track of what files it has already processed. If you remove it and restart logstash it should reprocess all of your logs.
If there is a specific log you want to reparse, the easiest way to do it is to do this:
mv logfile logfile.copy
cp logfile.copy logfile
rm logfile.copy
This gives it a new inode and makes logstash think it is a new log.
I'd like to monitor syslog events every hour. I use dategrep to get the last hour but on log rotation the last hour may span to the previous syslog.
Is there an expansion to achieve listing the two recent syslog files in ascending order?
$(ls -tr syslog* | tail -n 2)
The output should be
syslog.1 syslog # when syslog.1 exists
or
syslog # when it doesn't
I've tried syslog{.1,} but it always outputs syslog.1.
Thank you!
I'm trying to find the best and most efficient way to resume reading a file from a given point.
The given file is being written frequently (this is a log file).
This file is rotated on a daily basis.
In the log file I'm looking for a pattern 'slow transaction'. End of such lines have a number into parentheses. I want to have the sum of the numbers.
Example of log line:
Jun 24 2015 10:00:00 slow transaction (5)
Jun 24 2015 10:00:06 slow transaction (1)
This is easy part that I could do with awk command to get total of 6 with above example.
Now my challenge is that I want to get the values from this file on a regular basis. I've an external system that polls a custom OID using SNMP. When hitting this OID the Linux host runs a couple of basic commands.
I want this SNMP polling event to get the number of events since the last polling only. I don't want to have the total every time, just the total of the newly added lines.
Just to mention that only bash can be used, or basic commands such as awk sed tail etc. No perl or advanced programming language.
I hope my description will be clear enough. Apologizes if this is duplicate. I did some researches before posting but did not find something that precisely correspond to my need.
Thank you for any assistance
In addition to the methods in the comment link, you can also simply use dd and stat to read the logfile size, save it and sleep 300 then check the logfile size again. If the filesize has changed, then skip over the old information with dd and read the new information only.
Note: you can add a test to handle the case where the logfile is deleted and then restarted with 0 size (e.g. if $((newsize < size)) then read all.
Here is a short example with 5 minute intervals:
#!/bin/bash
lfn=${1:-/path/to/logfile}
size=$(stat -c "%s" "$lfn") ## save original log size
while :; do
newsize=$(stat -c "%s" "$lfn") ## get new log size
if ((size != newsize)); then ## if change, use new info
## use dd to skip over existing text to new text
newtext=$(dd if="$lfn" bs="$size" skip=1 2>/dev/null)
## process newtext however you need
printf "\nnewtext:\n\n%s\n" "$newtext"
size=$((newsize)); ## update size to newsize
fi
sleep 300
done
I am new to shell scripting, and want to implement a script on my server which will automatically send e-mail alerts if:
Disk usage exceeds 90%
Disk usage exceeds 95% (In addition to the previous e-mail)
My filesystem is abc:/xyz/abc and my mount is /pqr. How can I set this up via scripts?
You can use the df command to check the file system usage. As a starting point, you can use the below command:
df -h | awk -v val=90 '$1=="/pqr"{x=int($5)>val?1:0;print x}'
The above command will print 1 if more than threshold, else print 0. The threshold is set in val.
Note: Please ensure the 5th column of your df output is the use percentage, else use appropriate column.
I have a script where I'm fetching the logs from the tomcat and sending that into the my cloud resource. Everything works well, but I have a problem when my tomcat rotates the log.
When the logs get rotated its been prefixed with date ( log gets rotated every day ). Since my script just runs every half an hour I may miss the logs when it gets rotated, because I'm fetching the logs with their static name, in the example logfile.log.
Before getting rotated the file will look like this :
logfile.log
After getting rotated, it will look like this :
logfile.log.2012-10-09
Are there any ways to get rid of this problem?
Edit:
My script :
cp /tomcat/logs/$logname $fileName
gzip $fileName
s3cmd put $fileName.gz s3://x.x.x.x.x/$folderName
Thanks in advance.
I think the best way to backup you logs is to do a check according to the mtime of the logfiles.
You can keep the log file mtime of the last backup somewhere, then check both rotated log files and current log file. If there is a rotated log file that newer then the last mtime stored, you could append the current log file to the rotated one and then backup. If only current log file is newer, then just backup it.
The mtime of the file could be retrieved by: LC_ALL=C stat logfile.log | grep '^Modify' | cut -d: -f2-, or the unix timestamp by date "+%s" --date="$(LC_ALL=C stat logfile.log | grep '^Modify' | cut -d: -f2-)"