I am launching a website, and I wanted to setup a Bash one-liner so when someone hits the site it would make a beep using the internal buzzer.
So far it's working using the following.
tail -f access_log | while read x ; do echo -ne '\007' $x '\n' ; done
Tail follows the access_log and dumps to STDOUT, get STDOUT line at a time, echo the line with '\007' "internal beep hex code", and done...
This works like a beauty... Every hit shows the line from the log and beeps... However, it got annoying very quickly, so ideally I wanted to filter the tail -f /access/log before it's piped into the while so that read only gets lines I care about. I was thinking grep "/index.php" would be a good indication of visitors...
This is where the issue is...
I can do...
tail -f access_log | while read x ; do echo -ne '\007' $x '\n' ; done
beeps on everything
and i can do...
tail -f access_log | grep "/index.php"
and pages are shown with no beep, but when i do
tail -f access_log | grep "/index.php" | while read x ; do echo -ne '\007' $x '\n' ; done
Nothing happens, no line from log, no beep.
I think the grep is messing it up somewhere, but I can't figure out where.
I'd love it to be a one liner, and I know it should really be done in a script and would be easier, but it doesn't explain why the above, which I think should work, isn't.
Grep's output is buffered when it's used in a pipe. Use --line-buffered to force it to use line buffering so it outputs lines immediately.
tail -f access_log | grep --line-buffered "/index.php" | while read x ; do echo -ne '\007' $x '\n' ; done
You could also combine the grep and while loop into a single awk call:
tail -f access_log | awk '/\/index.php/ { print "\007" $0 }'
Grep buffers output when standard output is not a terminal. You need to pass the --line-buffered switch to grep to force it to flush standard output whenever it writes a line.
Using sed -u for unbuffered:
Lighter than awk and grep, using sed could be simple, quick and efficient:
tail -f access.log | sed -une "s#/index.php#&\o7#p"
sed will replace /index.php by found string & plus beep: \o7, then print lines where something was replaced. With -u, sed will read lines by lines, unbuffered.
path='/index.php'
tail -f access.log | sed -une "s#${path}#&\o7#p"
Related
Is that possible to use grep on a continuous stream?
What I mean is sort of a tail -f <file> command, but with grep on the output in order to keep only the lines that interest me.
I've tried tail -f <file> | grep pattern but it seems that grep can only be executed once tail finishes, that is to say never.
Turn on grep's line buffering mode when using BSD grep (FreeBSD, Mac OS X etc.)
tail -f file | grep --line-buffered my_pattern
It looks like a while ago --line-buffered didn't matter for GNU grep (used on pretty much any Linux) as it flushed by default (YMMV for other Unix-likes such as SmartOS, AIX or QNX). However, as of November 2020, --line-buffered is needed (at least with GNU grep 3.5 in openSUSE, but it seems generally needed based on comments below).
I use the tail -f <file> | grep <pattern> all the time.
It will wait till grep flushes, not till it finishes (I'm using Ubuntu).
I think that your problem is that grep uses some output buffering. Try
tail -f file | stdbuf -o0 grep my_pattern
it will set output buffering mode of grep to unbuffered.
If you want to find matches in the entire file (not just the tail), and you want it to sit and wait for any new matches, this works nicely:
tail -c +0 -f <file> | grep --line-buffered <pattern>
The -c +0 flag says that the output should start 0 bytes (-c) from the beginning (+) of the file.
In most cases, you can tail -f /var/log/some.log |grep foo and it will work just fine.
If you need to use multiple greps on a running log file and you find that you get no output, you may need to stick the --line-buffered switch into your middle grep(s), like so:
tail -f /var/log/some.log | grep --line-buffered foo | grep bar
you may consider this answer as enhancement .. usually I am using
tail -F <fileName> | grep --line-buffered <pattern> -A 3 -B 5
-F is better in case of file rotate (-f will not work properly if file rotated)
-A and -B is useful to get lines just before and after the pattern occurrence .. these blocks will appeared between dashed line separators
But For me I prefer doing the following
tail -F <file> | less
this is very useful if you want to search inside streamed logs. I mean go back and forward and look deeply
Didn't see anyone offer my usual go-to for this:
less +F <file>
ctrl + c
/<search term>
<enter>
shift + f
I prefer this, because you can use ctrl + c to stop and navigate through the file whenever, and then just hit shift + f to return to the live, streaming search.
sed would be a better choice (stream editor)
tail -n0 -f <file> | sed -n '/search string/p'
and then if you wanted the tail command to exit once you found a particular string:
tail --pid=$(($BASHPID+1)) -n0 -f <file> | sed -n '/search string/{p; q}'
Obviously a bashism: $BASHPID will be the process id of the tail command. The sed command is next after tail in the pipe, so the sed process id will be $BASHPID+1.
Yes, this will actually work just fine. Grep and most Unix commands operate on streams one line at a time. Each line that comes out of tail will be analyzed and passed on if it matches.
This one command workes for me (Suse):
mail-srv:/var/log # tail -f /var/log/mail.info |grep --line-buffered LOGIN >> logins_to_mail
collecting logins to mail service
Coming some late on this question, considering this kind of work as an important part of monitoring job, here is my (not so short) answer...
Following logs using bash
1. Command tail
This command is a little more porewfull than read on already published answer
Difference between follow option tail -f and tail -F, from manpage:
-f, --follow[={name|descriptor}]
output appended data as the file grows;
...
-F same as --follow=name --retry
...
--retry
keep trying to open a file if it is inaccessible
This mean: by using -F instead of -f, tail will re-open file(s) when removed (on log rotation, for sample).
This is usefull for watching logfile over many days.
Ability of following more than one file simultaneously
I've already used:
tail -F /var/www/clients/client*/web*/log/{error,access}.log /var/log/{mail,auth}.log \
/var/log/apache2/{,ssl_,other_vhosts_}access.log \
/var/log/pure-ftpd/transfer.log
For following events through hundreds of files... (consider rest of this answer to understand how to make it readable... ;)
Using switches -n (Don't use -c for line buffering!).By default tail will show 10 last lines. This can be tunned:
tail -n 0 -F file
Will follow file, but only new lines will be printed
tail -n +0 -F file
Will print whole file before following his progression.
2. Buffer issues when piping:
If you plan to filter ouptuts, consider buffering! See -u option for sed, --line-buffered for grep, or stdbuf command:
tail -F /some/files | sed -une '/Regular Expression/p'
Is (a lot more efficient than using grep) a lot more reactive than if you does'nt use -u switch in sed command.
tail -F /some/files |
sed -une '/Regular Expression/p' |
stdbuf -i0 -o0 tee /some/resultfile
3. Recent journaling system
On recent system, instead of tail -f /var/log/syslog you have to run journalctl -xf, in near same way...
journalctl -axf | sed -une '/Regular Expression/p'
But read man page, this tool was built for log analyses!
4. Integrating this in a bash script
Colored output of two files (or more)
Here is a sample of script watching for many files, coloring ouptut differently for 1st file than others:
#!/bin/bash
tail -F "$#" |
sed -une "
/^==> /{h;};
//!{
G;
s/^\\(.*\\)\\n==>.*${1//\//\\\/}.*<==/\\o33[47m\\1\\o33[0m/;
s/^\\(.*\\)\\n==> .* <==/\\o33[47;31m\\1\\o33[0m/;
p;}"
They work fine on my host, running:
sudo ./myColoredTail /var/log/{kern.,sys}log
Interactive script
You may be watching logs for reacting on events?
Here is a little script playing some sound when some USB device appear or disappear, but same script could send mail, or any other interaction, like powering on coffe machine...
#!/bin/bash
exec {tailF}< <(tail -F /var/log/kern.log)
tailPid=$!
while :;do
read -rsn 1 -t .3 keyboard
[ "${keyboard,}" = "q" ] && break
if read -ru $tailF -t 0 _ ;then
read -ru $tailF line
case $line in
*New\ USB\ device\ found* ) play /some/sound.ogg ;;
*USB\ disconnect* ) play /some/othersound.ogg ;;
esac
printf "\r%s\e[K" "$line"
fi
done
echo
exec {tailF}<&-
kill $tailPid
You could quit by pressing Q key.
you certainly won't succeed with
tail -f /var/log/foo.log |grep --line-buffered string2search
when you use "colortail" as an alias for tail, eg. in bash
alias tail='colortail -n 30'
you can check by
type alias
if this outputs something like
tail isan alias of colortail -n 30.
then you have your culprit :)
Solution:
remove the alias with
unalias tail
ensure that you're using the 'real' tail binary by this command
type tail
which should output something like:
tail is /usr/bin/tail
and then you can run your command
tail -f foo.log |grep --line-buffered something
Good luck.
Use awk(another great bash utility) instead of grep where you dont have the line buffered option! It will continuously stream your data from tail.
this is how you use grep
tail -f <file> | grep pattern
This is how you would use awk
tail -f <file> | awk '/pattern/{print $0}'
Not sure how to explain this but, what I am trying to achieve is this:
- tailing a file and grepping for a patter A
- then I want to pipe into another customGrepFunction where it matches pattern B, and if B matches echo something out. Need the customGrepFunction in order to do some other custom stuff.
The sticky part here is how to make the grepCustomFunction work here.In other words when only patternA matches echo the whole line and when both patterA & patternB match printout something custom:
when I only run:
tail -f file.log | grep patternA
I can see the pattenA rows are being printed/tailed however when I add the customGrepFunction nothing happens.
tail -f file.log | grep patternA | customGrepFunction
And the customGrepFunction should be available globally in my bin folder:
customGrepFunction(){
if grep patternB
then
echo "True"
fi
}
I have this setup however it doesn't do what I need it to do, it only echos True whenever I do Ctrl+C and exit the tailing.
What am I missing here?
Thanks
What's Going Wrong
The code: if grep patternB; then echo "true"; fi
...waits for grep patternB to exit, which will happen only when the input from tail -f file.log | grep patternA hits EOF. Since tail -f waits for new content forever, there will never be an EOF, so your if statement will never complete.
How To Fix It
Don't use grep on the inside of your function. Instead, process content line-by-line and use bash's native regex support:
customGrepFunction() {
while IFS= read -r line; do
if [[ $line =~ patternB ]]; then
echo "True"
fi
done
}
Next, make sure that grep isn't buffering content (if it were, then it would be written to your code only in big chunks, delaying until such a chunk is available). The means to do this varies by implementation, but with GNU grep, it would look like:
tail -f file.log | grep --line-buffered patternA | customGrepFunction
On a Mac, I want to determine if there are any sleep assertions present, using pmset. If there are, extract only that information and omit unnecessary information.
If grep returns nothing I want to print "Nothing".
if pmset -g | grep pre ; then pmset -g | grep pre | cut -d'(' -f2 | cut -d')' -f1 ; else printf "Nothing\n" ; fi
The problem is that the first grep result is printed, and so is the formatted one. For example this is what I get if a backup is in progress:
sleep 15 (sleep prevented by backupd)
sleep prevented by backupd
I don't want the first line, and want to discard it. I only want the second line to print ("sleep prevented by backupd").
If the grep result is empty I want to indicate that with the text "Nothing". The above script works OK for that.
There are probably many more elegant solutions but I've been searching days for one.
If i understand your question properly, you simply need to discard the output of first grep irrespective of the output it provides. If it's so, then you can use -q option provided by grep.
From the man page for 'grep':
-q, --quiet, --silent
Quiet; do not write anything to standard output. Exit immediately with zero status if any match is found, even if an error was
detected. Also see the -s or --no-messages option. (-q is specified by POSIX.)
Something like this:
if ifconfig | grep -q X; then
ifconfig | grep Mi | cut -d'(' -f2
else
printf "Nothing\n"
fi
Obviously in the above example, output of ifconfig will not change every time. Just used as an example. ;)
Redirect the output to /dev/null:
if pmset -g | grep pre >/dev/null 2>&1 ; then
pmset -g | grep pre | cut -d'(' -f2 | cut -d')' -f1
else
printf "Nothing\n"
fi
This is maybe a little more succinct. It gets grep to only output the part of the line that matches the pattern instead of the whole line, by using grep -o:
#!/bin/bash
SLEEP=$(pmset -g | grep -o "sleep prevented.*[^)]")
if [ -z "$SLEEP" ]; then
echo Nothing
else
echo $SLEEP
fi
The pattern is sleep prevented and any characters following until a ) is encountered.
I want to grab data from this kind of log.
Nov 12 13:46:14 Home cxxd[8892]: 208 11/12 13:46:14| qc=IN (1), qt=A (1), query="www.yahoo.com."
Implemented this which gives me the URL. But does not work with "TAIL -F" so that I could monitor live just the urls.
tail -100 /var/log/system.log | grep "query=" | sed -e "s/.*query=//" | sed -e "s/\"//g" | sed -e "s/.$/ /"
Please suggest or enhance
I expect your multiple sed scripts do work with tail -F output, just not as you expect.
The C standard IO libraries will perform buffering to improve performance. The IO library can do (a) no buffering (b) line-buffering (c) block-buffering. The line-buffering is normally chosen if the output is going to a terminal. But if the output is going to a file or pipe, then block buffering is normally chosen. (It's more complicated than this -- the behavior changes if the file descriptor in question is being used for stdout or stderr or another file. See setvbuf(3) for full details.)
So, while the block-buffering you're seeing now is probably better for performance, it does mean you can wait a while before ever seeing any output, as each command will eventually accumulate a block of data. At least grep(1) allows the --line-buffered command line option to use line-buffering -- and sed(1) allows the --unbuffered command line option to flush output buffers more often. So try this:
tail -f /var/log/system.log | grep --line-buffered "query=" | sed -u -e "s/.*query=//" | sed -u -e "s/\"//g" | sed -u -e "s/.$/ /"
(I didn't find any similar options for tail(1), but even if it sends blocks of data to the others, the changes to grep(1) and sed(1) will drastically help.)
Try reducing the number of pipes by replacing multiple calls to grep and sed to one with awk:
tail -f /var/log/system.log | awk -F'=' '/query=/ { sub(/^"/, "", $NF); sub(/."$/, "", $NF); print $NF }'
...which takes every line matching "query=" and grabs everything after the last '=', replaces the first '"' and the trailing '."' and prints the result.
Try the tail -f and grep argument --line-buffered
I am using 'tail -f' to follow a log file as it's updated; next I pipe the output of that to grep to show only the lines containing a search term ("org.springframework" in this case); finally I'd like to make is piping the output from grep to a third command, 'cut':
tail -f logfile | grep org.springframework | cut -c 25-
The cut command would remove the first 25 characters of each line for me if it could get the input from grep! (It works as expected if I eliminate 'grep' from the chain.)
I'm using cygwin with bash.
Actual results: When I add the second pipe to connect to the 'cut' command, the result is that it hangs, as if it's waiting for input (in case you were wondering).
Assuming GNU grep, add --line-buffered to your command line, eg.
tail -f logfile | grep --line-buffered org.springframework | cut -c 25-
Edit:
I see grep buffering isn't the only problem here, as cut doesn't allow linewise buffering.
you might want to try replacing it with something you can control, such as sed:
tail -f logfile | sed -u -n -e '/org\.springframework/ s/\(.\{0,25\}\).*$/\1/p'
or awk
tail -f logfile | awk '/org\.springframework/ {print substr($0, 0, 25);fflush("")}'
On my system, about 8K was buffered before I got any output. This sequence worked to follow the file immediately:
tail -f logfile | while read line ; do echo "$line"| grep 'org.springframework'|cut -c 25- ; done
What you have should work fine -- that's the whole idea of pipelines. The only problem I see is that, in the version of cut I have (GNU coreutiles 6.10), you should use the syntax cut -c 25- (i.e. use a minus sign instead of a plus sign) to remove the first 24 characters.
You're also searching for different patterns in your two examples, in case that's relevant.