Running tail -f /var/log/* can sometimes show junk/garbage characters and trash the screen with control codes.
What's a good way to filter those out, in order to see a clean output with minimal loss of information?
Pipe to sed, to strip out ANSI codes (those likely to trash the console).
Pipe to strings, to only print actual strings, because some files in /var/log/ contains binary data.
tail -f /var/log/* | sed $'s#\e[\[(][[:digit:]]*[[:print:]]##g' | strings
You can add these helper aliases to your shell profile by executing this:
cat >>~/.profile <<EOF
# ANSI codes stripping helpers
alias noansi="sed $'s#\e[\[(][[:digit:]]*[[:print:]]##g'"
alias noansistrings="noansi|strings"
EOF
. ~/.profile
Then for example, you will be able to run:
tail -f /var/log/* | noansistrings
When I get output in bash I get my standard 2 colour screen. Is there any way I can, by default, highlight a line if it contains some key text output?
E.g. if it contains the word "FAIL" then the line is coloured red.
I’ve read this https://unix.stackexchange.com/questions/46562/how-do-you-colorize-only-some-keywords-for-a-bash-script
but am looking for something simpler than having to write a wrapper script which I’d inevitably have to debug at some time in the future.
For a simple workaround, pipe it through grep --color to turn some words red.
Add a fallback like ^ to print lines which do not contain any matches otherwise.
grep --color -e 'FAIL' -e '^' <<<$'Foo\nBar FAIL Baz\nIck'
Grep output with multiple Colors? describes a hack for getting multiple colors if you need that.
If you're happy to install a BASH script and ack, the hhlighter package has useful default colours and an easy interface https://github.com/paoloantinori/hhighlighter:
You can use it like so to highlight rows that start with FAIL:
h -i 'FAIL.*'
or that contain FAIL:
h -i '.*FAIL.*'
or for various common log entries:
h -i '.*FAIL.*' '.*PASS.*' '.*WARN.*'
Building on tripleee's answer, following command will highlight the matching line red and preserve the other lines:
your_command | grep --color -e ".*FAIL.*" -e "^"
If you prefer a completely inverted line, with gnu grep:
your_command | GREP_COLORS='mt=7' grep --color -e ".*FAIL.*" -e "^"
(updated with mklement0 feedback)
This will highlight not only a word, but the whole line:
echo "foo bar error baz" | egrep --color '*.FAIL.*|$'
A search phrase should be enclosed by .* on either side. It will cause highlighting before and after the search word or phrase.
References
https://unix.stackexchange.com/a/330613/341457
How .* (dot star) works?
My program is sending characters-colored text to some log file:
echo -e "\\033[38;5;2m-->\\033[0m Starting program." | tee $LogFile -a
Resulting in a perfectly colored log line, but I would like to simultaneously create another logfile without ANSI codes, due to I need to browse this log in Windows (I know there are some ANSI browsers for Windows, but I prefer to browse the log files using Total Commander, that has no decent plugin for that).
So, I need three outputs in my log line:
Some colored --> Starting program. line to screen.
The same colored --> Starting program. line to the logfile with ANSI codes.
The same not colored --> Starting program. line to the logfile (supposedly without ANSI codes, or so I think).
The above line was fine thanks to the tee command, that solves points 1 and 2, but I don't know how to add some option to save to another file but without ANSI codes, at least without replicating the full line.
Maybe using redirectors, file descriptor, with the help of mkfifo?
My workaround for now is, by the way, duplication of outputs (what becomes a bit awkward):
echo -e "\\033[38;5;2m--> Starting program.\\033[0m" | tee $LogFile -a
echo "--> Starting program." >> $NotANSILogFile
To send the ansi codes to a log file, ansi.log, and to the screen, while also sending a non-ansi version to a log file called nonansi.log, use:
echo -e "\\033[38;5;2m-->\\033[0m Starting program." | tee -a ansi.log | tee /dev/tty | sed $'s/\E[^m]*m//g' >>nonansi.log
How it works
tee -a ansi.log
The first tee command appends the ansi-encoded string the the log file ansi.log.
tee /dev/tty
The second tee command sends the ansi-encoded string to the screen. (/dev/tty is the file name of the current screen.)
sed $'s/\E[^m]*m//g' >>nonansi.log
The final command, sed removes the ansi sequences and the result is appended to nonansi.log.
The sed command is contained in a bash $'...' string so that the escape character can be represented simply as \E. This substitute command looks for sequences that start with escape, \E, and end with m and removes them. The final g tells sed to perform this substitution on every escape sequence on the line, not just the first one.
If you have GNU sed, an alternative is to use GNU's \o notation instead:
sed 's/\o033[^m]*m//g' >>nonansi.log
If you have neither GNU sed nor bash, then you need to see what facilities your sed or your shell provide for the Esc character.
Hiding the details in a shell function
If you need to make multiple log entries, it might be easiest to create a shell function, logger, to hold all the messy details:
logger() { echo -e "$*" | tee -a ansi.log | tee /dev/tty | sed $'s/\E[^m]*m//g' >>nonansi.log; }
With this function defined, then any log entry can be performed by providing the ansi-encoded string as an argument:
logger "\\033[38;5;2m-->\\033[0m Start."
This method works (needs perl installed) to send output to screen (ANSI) and two files: $LogFile (ANSI) and $LogFile.plain (ANSI-free):
echo -e "\\033[38;5;2m--> Starting program.\\033[0m" | tee >(perl -pe 's/\e\[?.*?[\#-~]//g'>>"$LogFile".plain) $LogFile -a
Details:
The tee command splits output to both screen(stdout) and to the perl command.
The perl line filters the ANSI codes.
The output of perl is redirected to the plain Non-ANSI log file (used LogFile.plain for its filename).
To send output only to files (not to screen) do (same with just a classic >/dev/null).
echo -e "\\033[38;5;2m--> Starting program.\\033[0m" | tee >(perl -pe 's/\e\[?.*?[\#-~]//g'>>"$LogFile".plain) $LogFile -a >/dev/null
Notes:
Used >> instead of >, as expected for an incremental log file.
Used -a for the tee command for the same reason above.
I would prefer to use sed instead of perl, but all the sed examples I have found until now do not work. If someone knows, please report.
I'm testing mobile Android devices and I would like to redirect the device log on a file whose name indicates both the date and time of my test, and the device model that is being tested.
For the first issue, I have already resolved with
now=$(date +"%b_%d_%Y_%k_%M");adb logcat -c;adb logcat|tee $now
So:
$ echo $now
Jan_03_2012_13_09
and the tee command creates a file with this filename.
As for the device model I have written two bash lines that obtain it from adb shell, namely
device=$(adb shell cat /system/build.prop | grep "^ro.product.device=")
deviceshortname=$(echo $device | sed 's/ro.product.device=//g')
(not optimal as I am not very good in bash programming... :) but I manage to get
$ echo $deviceshortname
LT15i
My problem is how to combine $now and $deviceshortname to obtain a filename such as:
LT15i_Jan_03_2012_13_19
I tried to set another variable:
filename=($(echo $deviceshortname"_"$now))
and got:
$ echo $filename
LT15i_Jan_03_2012_13_19
but if I try redirecting the log:
$ adb logcat | tee $filename
I obtain such file:
-rw-r--r--+ 1 ele None 293 Jan 3 13:21 ?[01;31m?[K?[m?[KLT15i_Jan_03_2012_13_19
I don't know why these strange characters and what I'm doing wrong.
Something is adding color to your output. It might be grep(1), it might adb, it might be baked into the /system/build.prop resource that you're reading.
If you're lucky, it is being added by grep(1) -- because that is supremely easy to disable with --color=no:
device=$(adb shell cat /system/build.prop | grep --color=no "^ro.product.device=")
deviceshortname=$(echo $device | sed 's/ro.product.device=//g')
If the colors are being added by adb, then perhaps it has a command line option that asks it to avoid colorizing the output.
If the colors are hard-coded into the /sys/build.prop resource in some way, then you'll need some little tool that filters out the color codes. I don't have one handy (and it's bedtime) but you can probably build one starting with tr(1) to delete \033 ASCII ESC characters.
Looks like an ANSI sequence used by adb to color the output.
I'm not sure if I'm missing something, but this works for me
p1=foo
p2=$(date +%d_%m_%Y)
cat sample_file.txt | tee $p1"_"$p2
Just type: echo ${deviceshortname}${now} and it will do the trick.
I find grep's --color=always flag to be tremendously useful. However, grep only prints lines with matches (unless you ask for context lines). Given that each line it prints has a match, the highlighting doesn't add as much capability as it could.
I'd really like to cat a file and see the entire file with the pattern matches highlighted.
Is there some way I can tell grep to print every line being read regardless of whether there's a match? I know I could write a script to run grep on every line of a file, but I was curious whether this was possible with standard grep.
Here are some ways to do it:
grep --color 'pattern\|$' file
grep --color -E 'pattern|$' file
egrep --color 'pattern|$' file
The | symbol is the OR operator. Either escape it using \ or tell grep that the search text has to be interpreted as regular expressions by adding -E or using the egrep command instead of grep.
The search text "pattern|$" is actually a trick, it will match lines that have pattern OR lines that have an end. Because all lines have an end, all lines are matched, but the end of a line isn't actually any characters, so it won't be colored.
To also pass the colored parts through pipes, e.g. towards less, provide the always parameter to --color:
grep --color=always 'pattern\|$' file | less -r
grep --color=always -E 'pattern|$' file | less -r
egrep --color=always 'pattern|$' file | less -r
Here's something along the same lines. Chances are, you'll be using less anyway, so try this:
less -p pattern file
It will highlight the pattern and jump to the first occurrence of it in the file.
You can jump to the next occurence with n and to the previous occurence with p. Quit with q.
I'd like to recommend ack -- better than grep, a power search tool for programmers.
$ ack --color --passthru --pager="${PAGER:-less -R}" pattern files
$ ack --color --passthru pattern files | less -R
$ export ACK_PAGER_COLOR="${PAGER:-less -R}"
$ ack --passthru pattern files
I love it because it defaults to recursive searching of directories (and does so much smarter than grep -r), supports full Perl regular expressions (rather than the POSIXish regex(3)), and has a much nicer context display when searching many files.
You can use my highlight script from https://github.com/kepkin/dev-shell-essentials
It's better than grep because you can highlight each match with its own color.
$ command_here | highlight green "input" | highlight red "output"
You can also create an alias. Add this function in your .bashrc (or .bash_profile on osx)
function grepe {
grep --color -E "$1|$" $2
}
You can now use the alias like this: "ifconfig | grepe inet" or "grepe css index.html".
(PS: don't forget to source ~/.bashrc to reload bashrc on current session)
Use colout program: http://nojhan.github.io/colout/
It is designed to add color highlights to a text stream. Given a regex and a color (e.g. "red"), it reproduces a text stream with matches highlighted. e.g:
# cat logfile but highlight instances of 'ERROR' in red
colout ERROR red <logfile
You can chain multiple invocations to add multiple different color highlights:
tail -f /var/log/nginx/access.log | \
colout ' 5\d\d ' red | \
colout ' 4\d\d ' yellow | \
colout ' 3\d\d ' cyan | \
colout ' 2\d\d ' green
Or you can achieve the same thing by using a regex with N groups (parenthesised parts of the regex), followed by a comma separated list of N colors.
vagrant status | \
colout \
'\''(^.+ running)|(^.+suspended)|(^.+not running)'\'' \
green,yellow,red
The -z option for grep is also pretty slick!
cat file1 | grep -z "pattern"
As grep -E '|pattern' has already been suggested, just wanted to clarify that it's possible to highlight a whole line too.
For example, tail -f somelog | grep --color -E '| \[2\].*' (specifically, the part -E '|):
I use rcg from "Linux Server Hacks", O'Reilly. It's perfect for what you want and can highlight multiple expressions each with different colours.
#!/usr/bin/perl -w
#
# regexp coloured glasses - from Linux Server Hacks from O'Reilly
#
# eg .rcg "fatal" "BOLD . YELLOW . ON_WHITE" /var/adm/messages
#
use strict;
use Term::ANSIColor qw(:constants);
my %target = ( );
while (my $arg = shift) {
my $clr = shift;
if (($arg =~ /^-/) | !$clr) {
print "Usage: rcg [regex] [color] [regex] [color] ...\n";
exit(2);
}
#
# Ugly, lazy, pathetic hack here. [Unquote]
#
$target{$arg} = eval($clr);
}
my $rst = RESET;
while(<>) {
foreach my $x (keys(%target)) {
s/($x)/$target{$x}$1$rst/g;
}
print
}
I added this to my .bash_aliases:
highlight() {
grep --color -E "$1|\$"
}
The sed way
As there is already a lot of different solution, but none show sed as solution,
and because sed is lighter and quicker than grep, I prefer to use sed for this kind of job:
sed 's/pattern/\o33[47;31;1m&\o033[0m/' file
This seems less intuitive.
\o33 is the sed syntax to generate the character octal 033 -> Escape.
(Some shells and editors also allow entering <Ctrl>-<V> followed by <Esc>, to type the character directly.)
Esc [ 47 ; 31 ; 1 m is an ANSI escape code: Background grey, foreground red and bold face.
& will re-print the pattern.
Esc [ 0 m returns the colors to default.
You could also highlight the entire line, but mark the pattern as red:
sed -E <file -e \
's/^(.*)(pattern)(.*)/\o33[30;47m\1\o33[31;1m\2\o33[0;30;47m\3\o33[0m/'
Dynamic tail -f, following logfiles
One of advantage of using sed: You could send a alarm beep on console, using bell ascii character 0x7. I often use sed like:
sudo tail -f /var/log/kern.log |
sed -ue 's/[lL]ink .*\([uU]p\|[dD]own\).*/\o33[47;31;1m&\o33[0m\o7/'
-u stand for unbuffered. This ensure that line will be treated immediately.
So I will hear some beep instantly, when I connect or disconnect my ethernet cable.
Of course, instead of link up pattern, you could watch for USB in same file, or even search for from=.*alice#bobserver.org in /var/log/mail.log (If
you're Charlie, anxiously awaiting an email from Alice;)...
To highlight patterns while viewing the whole file, h can do this.
Plus it uses different colors for different patterns.
cat FILE | h 'PAT1' 'PAT2' ...
You can also pipe the output of h to less -R for better reading.
To grep and use 1 color for each pattern, cxpgrep could be a good fit.
Use ripgrep, aka rg: https://github.com/BurntSushi/ripgrep
rg --passthru...
Color is the default:
rg -t tf -e 'key.*tfstate' -e dynamodb_table
--passthru
Print both matching and non-matching lines.
Another way to achieve a similar effect is by modifying your pattern to
match the empty string.
For example, if you are searching using rg foo then using
rg "^|foo" instead will emit every line in every file searched, but only
occurrences of foo will be highlighted.
This flag enables the same behavior without needing to modify the pattern.
Sacrilege, granted, but grep has gotten complacent.
brew/apt/rpm/whatever install ripgrep
You'll never go back.
another dirty way:
grep -A80 -B80 --color FIND_THIS IN_FILE
I did an
alias grepa='grep -A80 -B80 --color'
in bashrc.
Here is a shell script that uses Awk's gsub function to replace the text you're searching for with the proper escape sequence to display it in bright red:
#! /bin/bash
awk -vstr=$1 'BEGIN{repltext=sprintf("%c[1;31;40m&%c[0m", 0x1B,0x1B);}{gsub(str,repltext); print}' $2
Use it like so:
$ ./cgrep pattern [file]
Unfortunately, it doesn't have all the functionality of grep.
For more information , you can refer to an article "So You Like Color" in Linux Journal
One other answer mentioned grep's -Cn switch which includes n lines of Context. I sometimes do this with n=99 as a quick-and-dirty way of getting [at least] a screenfull of context when the egrep pattern seems too fiddly, or when I'm on a machine on which I've not installed rcg and/or ccze.
I recently discovered ccze which is a more powerful colorizer. My only complaint is that it is screen-oriented (like less, which I never use for that reason) unless you specify the -A switch for "raw ANSI" output.
+1 for the rcg mention above. It is still my favorite since it is so simple to customize in an alias. Something like this is usually in my ~/.bashrc:
alias tailc='tail -f /my/app/log/file | rcg send "BOLD GREEN" receive "CYAN" error "RED"'
Alternatively you can use The Silver Searcher and do
ag <search> --passthrough
I use following command for similar purpose:
grep -C 100 searchtext file
This will say grep to print 100 * 2 lines of context, before & after of the highlighted search text.
It might seem like a dirty hack.
grep "^\|highlight1\|highlight2\|highlight3" filename
Which means - match the beginning of the line(^) or highlight1 or highlight2 or highlight3. As a result, you will get highlighted all highlight* pattern matches, even in the same line.
Ok, this is one way,
wc -l filename
will give you the line count -- say NN, then you can do
grep -C NN --color=always filename
If you want highlight several patterns with different colors see this bash script.
Basic usage:
echo warn error debug info 10 nil | colog
You can change patterns and colors while running pressing one key and then enter key.
Here's my approach, inspired by #kepkin's solution:
# Adds ANSI colors to matched terms, similar to grep --color but without
# filtering unmatched lines. Example:
# noisy_command | highlight ERROR INFO
#
# Each argument is passed into sed as a matching pattern and matches are
# colored. Multiple arguments will use separate colors.
#
# Inspired by https://stackoverflow.com/a/25357856
highlight() {
# color cycles from 0-5, (shifted 31-36), i.e. r,g,y,b,m,c
local color=0 patterns=()
for term in "$#"; do
patterns+=("$(printf 's|%s|\e[%sm\\0\e[0m|g' "${term//|/\\|}" "$(( color+31 ))")")
color=$(( (color+1) % 6 ))
done
sed -f <(printf '%s\n' "${patterns[#]}")
}
This accepts multiple arguments (but doesn't let you customize the colors). Example:
$ noisy_command | highlight ERROR WARN
Is there some way I can tell grep to print every line being read
regardless of whether there's a match?
Option -C999 will do the trick in the absence of an option to display all context lines. Most other grep variants support this too. However: 1) no output is produced when no match is found and 2) this option has a negative impact on grep's efficiency: when the -C value is large this many lines may have to be temporarily stored in memory for grep to determine which lines of context to display when a match occurs. Note that grep implementations do not load input files but rather reads a few lines or use a sliding window over the input. The "before part" of the context has to be kept in a window (memory) to output the "before" context lines later when a match is found.
A pattern such as ^|PATTERN or PATTERN|$ or any empty-matching sub-pattern for that matter such as [^ -~]?|PATTERN is a nice trick. However, 1) these patterns don't show non-matching lines highlighted as context and 2) this can't be used in combination with some other grep options, such as -F and -w for example.
So none of these approaches are satisfying to me. I'm using ugrep, and enhanced grep with option -y to efficiently display all non-matching output as color-highlighted context lines. Other grep-like tools such as ag and ripgrep also offer a pass-through option. But ugrep is compatible with GNU/BSD grep and offers a superset of grep options like -y and -Q. For example, here is what option -y shows when combined with -Q (interactive query UI to enter patterns):
ugrep -Q -y FILE ...
Also try:
egrep 'pattern1|pattern2' FILE.txt | less -Sp 'pattern1|pattern2'
This will give you a tabular output with highlighted pattern/s.