Not sure how to explain this but, what I am trying to achieve is this:
- tailing a file and grepping for a patter A
- then I want to pipe into another customGrepFunction where it matches pattern B, and if B matches echo something out. Need the customGrepFunction in order to do some other custom stuff.
The sticky part here is how to make the grepCustomFunction work here.In other words when only patternA matches echo the whole line and when both patterA & patternB match printout something custom:
when I only run:
tail -f file.log | grep patternA
I can see the pattenA rows are being printed/tailed however when I add the customGrepFunction nothing happens.
tail -f file.log | grep patternA | customGrepFunction
And the customGrepFunction should be available globally in my bin folder:
customGrepFunction(){
if grep patternB
then
echo "True"
fi
}
I have this setup however it doesn't do what I need it to do, it only echos True whenever I do Ctrl+C and exit the tailing.
What am I missing here?
Thanks
What's Going Wrong
The code: if grep patternB; then echo "true"; fi
...waits for grep patternB to exit, which will happen only when the input from tail -f file.log | grep patternA hits EOF. Since tail -f waits for new content forever, there will never be an EOF, so your if statement will never complete.
How To Fix It
Don't use grep on the inside of your function. Instead, process content line-by-line and use bash's native regex support:
customGrepFunction() {
while IFS= read -r line; do
if [[ $line =~ patternB ]]; then
echo "True"
fi
done
}
Next, make sure that grep isn't buffering content (if it were, then it would be written to your code only in big chunks, delaying until such a chunk is available). The means to do this varies by implementation, but with GNU grep, it would look like:
tail -f file.log | grep --line-buffered patternA | customGrepFunction
Related
in my script i need to loop through lines in a file, once i find some specific line i need to save it to variable so later on i can use it outside the loop, i tried the following but it wont' work:
count=0
res=""
python my.py -p 12345 |
while IFS= read -r line
do
count=$((count+1))
if [ "$count" -eq 5 ]; then
res=`echo "$line" | xargs`
fi
done
echo "$res"
it output nothing, i also tried this,
res=""
... in the loop...
res=$res`echo "$line" | xargs`
still nothing. please help. thanks.
Update: Thanks for all the help. here is my final code:
res=python my.py -p 12345 | sed -n '5p' | xargs
for finding a specific line in a file, have you considered using grep?
grep "thing I'm looking for" /path/to/my.file
this will output the lines that match the thing you're looking for. Moreover this can be piped to xargs as in your question.
If you need to look at a particularly numbered line of a file, consider using the head and tail commands (which can also be piped to grep).
cat /path/to/my.file | head -n5 | tail -n1 | grep "thing I'm looking for"
These commands take the first lines specified (in this case, 5 and 1 respectively) and only prints those out. Hopefully this will help you accomplish your task.
Happy coding! Leave a comment if you have any questions.
How do I grep without actually filtering, or highlighting?
The goal is to find out if a certain text is in the output, without affecting the output. I could tee to a file and then inspect the file offline, but, if the output is large, that is a waste of time, because it processes the output only after the process is finished:
command | tee file
file=`mktemp`
if grep -q pattern "$file"; then
echo Pattern found.
fi
rm "$file"
I thought I could also use grep's before (-B) and after (-A) flags to achieve live processing, but that won't output anything if there are no matches.
# Won't even work - DON'T USE.
if command | grep -A 1000000 -B 1000000 pattern; then
echo Pattern found.
fi
Is there a better way to achieve this? Something like a "pretend you're grepping and set the exit code, but don't grep anything".
(Really, what I will be doing is to pipe stderr, since I'm looking for a certain error, so instead of command | ... I will use command 2> >(... >&2; result=${PIPESTATUS[*]}), which achieves the same, only it works on stderr.)
If all you want to do is set the exit code if a pattern is found, then this should do the trick:
awk -v rc=1 '/pattern/ { rc=0 } 1; END {exit rc}'
The -v rc=1 creates a variable inside the Awk program called rc (short for "return code") and initializes it to the value 1. The stanza /pattern/ { rc=0 } causes that variable to be set to 0 whenever a line is encountered that matches the regular expression pattern. The 1; is an always-true condition with no action attached, meaning the default action will be taken on every line; that default action is printing the line out, so this filter will copy its input to its output unchanged. Finally, the END {exit rc} runs when there is no more input left to process, and ensures that awk terminates with the value of the rc variable as its process exit status: 0 if a match was found, 1 otherwise.
The shell interprets exit code 0 as true and nonzero as false, so this command is suitable for use as the condition of a shell if or while statement, possibly at the end of a pipeline.
To allow output with search result you can use awk:
command | awk '/pattern/{print "Pattern found"} 1'
This will print "Pattern found" when pattern is matched in any line. (Line will be printed later)
If you want Line to print before then use:
command | awk '{print} /pattern/{print "Pattern found"}'
EDIT: To execute any command on match use:
command | awk '/pattern/{system("some_command")} 1'
EDIT 2: To take care of special characters in keyword use this:
command | awk -v search="abc*foo?bar" 'index($0, search) {system("some_command"); exit} 1'
Try this script. It will not modify anything of output of your-command and sed exit with 0 when pattern is found, 1 otherwise. I think its what you want from my understand of your question and comment.:
if your-command | sed -nr -e '/pattern/h;p' -e '${x;/^.+$/ q0;/^.+$/ !q1}'; then
echo Pattern found.
fi
Below is some test case:
ubuntu-user:~$ if echo patt | sed -nr -e '/pattern/h;p' -e '${x;/^.+$/ q0;/^.+$/ !q1}'; then echo Pattern found.; fi
patt
ubuntu-user:~$ if echo pattern | sed -nr -e '/pattern/h;p' -e '${x;/^.+$/ q0;/^.+$/ !q1}'; then echo Pattern found.; fi
pattern
Pattern found.
Note previous script fails to work when there is no ouput from your-command because then sed will not run sed expression and exit with 0 all the time.
I take it you want to print out each line of your output, but at the same time, track whether or not a particular pattern is found. Simply passing the output to sed or grep would affect the output. You need to do something like this:
pattern=0
command | while read line
do
echo "$line"
if grep -q "$pattern" <<< "$lines"
then
((pattern+=1))
fi
done
if [[ $pattern -gt 0 ]]
then
echo "Pattern was found $pattern times in the output"
else
echo "Didn't find the pattern at all"
fi
ADDENDUM
If the original command has both stdout and stderr output, which come in a specific order, with the two possibly interleaved, then will your solution ensure that the outputs are interleaved as they normally would?
Okay, I think I understand what you're talking about. You want both STDERR and STDOUT to be grepped for this pattern.
STDERR and STDOUT are two different things. They both appear on the terminal window because that's where you put them. The pipe (|) only takes STDOUT. STDERR is left alone. In the above, only the output of STDOUT would be used. If you want both STDOUT and STDERR, you have to redirect STDERR into STDOUT:
pattern=0
command 2>&1 | while read line
do
echo "$line"
if grep -q "$pattern" <<< "$lines"
then
((pattern+=1))
fi
done
if [[ $pattern -gt 0 ]]
then
echo "Pattern was found $pattern times in the output"
else
echo "Didn't find the pattern at all"
fi
Note the 2>&1. This says to take STDERR (which is File Descriptor 2) and redirect it into STDOUT (File Descriptor 1). Now, both will be piped into that while read loop.
The grep -q will prevent grep from printing out its output to STDOUT. It will print to STDERR, but that shouldn't be an issue in this case. Grep only prints out STDERR if it cannot open a file requested, or the pattern is missing.
You can do this:
echo "'search string' appeared $(command |& tee /dev/stderr | grep 'search string' | wc -l) times"
This will print the entire output of command followed by the line:
'search string' appeared xxx times
The trick is, that the tee command is not used to push a copy into a file, but to copy everything in stdout to stderr. The stderr stream is immediately displayed on the screen as it is not connected to the pipe, while the copy on stdout is gobbled up by the grep/wc combination.
Since error messages are usually emitted to stderr, and you said that you want to grep for error messages, the |& operator is used for the first pipe to combine the stderr of command into its stdout, and push both into the tee command.
On a Mac, I want to determine if there are any sleep assertions present, using pmset. If there are, extract only that information and omit unnecessary information.
If grep returns nothing I want to print "Nothing".
if pmset -g | grep pre ; then pmset -g | grep pre | cut -d'(' -f2 | cut -d')' -f1 ; else printf "Nothing\n" ; fi
The problem is that the first grep result is printed, and so is the formatted one. For example this is what I get if a backup is in progress:
sleep 15 (sleep prevented by backupd)
sleep prevented by backupd
I don't want the first line, and want to discard it. I only want the second line to print ("sleep prevented by backupd").
If the grep result is empty I want to indicate that with the text "Nothing". The above script works OK for that.
There are probably many more elegant solutions but I've been searching days for one.
If i understand your question properly, you simply need to discard the output of first grep irrespective of the output it provides. If it's so, then you can use -q option provided by grep.
From the man page for 'grep':
-q, --quiet, --silent
Quiet; do not write anything to standard output. Exit immediately with zero status if any match is found, even if an error was
detected. Also see the -s or --no-messages option. (-q is specified by POSIX.)
Something like this:
if ifconfig | grep -q X; then
ifconfig | grep Mi | cut -d'(' -f2
else
printf "Nothing\n"
fi
Obviously in the above example, output of ifconfig will not change every time. Just used as an example. ;)
Redirect the output to /dev/null:
if pmset -g | grep pre >/dev/null 2>&1 ; then
pmset -g | grep pre | cut -d'(' -f2 | cut -d')' -f1
else
printf "Nothing\n"
fi
This is maybe a little more succinct. It gets grep to only output the part of the line that matches the pattern instead of the whole line, by using grep -o:
#!/bin/bash
SLEEP=$(pmset -g | grep -o "sleep prevented.*[^)]")
if [ -z "$SLEEP" ]; then
echo Nothing
else
echo $SLEEP
fi
The pattern is sleep prevented and any characters following until a ) is encountered.
I'm trying to use grep to go through some logs and only select the most recent entries. The logs have years of heavy traffic on them so it's silly to do
tac error.log | grep 2012
tac error.log | grep "Jan.2012"
etc.
and wait for 10 minutes while it goes through several million lines which I already know are not going to match. I know there is the -m option to stop at the first match but I don't know of a way to make it stop at first non-match. I could do something like grep -B MAX_INT -m 1 2011 but that's hardly an optimal solution.
Can grep handle this or would awk make more sense?
How about using awk like this:
tac error.log | awk '{if(/2012/)print;else exit}'
This should exit as soon as a line not matching 2012 is found.
Here is a solution in python:
# foo.py
import sys, re
for line in sys.stdin:
if re.match(r'2012', line):
print line,
continue
break
you#host> tac foo.txt | python foo.py
I don't think grep supports this.
But here is my "why did we have awk again" answer:
tail -n `tac biglogfile | grep -vnm1 2012 | sed 's/:.*//' | xargs expr -1 +` biglogfile
Note that this isn't going to be exact if your log is being written to.
The excellent one-line scripts for sed page to the rescue:
# print section of file between two regular expressions (inclusive)
sed -n '/Iowa/,/Montana/p' # case sensitive
In other words, you should be able to do the following:
sed -n '/Jan 01 2012/,/Feb 01 2012/p' error.log | grep whatevs
Here is an example that parses the user's dot-plan file and stops at the first non-matching line:
PID=$$
while read ln; do
echo $ln | {
if grep "^[-*+] " >/dev/null; then
# matched
echo -e $ln
elif grep "^[#]" >/dev/null; then
# ignore comment line
:
else
# stop at first non-matching line
kill $PID
fi
}
done <$HOME/.plan
Of course this approach is considerably slower than if Grep reads the lines but at least you can incorporate several cases (not just the non-match).
For more complex scripts, it is worth noting that Bash can also apply regular expressions to variables, i.e. you can also do completely without grep.
I am writing some BASH shell script that will continuously check a file to see if the file already contains "Completed!" before proceeding. (Of course, assume the file is being updated and will eventually contain the phrase "Completed!")
I am not sure how to do this. Thank you for your help.
You can do something like:
while ! grep -q -e 'Completed!' file ; do
sleep 1 # Or some other number of seconds
done
# Here the file contains completed
Amongst the standard utilities, tail has an option to keep reading from a file: tail -f. So filter the output of tail -f.
<some_file tail -f -n +1 | grep 'Completed!' | head -n 1 >/dev/null
There may be a delay due to buffering. You can at least reduce the delay by using fewer tools in the pipeline. In fact, some implementations of tail never buffer when you do tail -f, so the following snippet will return as soon as Completed! is written to the file.
<some_file tail -f -n +1 | sed -e '/Completed!/ q'
This assumes that the file is being appended to by some other tool. If the file is overwritten by the data-producing program after you start tail, this solution won't work. You can search the file periodically. On some systems you can call a notification mechanism to know whenever the file changes, e.g. with inotifywait under Linux.
I've done this in Kornshell:
tail -f somefile | while read line
do
echo $line
[[ $line == *Completed!* ]] && break
done
Note no quotes around the *Completed!* string. This allows the double square brackets to do glob pattern matching instead of string matching.
This seems to work in BASH too. However, the line with the Completed must end in a NL. Otherwise, it'll take an extra line before it breaks the loop.
You can use grep too:
tail -f somefile | while read line
do
echo $line
grep -iq "Completed!" && break
done
The -q parameter means quiet. If your grep doesn't take the -q parameter, you might have to pipe it to /dev/null. The -i is ignore case. Whether you want to do that is up to you.
The advantage is that you aren't doing any processing unless there's a line to read. Using sleep may mean you miss the line, or that you're processing when no line has been added to the file.
Using grep in a pipe you may turn on line buffering mode by adding the --line-buffered option!