grep for a pattern "variable=value" and returning only matching entries having a value > threshold - bash

I am searching a kubernetes pod logs for the pattern “variable=value” ( e.g., variable=10 or variable=500) using the command below:
Kubectl logs -f | grep “variable=”
My question is that wether it is possible to modify the command above in order to return the logs where the variable value is greater than some threshold, e.g, for threshold=300, variable=301, variable=302 would be filetered in, but variable=299 would be filtered out
I know I can develop a small program for this, but rather I want a rapid solution in the command line direcly without the hassle of writing a small prgram.

You didn't provide any sample input so it's a guess but this may be what you're trying to do:
awk -F'=' '($1=="variable") && ($2>300)' file
If that's not all you need then please edit your question to include a Minimal, Reproducible Example with concise, testable sample input, expected output and your attempt to solve the problem yourself so we can help you further. See [ask] and look at existing questions that have been upvoted and answered for examples.

As a test, I've created this file:
Prompt> cat test.txt
var=2
var=22
var=222
blabla
First, I filter our the variable assignment lines:
Prompt> grep "var=" test.txt
var=2
var=22
var=222
Then, I filter on the condition of the values, in two ways:
Prompt> grep "var=" test.txt | awk -F '=' '{if ($2 > 25) print $1 "=" $2}'
var=222
Prompt> grep "var=" test.txt | awk -F '=' '{if ($2 < 25) print $1 "=" $2}'
var=2
var=22

I would use perl, here I am printing the logs from a pod called bash-pod. Filtering, only the lines with threshold=<integer-greater-than-80> are present.
kubectl logs -f bash-pod |perl -ne '/.*threshold=(\d+).*/m;print if $1> 80'
The above code is tested with the below pod:
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: bash-pod
name: bash-pod
spec:
containers:
- image: bash
name: bash-pod
resources: {}
command: ['bash','-c','while true;do echo "threshold=$(( ( RANDOM % 100 ) + 1 ))";sleep 1;done']
dnsPolicy: ClusterFirst
restartPolicy: Never
status: {}
Output:
kubectl logs -f bash-pod |perl -ne '/.*threshold=(\d+).*/m;print if $1> 80'
threshold=82
threshold=90
threshold=89
threshold=90
threshold=85
threshold=83
threshold=86
threshold=83
...
.....
Note: If you want to take float into consideration then you can use .*threshold=(\d+(?:\.\d+)?).* as new regex.

Here's a flexible solution that allows you specify parameters you like, and functions the same :
< test_filter_threshold.txt |
mawk 'NF*=(____=="<")!=((___~"."?+___:___\
)<=+$(NF*=(!_<NF)*(__==$!_)))' FS== OFS== \
__="var" ___="1" ____=">="
var=2
var=22
var=222
___=4 ____='<'
var=2
___=200 ____='<'
var=2
var=22
___=200 ____='>='
var=222
___=200 ____= # "<" is strictly less than,
# all else def. to ">="
var=222
___='3.14159' ____='>=' # float point thresholds (TRHD) are valid
var=22
var=222
___= ____= # missing TRHD def. to all
var=2
var=22
var=222
___='abc' ____='>=' # invalid TRHD def. to non-zero positive
var=2
var=222
1 var=2
2 var=-22
3 var=222
4 blabla

Related

Wrong search result in a file through Bash script

I am searching an event field in a file but is giving wrong output. I am searching gpio-keys event in input devices for which I have written a script, but I'm unable to print anything in output file (in my case I am writing in a button device file it is null always). Please help me to figure out this. Where am I doing wrong in script file?
Bash script:
#!/bin/bash
if grep -q "gpio-keys" /proc/bus/input/devices ; then
EVENT=$(cat /proc/bus/input/devices | grep "Handlers=kbd")
foo= `echo $EVENT | awk '{for(i=1;i<=NF;i++) if($i=="evbug")printf($(i-1))}'`
#foo=${EVENT:(-7)}
echo -n $foo > /home/ubuntu/Setups/buttonDevice
fi
i am still not able to get anything in buttondevce
That's no wonder, since in the input line
H: Handlers=kbd event0
there's nowhere the evbug your awk script is looking for.
I my case it is event0 but it may vary also depends on how kernel allows.
If it is event0 or similar, then it's nonsensical to look for evbug. Change the statement
if($i=="evbug")printf($(i-1))
to
if ($i~"event") print $i
(using regular expression match).
I have rewritten my script like above. but through it, I have got two events(event0, event3) but … my input devices are many but i want the gpio-keys event
Aha - in order to take only the handler line from the gpio-keys section, you can use sed with an address range:
EVENT=`sed -n '/gpio-keys/,/Handlers=kbd/s/.*Handlers=kbd //p' </proc/bus/input/devices`
Prakash, I don't have access to your google drive. But I just want to give you some suggestion:-
foo= `echo $EVENT | awk '{for(i=1;i<=NF;i++) if($i=="evbug")printf($(i-1))}'`
This is old style now. Better use like below:-
foo=$(echo $EVENT | awk '{for(i=1;i<=NF;i++) if($i=="evbug")printf($(i-1))}')
Also always use double quotes "" when echoing a variable. See below:-
echo -n "$foo" > /home/ubuntu/Setups/buttonDevice
Try with the below code it will work for you
#!/bin/bash
if grep "gpio-keys" /proc/bus/input/devices >/dev/null ; then
cat /proc/bus/input/devices | grep "Handlers=kbd" | awk '{for(i=1;i<=NF;i++){ if($i ~ /eve/){printf "%s \n", $i} } }') > /home/ubuntu/Setups/buttonDevice
fi
The output in buttonDevice would be
event0
event1
.
.
.
.
event100

Find two strings in a list of files and get filename [duplicate]

This question already has answers here:
Find files containing multiple strings
(6 answers)
Closed 4 years ago.
I have the following files:
100005.txt 107984.txt 116095.txt 124152.txt 133339.txt 139345.txt 18147.txt 25750.txt 32647.txt 40390.txt 48979.txt 56502.txt 64234.txt 72964.txt 80311.txt 888.txt 95969.txt
100176.txt 108084.txt 116194.txt 124321.txt 133435.txt 139438.txt 18331.txt 25940.txt 32726.txt 40489.txt 49080.txt 56506.txt 64323.txt 73063.txt 80481.txt 88958.txt 9601.txt
100347.txt 108255.txt 116378.txt 124494.txt 133531.txt 139976.txt 18420.txt 26034.txt 32814.txt 40589.txt 49082.txt 56596.txt 64414.txt 73163.txt 80580.txt 89128.txt 96058.txt
100447.txt 108343.txt 116467.txt 124594.txt 133627.txt 140519.txt 18509.txt 26128.txt 32903.txt 40854.txt 49254.txt 56768.txt 64418.txt 73498.txt 80616.txt 89228.txt 96148.txt
100617.txt 108432.txt 11647.txt 124766.txt 133728.txt 14053.txt 1866.txt 26227.txt 32993.txt 41026.txt 49308.txt 56857.txt 6449.txt 73670.txt 80704.txt 89400.txt 96239.txt
10071.txt 108521.txt 116556.txt 124854.txt 133830.txt 141062.txt 18770.txt 26327.txt 33093.txt 41197.txt 49387.txt 57029.txt 64508.txt 7377.txt 80791.txt 89500.txt 96335.txt
100788.txt 10897.txt 116746.txt 124943.txt 133866.txt 141630.txt 18960.txt 2646.txt 33194.txt 41296.txt 4971.txt 57128.txt 64680.txt 73841.txt 80880.txt 89504.txt 96436.txt
Some of the files look like:
spec:
annotations:
name: "ubuntu4"
labels:
key: "cont_name"
value: "ubuntuContainer4"
labels:
key: "cont_service"
value: "UbuntuService4"
task:
container:
image: "ubuntu:latest"
args: "tail"
args: "-f"
args: "/dev/null"
mounts:
source: "/home/testVolume"
target: "/opt"
replicated:
replicas: 1
I want to get every filename that contains ubuntu AND replicas.
I have tried awk '/ubuntu/ && /replicas/{print FILENAME}' *.txt but it doesn't seem to work for me.
Any ideas on how to fix this?
Grep can return a list of the files that match a string. You can nest that grep call so that you first get a list of files that match ubuntu, then use that list of files to get a list of files that match replicas.
grep -l replicas $( grep -l ubuntu *.txt )
This does assume that at least one file will match ubuntu. To get around that limitation, you can add a test for the existence of one file first, and then do the combined search:
grep -q ubuntu *.txt && grep -l replicas $( grep -l ubuntu *.txt )
Check if both strings appear in a given file by using a counter for each and then checking if they were incremented. You can do this with BEGINFILE, available on GNU awk:
awk 'BEGINFILE {ub=0; re=0}
/ubuntu/ {ub++}
/replicas/ {re++}
(ub>0 && re>0) {print FILENAME; nextfile}' *.txt
This sets two counters to 0 when it starts to read a file: one for one string and another one for the other. When one of the patterns is found, it increments its corresponding counter. Then it keeps checking if the two counters have been incremented. If so, it prints its filename using the FILENAME variable that contains that string. Also, it skips the rest of the file using nextfile, since there is no need to continue checking for the patterns.
awk '/ubuntu/ && /replicas/{print FILENAME}' *.txt
looks for both regexps on the same line. To find them both in the same file but possibly on separate lines with GNU awk for ENDFILE is:
awk '/ubuntu/{u=1} /replicas/{r=1} ENDFILE{if (u && r) print FILENAME; u=r=0}' *.txt
or more efficiently adding gawks nextfile construct and preferentially switching to BEGINFILE (as #fedorqui already showed) instead of ENDFILE since all that remains between file reads is to set the 2 variables:
awk 'BEGINFILE{u=r=0} /ubuntu/{u=1} /replicas/{r=1} u && r{print FILENAME; nextfile}' *.txt
With other awks it'd be:
awk '
FNR==1{prt()} /ubuntu/{u=1} /replicas/{r=1} END{prt()}
function prt() {if (u && r) print fname; fname=FILENAME; u=r=0}
' *.txt
If no subdirs have to been visited:
for f in *.txt
do
grep -q -m1 'ubuntu' $f && grep -q -m1 'replicas' $f && echo "found: $f"
done
or as oneliner:
for f in *.txt ; do grep -q -m1 'ubuntu' $f && grep -q -m1 replicas $f && echo found:$f ; done
The -q makes grep quiet, so the matches aren't display, the -m1 only searches for 1 match, so grep can report a match fast.
The && is short circuiting, so if the first grep doesn't find anything, the second isn't tried.
For working on the files further down the pipeline, you will of course eliminate the chatty "found: ".

Print line after the match in grep [duplicate]

This question already has answers here:
How to show only next line after the matched one?
(14 answers)
Closed 6 years ago.
I'm trying to get the current track running from 'cmus-remote -Q'
Its always underneath of this line
tag genre Various
<some track>
Now, I need to keep it simple because I want to add it to my i3 bar. I used
cmus-remote -Q | grep -A 1 "tag genre"
but that grep's the 'tag' line AND the line underneath.
I want ONLY the line underneath.
With sed:
sed -n '/tag genre/{n;p}'
Output:
$ cmus-remote -Q | sed -n '/tag genre/{n;p}'
<some track>
If you want to use grep as the tool for this, you can achieve it by adding another segment to your pipeline:
cmus-remote -Q | grep -A 1 "tag genre" | grep -v "tag genre"
This will fail in cases where the string you're searching for is on two lines in a row. You'll have to define what behaviour you want in that case if we're going to program something sensible for it.
Another possibility would be to use a tool like awk, which allows for greater compexity in the line selection:
cmus-remote -Q | awk '/tag genre/ { getline; print }'
This searches for the string, then gets the next line, then prints it.
Another possibility would be to do this in bash alone:
while read line; do
[[ $line =~ tag\ genre ]] && read line && echo "$line"
done < <(cmus-remote -Q)
This implements the same functionality as the awk script, only using no external tools at all. It's likely slower than the awk script.
You can use awk instead of grep:
awk 'p{print; p=0} /tag genre/{p=1}' file
<some track>
/tag genre/{p=1} - sets a flag p=1 when it encounters tag genre in a line.
p{print; p=0} when p is non-zero then it prints a line and resets p to 0.
I'd suggest using awk:
awk 'seen && seen--; /tag genre/ { seen = 1 }'
when seen is true, print the line.
when seen is true, decrement the value, so it will no longer true after the desired number of lines are printed
when the pattern matches, set seen to the number of lines to be printed

Parse a two values from the file

Part of my file looks like so:
STATUS REPORT FOR JOB: Job_logging
Generated: 2014-03-14 07:05:03
Job start time=2014-03-13 06:37:49
Job end time=2014-03-13 06:37:51
Job elapsed time=00:00:02
Job status=1 (Finished OK)
Stage: Oracle_Connector_0, 1 rows input
Stage start time=2014-03-13 06:37:51, end time=2014-03-13 06:37:51, elapsed=00:00:00
Link: DSLink2, 1 rows
Stage: Peek_3, 1 rows input
Stage start time=2014-03-13 06:37:51, end time=2014-03-13 06:37:51, elapsed=00:00:00
Status code = 0
Link: DSLink2, 1 rows
I need to extract values that stand for Job start time and Job end time
So i need 2014-03-13 06:37:49 and 2014-03-13 06:37:51 to be saved into two separate variables: v1 and v2.
How do I do that using BASH?
I've already killed about an hour playing with strings concatanation and sed but still got nothing.
Little help, please?
Using awk it can be found in single line:
awk -F 'Job (start|end) time=' 'NF>1{print $2}' file
2014-03-13 06:37:49
2014-03-13 06:37:51
To read both values in variables:
IFS=';' && read v1 v2 < <(awk -F 'Job (start|end) time=' 'NF>1{printf "%s;", $2}' file)
You can use grep for this:
$ grep -Po '(?<=Job start time=).*' file
2014-03-13 06:37:49
$ grep -Po '(?<=Job end time=).*' file
2014-03-13 06:37:51
It used a look-behind that checks what comes after Job start/end time= in the given file.
And to store into a variable, use
$ var=$(grep -Po '(?<=Job end time=).*' file)
$ echo "$var"
2014-03-13 06:37:51
This is simple and easy to read:
grep "Job start time" test.txt | cut -d"=" -f2
grep "Job end time" test.txt | cut -d"=" -f2
This searches for the lines containing your specific string, sets the delimeter as = between the two, and takes the field on the right of it.
With sed:
v1=$(sed -rn 's/.*Job start time=(.*)/\1/p' yourfile)
v2=$(sed -rn 's/.*Job end time=(.*)/\1/p' yourfile)
eval `sed -n 's/^ *Job start time=\(.*\)/v1="\1"/p
s/^ *Job end time=\(.*\)/v2="\1"/p' YourFile`
tested on aix/bash (so no GNU sed)

How do I get all the text between the last two instances of a token in bash?

I’m using bash and running the following command to get all the file text between two tokens (including the tokens themselves):
cat /usr/java/jboss/standalone/log/server.log | sed -n \
'/Starting deployment of "myproject.war"/,/Registering web context: \/myproject/p'
However, sometimes the tokens occur multiple times in the file. How do I adjust the above so that only the text between the last two occurrences of the tokens (including the tokens themselves) will be returned?
How about some tic-tac-toe.
tac /usr/java/jboss/standalone/log/server.log |
awk '/Registering web context: \/myproject/{p=1;++cnt}/Starting deployment of "myproject.war"/{if(cnt==2){print $0;exit};print $0;p=0}p' |
tac
This solution is not efficient, but easier to understand:
file='/usr/java/jboss/standalone/log/server.log'
s1='Starting deployment of "myproject.war"'
s2='Registering web context: \/myproject'
sed -n '/'"$s1"'/,/'"$s2"'/p' "$file" |
tac |
awk '/'"$s1"'/ {print;exit} 1' |
tac
Lets sed report ALL ranges first.
Reverses the result using tac (on OSX, use tail -r).
Using awk, outputs everything up to and including the first occurrence of the first substring, which - in the reversed result - spans the end of the last range to the start of the last range.
Reverses the output from awk to render the last range in correct order.
Note: For consistency with the variable use in the sed command I've spliced a variable reference directly into the awk program, too, which is otherwise poor practice (use -v to pass variables instead).
You can do this in native bash -- no need for awk, tac, or any other external tool.
token1='Starting deployment of "myproject.war"'
token2='Registering web context: /myproject/'
writing=0
while read -r; do
(( ! writing )) && [[ $REPLY = $token1 ]] && {
# start collecting content, into an empty buffer, when we see token1
writing=1 # set flag to store lines we see
collected_content=() # clear the array of lines found so far
}
(( writing )) && {
# when the flag is set, collect content into an array
collected_content+=( "$REPLY" )
}
[[ $REPLY = $token2 ]] && {
# stop collecting content when we see token2
writing=0
}
done <server.log # redirect from the log into the loop
# print all collected lines
printf '%s\n' "${collected_content[#]}"
This awk can work:
awk '/Starting deployment of "myproject.war"/{i=0; s=1; delete a;}
s{a[++i]=$0}
/Registering web context: \/myproject/{s=0}
END {print i; for (k=1; k<=i; k++) print a[k]}' file
With perl:
perl -0xFF -nE '#x = /WWWW Starting deployment of "myproject.war"(.*?)Registering web context: \/myproject/sg; say $x[-1] ' file

Resources