filter information by using sed Unix command - shell

I try to filter the system log by running the sed command
like:
error
info_a1
info_a2
info_a3
crit
info_b
error
info_c1
info_c2
warn
info_d
error
info_x
error
info_p
info
info_w
error
info_z1
info_z2
I expect to get all the error messages likeļ¼š
error
info_a1
info_a2
info_a3
error
info_c1
info_c2
error
info_x
error
info_p
error
info_z1
info_z2
i use
sed -n "/error/, /[info|warn|crit|]/p"
but it does not work pretty well.

awk '
/^error/ {p=1}
/^(info|warn|crit)/ {p=0}
p
'

If your objective is to print the error message AND the next line after it, then use
cat file_name | grep 'error' -A 1 | grep -v '\-\-'

if you just want to get all lines with "error" in it,
grep "error" file
if you insist on sed
sed -n '/error/p' file
or even awk
awk '/error/' file
or Ruby
ruby -ne 'print if /error' file

Here is a sed version:
sed -n ':a;/^error/{h;:b;n;${/^[[:blank:]]/{H;x;/^error/{p;q}}};/^[^[:blank:]]/{x;p;x;ba};H;bb}' inputfile
Explanation:
:a # label a
/^error/{ # if pattern space (patt) begins with "error"
h # copy the line to hold space (hold)
:b # label b
n # read the next line
${ # if it's the last line of input
/^[[:blank:]]/{ # if it begins with a space, tab, etc.
H # append it to hold
x # exchange patt and hold
/^error/{ # if patt begins with "error"
p # print it
q # quit
}
}
}
/^[^[:blank:]]/{ # if patt starts with a non-blank (a new block of input has started
x # exchange patt and hold
p # print patt
x # exchange patt and hold
ba # branch to label a
}
H # append patt to hold (save a line of the current block)
bb # branch to b
}

Related

How i should use sed for delete specific strings and allow duplicate with more characters?

i had generate a list of file, and this had 17417 lines like :
./usr
./usr/share
./usr/share/mime-info
./usr/share/mime-info/libreoffice7.0.mime
./usr/share/mime-info/libreoffice7.0.keys
./usr/share/appdata
./usr/share/appdata/libreoffice7.0-writer.appdata.xml
./usr/share/appdata/org.libreoffice7.0.kde.metainfo.xml
./usr/share/appdata/libreoffice7.0-draw.appdata.xml
./usr/share/appdata/libreoffice7.0-impress.appdata.xml
./usr/share/appdata/libreoffice7.0-base.appdata.xml
./usr/share/appdata/libreoffice7.0-calc.appdata.xml
./usr/share/applications
./usr/share/applications/libreoffice7.0-xsltfilter.desktop
./usr/share/applications/libreoffice7.0-writer.desktop
./usr/share/applications/libreoffice7.0-base.desktop
./usr/share/applications/libreoffice7.0-math.desktop
./usr/share/applications/libreoffice7.0-startcenter.desktop
./usr/share/applications/libreoffice7.0-calc.desktop
./usr/share/applications/libreoffice7.0-draw.desktop
./usr/share/applications/libreoffice7.0-impress.desktop
./usr/share/icons
./usr/share/icons/gnome
./usr/share/icons/gnome/16x16
./usr/share/icons/gnome/16x16/mimetypes
./usr/share/icons/gnome/16x16/mimetypes/libreoffice7.0-oasis-formula.png
The thing is i want to delete the lines like :
./usr
./usr/share
./usr/share/mime-info
./usr/share/appdata
./usr/share/applications
./usr/share/icons
./usr/share/icons/gnome
./usr/share/icons/gnome/16x16
./usr/share/icons/gnome/16x16/mimetypes
and the "." at the start, for the result must be like :
/usr/share/mime-info/libreoffice7.0.mime
/usr/share/mime-info/libreoffice7.0.keys
/usr/share/appdata/libreoffice7.0-writer.appdata.xml
/usr/share/appdata/org.libreoffice7.0.kde.metainfo.xml
/usr/share/appdata/libreoffice7.0-draw.appdata.xml
/usr/share/appdata/libreoffice7.0-impress.appdata.xml
/usr/share/appdata/libreoffice7.0-base.appdata.xml
/usr/share/appdata/libreoffice7.0-calc.appdata.xml
/usr/share/applications/libreoffice7.0-xsltfilter.desktop
/usr/share/applications/libreoffice7.0-writer.desktop
/usr/share/applications/libreoffice7.0-base.desktop
/usr/share/applications/libreoffice7.0-math.desktop
/usr/share/applications/libreoffice7.0-startcenter.desktop
/usr/share/applications/libreoffice7.0-calc.desktop
/usr/share/applications/libreoffice7.0-draw.desktop
/usr/share/applications/libreoffice7.0-impress.desktop
/usr/share/icons/gnome/16x16/mimetypes/libreoffice7.0-oasis-formula.png
This is possible using sed ? or is more practical using another tool
With your list in the filename list, you could do:
sed -n 's/^[.]//;/\/.*[._].*$/p' list
Where:
sed -n suppresses printing of pattern-space; then
s/^[.]// is the substitution form that simply removes the first character '.' from each line; then
/\/.*[._].*$/p matches line that contain a '.' or '_' (optional) after the last '/' with p causing that line to be printed.
Example Use/Output
$ sed -n 's/^[.]//;/\/.*[._].*$/p' list
/usr/share/mime-info/libreoffice7.0.mime
/usr/share/mime-info/libreoffice7.0.keys
/usr/share/appdata/libreoffice7.0-writer.appdata.xml
/usr/share/appdata/org.libreoffice7.0.kde.metainfo.xml
/usr/share/appdata/libreoffice7.0-draw.appdata.xml
/usr/share/appdata/libreoffice7.0-impress.appdata.xml
/usr/share/appdata/libreoffice7.0-base.appdata.xml
/usr/share/appdata/libreoffice7.0-calc.appdata.xml
/usr/share/applications/libreoffice7.0-xsltfilter.desktop
/usr/share/applications/libreoffice7.0-writer.desktop
/usr/share/applications/libreoffice7.0-base.desktop
/usr/share/applications/libreoffice7.0-math.desktop
/usr/share/applications/libreoffice7.0-startcenter.desktop
/usr/share/applications/libreoffice7.0-calc.desktop
/usr/share/applications/libreoffice7.0-draw.desktop
/usr/share/applications/libreoffice7.0-impress.desktop
/usr/share/icons/gnome/16x16/mimetypes/libreoffice7.0-oasis-formula.png
Note, without GNU sed that allows chaining of expressions with ';' you would need:
sed -n -e 's/^[.]//' -e '/\/.*[._].*$/p' list
Assuming you want to delete the line(s) which is included other
pathname(s), would you please try:
sort -r list.txt | awk ' # sort the list in the reverse order
{
sub("^\\.", "") # remove leading dot
s = prev; sub("/[^/]+$", "", s) # remove the rightmost slash and following characters
if (s != $0) print # if s != $0, it means $0 is not a substring of the previous line
prev = $0 # keep $0 for the next line
}'
Result:
/usr/share/mime-info/libreoffice7.0.mime
/usr/share/mime-info/libreoffice7.0.keys
/usr/share/icons/gnome/16x16/mimetypes/libreoffice7.0-oasis-formula.png
/usr/share/applications/libreoffice7.0-xsltfilter.desktop
/usr/share/applications/libreoffice7.0-writer.desktop
/usr/share/applications/libreoffice7.0-startcenter.desktop
/usr/share/applications/libreoffice7.0-math.desktop
/usr/share/applications/libreoffice7.0-impress.desktop
/usr/share/applications/libreoffice7.0-draw.desktop
/usr/share/applications/libreoffice7.0-calc.desktop
/usr/share/applications/libreoffice7.0-base.desktop
/usr/share/appdata/org.libreoffice7.0.kde.metainfo.xml
/usr/share/appdata/libreoffice7.0-writer.appdata.xml
/usr/share/appdata/libreoffice7.0-impress.appdata.xml
/usr/share/appdata/libreoffice7.0-draw.appdata.xml
/usr/share/appdata/libreoffice7.0-calc.appdata.xml
/usr/share/appdata/libreoffice7.0-base.appdata.xml

Use `sed` to replace text in code block with output of command at the top of the code block

I have a markdown file that has snippets of code resembling the following example:
```
$ cat docs/code_sample.sh
#!/usr/bin/env bash
echo "Hello, world"
```
This means there there's a file at the location docs/code_sample.sh, whose contents is:
#!/usr/bin/env bash
echo "Hello, world"
I'd like to parse the markdown file with sed (awk or perl works too) and replace the bottom section of the code snippet with whatever the above bash command evaluates to, for example whatever cat docs/code_sample.sh evaluates to.
Perl to the rescue!
perl -0777 -pe 's/(?<=```\n)^(\$ (.*)\n\n)(?^s:.*?)(?=```)/"$1".qx($2)/meg' < input > output
-0777 slurps the whole file into memory
-p prints the input after processing
s/PATTERN/REPLACEMENT/ works similarly to a substitution in sed
/g replaces globally, i.e. as many times as it can
/m makes ^ match start of each line instead of start of the whole input string
/e evaluates the replacement as code
(?<=```\n) means "preceded by three backquotes and a newline"
(?^s:.*?) changes the behaviour of . to match newlines as well, so it matches (frugally because of the *?) the rest of the preformatted block
(?=```) means "followed by three backquotes`
qx runs the parameter in a shell and returns its output
A sed-only solution is easier if you have the GNU version with an e command.
That said, here's a quick, simplistic, and kinda clumsy version I knocked out that doesn't bother to check the values of previous or following lines - it just assumes your format is good, and bulls through without any looping or anything else. Still, for my example code, it worked.
I started by making an a, a b, and an x that is the markup file.
$: cat a
#! /bin/bash
echo "Hello, World!"
$: cat b
#! /bin/bash
echo "SCREW YOU!!!!"
$: cat x
```
$ cat a
foo
bar
" b a z ! "
```
```
$ cat b
foo
bar
" b a z ! "
```
Then I wrote s which is the sed script.
$: cat s
#! /bin/env bash
sed -En '
/^```$/,/^```$/ {
# for the lines starting with the $ prompt
/^[$] / {
# save the command to the hold space
x
# write the ``` header to the pattern space
s/.*/```/
# print the fabricated header
p
# swap the command back in
x
# the next line should be blank - add it to the current pattern space
N
# first print the line of code as-is with the (assumed) following blank line
p
# scrub the $ (prompt) off the command
s/^[$] //
# execute the command - store the output into the pattern space
e
# print the output
p
# put the markdown footer back
s/.*/```/
# and print that
p
}
# for the (to be discarded) existing lines of "content"
/^[^`$]/d
}
' $*
It does the job and might get you started.
$: s x
```
$ cat a
#! /bin/bash
echo "Hello, World!"
```
```
$ cat b
#! /bin/bash
echo "SCREW YOU!!!!"
```
Lots of caveats - better to actually check that the $ follows a line of backticks and is followed by a blank line, maybe make sure nothing bogus could be in the file to get executed... but this does what you asked, with (GNU) sed.
Good luck.
A rare case when use of getline would be appropriate:
$ cat tst.awk
state == "importing" {
while ( (getline line < $NF) > 0 ) {
print line
}
close($NF)
state = "imported"
}
$0 == "```" { state = (state ? "" : "importing") }
state != "imported" { print }
$ awk -f tst.awk file
See http://awk.freeshell.org/AllAboutGetline for getline uses and caveats.

Replacing a line in a file after grep command

I'm trying to replace a 0x1 to 0x0 in a file, I managed to get the line but I didn't manage to replace the string.
Here is the code:
grep -B 2 setSecure filePath | head -n 1
How can I proceed from here to change the string on the file? I'm using OS X, and I want to change the string only on this line.
Input File:
sometext
.line 344
const/4 v1, 0x1
iput v1, p0, Lnagra/nmp/sdk/NMPVideoView;->mCurrentState:I
const/4 v1, 0x1
invoke-virtual {v0, v1}, Landroid/view/SurfaceView;->setSecure(Z)V
.line 344
const/4 v1, 0x1
iput v1, p0, Lnagra/nmp/sdk/NMPVideoView;->mCurrentState:I
sometext
Output:
const/4 v1, 0x1
Please note there is many lines that start with "const/4 v1, 0x1" in the file.
I need to change only the "0x1" above the "invoke-virtual {v0, v1}, Landroid/view/SurfaceView;->setSecure(Z)V"
Could you please try following and let me know if this helps you.
awk '/const\/4 v1, 0x1/{prev=$0;$NF="0x0";val=$0;getline;print;getline;if($0 ~ /invoke-virtual {v0, v1}/){print val ORS $0} else {print prev};prev=val="";next} 1' Input_file
In case you need to save output into Input_file itself then append > temp_file && mv temp_file Input_file to above code too.
This is going to look a little ugly but is actually not terribly complicated. Using sed:
sed '1h; 1!H; $!d; x; s#\(const/4 v1, \)0x1\([^\n]*\n[^\n]*\n[^\n]*setSecure[^\n]*\n\)#\10x0\2#' filename
(pass -i to edit in-place after testing). This works by first reading the whole file into the hold buffer, then matching several lines at once:
1h # If we're processing the first line, write it to the hold buffer
1!H # if not, append it to the hold buffer
$!d # If we're not at the end of the file, we're done with this line here.
x # When we finally get here, we're at the end of the file, and the whole
# file is in the hold buffer. Swap hold buffer and pattern space, so the
# whole file is in the pattern space.
# Then: apply transformation. The regex is somewhat ugly because of all the \ns,
# but all that's really happening here is that we match const/4 v1, 0x1 followed
# by two lines of which the second contains "setSecure", and then replace the
# 0x1 with 0x0.
#
# To understand it, consider that \n[^\n]*\n matches newline followed by non-
# newline characters followed by another newline, which is exactly one line.
# Similarly, \n[^\n]*setSecure[^\n]*\n matches a line
s#\(const/4 v1, \)0x1\([^\n]*\n[^\n]*\n[^\n]*setSecure[^\n]*\n\)#\10x0\2#
Since you're using MacOS: MacOS by default uses BSD sed, which is limited in a number of ways. I think back in the day it had problems with \n in its code, so you may have to swap literal newlines in there for them. Although to be frank, if you're going to use sed under MacOS, it'd be least painful to just install GNU sed from homebrew.
You can't use grep for this task, grep only print and not modify the file.
You can use awk.
awk -v seen='setSecure' -v search='0x1' -v rplt='0x0' -v lign=1 '
NR<3 {
a[NR]=$0
next }
$0 ~ seen {
sub( search , rplt , a[lign] ) }
{
a[NR]=$0
print a[lign]
delete a[lign++] }
END {
for( i=lign ; i<=NR ; i++ )
print a[i] }
' input
With sed
sed '1N;N;/setSecure[^\n]*$/bA;P;D;:A;s/0x1/0x0/;P;D' input
sed '
1
N
# Always keep two lines in the pattern space
N
# get a third one
# and search for your word
# if find jump to A
/setSecure[^\n]*$/bA
# not find
# print the first line of the pattern space
P
# delete it and start a new cycle
D
:A
# the word is find
# do the replacement
s/0x1/0x0/
# print the first line of the pattern space
P
# delete it and start a new cycle
D
' input
If it's ok, you can add -i "" to replace the file
Using your already working command to find out the line number, the following sed command will replace all occurrences of 0x1 to 0x0, on lines starting with 'const/4', if it appears at least 2 lines before a line containing 'setSecure'
$ for LN in $(grep -n -B 2 setSecure filePath | grep 'const/4' | cut -d '-' -f 1 ) ; do sed -i ${LN}s/0x1/0x0/g filePath ; done
On Mac, following works :
$ for LN in $(grep -n -B 2 setSecure filePath | grep 'const/4' | cut -d '-' -f 1 ) ; do sed -i.bu ${LN}s/0x1/0x0/g filePath ; done

grep for Error and print all the lines containing 2 strings above and below Error

saaa vcahJJ HKak vk
Import xxx xXXXXX xxxx
aaaa aaaa aaaa ffffff
hhhhhh hhhhhh hhh hhh hhhhhh
Error reading readStatus api
aaa hhhh aaa aaaaa
gggggggg ggggg xxxxxxxxxx
uuuu hhhhhhhh fffffffff
query run ends
qidIdih II v iQE Iqe
I want to find the 'Error' string in the file containing above logs and then print all the info available between 2 strings 'Import' and 'ends'.
How can I do this using grep/sed
Tried this
but didn't get much.
Note: I dont know how many lines will be before and after. It may vary from above sample I have provided
How about:
$ awk 'BEGIN{RS=ORS="ends\n"} /Error/' file
RS is the input record separator which needs to be ends. ORSgets the same value for output purposes. Also, your example had /^Error/ but Error does not start the record (^).
Grep's -A 1 option will give you one line after; -B 1 will give you one line before; and -C 1 combines both to give you one line both before and after.
grep -C 1 "Error" <logfile>
As per your requirement, You can use-
sed -n '/Import/,/ends/p' filename
Here you are:
out=$( sed -n '/^Import/,/end$/p' file )
echo "$out" | grep Error >/dev/null && echo "$out"
This will capture the text between "Import" and "end" and print it only if the extracted text contains "Error".
You can try this sed
sed '/^Import/!d;:A;N;/\nImport/{h;s/.*\n//;bA};/ends$/!bA;h;s/\nError//;tB;d;:B;x' infile
Explanation :
sed '
/^Import/!d # if a line start with Import
:A
N # get an other line
/\nImport/{h;s/.*\n//;bA} # if the last line start with Import
# keep only this last line and return to A
/ends$/!bA # If the last line end with ends
h # keep all the lines in the hold space
s/\nError// # look for a line which start with Error
tB # if find jump to B
d # not find, delete all and return to the start
:B
x # restore all the lines and print
' infile

remove block of text between two lines based on content

I need to remove/filter a very large log file
i managed to bring the log-file into blocks of text starting with a line containing <-- or --> ending with a line containing Content-Length:
now if this block of text contains the word REGISTER it need to be deleted.
i found the flowing example:
# sed script to delete a block if /regex/ matches inside it
:t
/start/,/end/ { # For each line between these block markers..
/end/!{ # If we are not at the /end/ marker
$!{ # nor the last line of the file,
N; # add the Next line to the pattern space
bt
} # and branch (loop back) to the :t label.
} # This line matches the /end/ marker.
/regex/d; # If /regex/ matches, delete the block.
} # Otherwise, the block will be printed.
#---end of script---
written by Russell Davies on this page
but i do not know how to transport this to a single line statement to use in a pipe
my goal is to pipe a tail -F of the log file to the final version so it get updates by the minute
Try this:
awk '/<--|-->/{rec=""; f=1} f{rec = rec $0 ORS} /Content-Length:/{ if (f && (rec !~ "REGISTER")) printf "%s",rec; f=0}' file
If it doesn't do what you want, provide more info on what you want along with sample input and output.
To break down the above, here's each statement on separate lines with some comments:
awk '
/<--|-->/ {rec=""; f=1} # find the start of the record, reset the string to hold it and set a flag to indicate we've started processing a record
f {rec = rec $0 ORS} # append to the end of the string containing the current record
/Content-Length:/{ # find the end of the record
if (f && (rec !~ "REGISTER")) # print the record if it doesn't contain "REGISTER"
printf "%s",rec
f=0 # clear the "found record" indicator
}
' file
and if you have text between your records that you'd want printed, just add a test for the "found" flag not being set and invoke the default action of printing the current record (!f;)
awk '/<--|-->/{rec=""; f=1} f{rec = rec $0 ORS} !f; /Content-Length:/{ if (f && (rec !~ "REGISTER")) printf "%s",rec; f=0}' file
This might work for you (GNU sed);
sed '/<--\|-->/!b;:a;/Content-Length/!{$!{N;ba}};//{/REGISTER/d}' file
/<--\|-->/!b if a line does not contain <-- or --> print it
:a;/Content-Length/!{$!{N;ba}} keep appending lines until the string Content-Length or the end of file is encountered.
//{/REGISTER/d} if the line(s) read in contains Content-Length and REGISTER delete it/them else print it/them as normal.
If I get what you need correctly, you want to filter out the block, that is this only print the block:
tail -f logfile | sed -n '/\(<--\|-->\)/,/Content-Length:/ p'
If you want to delete it:
tail -f logfile | sed '/\(<--\|-->\)/,/Content-Length:/ d'

Resources