Ubuntu 16.04
Bash 4.4.0
I am using grep to search for the word 'error' in a json file which is a logfile. How can an empty line be added after each instance?
my command: grep error "${wDir}"/"${client}"/logs/server.json >> "$eLog"
the output:
{"name":"XXX_XXX","hostname":"xxx.xx.xxx","pid":5193,"level":30,"fbresponse":{"error":{"message":"(#200) User does not ....."}}}
{"name":"XXX_XXX","hostname":"xxx.xx.xxx","pid":5193,"level":30,"fbresponseraw":{"error":{"message":"(#200) User does not ..."}}}
{"name":"XXX_XXX","hostname":"xxx.xx.xxx","pid":5193,"level":30,"fbresponse":{"error":{"message":"(#200) User does not ....."}}}
{"name":"XXX_XXX","hostname":"xxx.xx.xxx","pid":5193,"level":30,"fbresponseraw":{"error":{"message":"(#200) User does not ..."}}}
The desired output:
{"name":"XXX_XXX","hostname":"xxx.xx.xxx","pid":5193,"level":30,"fbresponse":{"error":{"message":"(#200) User does not ....."}}}
{"name":"XXX_XXX","hostname":"xxx.xx.xxx","pid":5193,"level":30,"fbresponseraw":{"error":{"message":"(#200) User does not ..."}}}
{"name":"XXX_XXX","hostname":"xxx.xx.xxx","pid":5193,"level":30,"fbresponse":{"error":{"message":"(#200) User does not ....."}}}
{"name":"XXX_XXX","hostname":"xxx.xx.xxx","pid":5193,"level":30,"fbresponseraw":{"error":{"message":"(#200) User does not ..."}}}
You may use awk for search and insert an empty line:
awk '/error/ { print $0 ORS }' "${wDir}"/"${client}"/logs/server.json
By default ORS (output record separator) is \n.
Simple is good.
sed '/error/G' "${wDir}"/"${client}"/logs/server.json >> "$eLog"
or if you want it to be case-insensitive
sed '/error/IG' "${wDir}"/"${client}"/logs/server.json >> "$eLog"
examples:
$: cat x
a
error
b
c
foo error other stuff
d
e
foo other stuff ERROR ERROR
f
g
$: sed '/error/G' x
a
error
b
c
foo error other stuff
d
e
foo other stuff ERROR ERROR
f
g
$: sed '/error/IG' x
a
error
b
c
foo error other stuff
d
e
foo other stuff ERROR ERROR
f
g
With grep:
grep "aa" a.txt | xargs printf "%s\n\n"
output:
aa
aa
aa
aa
Related
I'm using this to color the output of a script/command:
commandWithOutput | sed -r 's/(pattern)/'"${COLOR_RED}"'\1'"${COLOR_DEFAULT}"'/g'
(This will color all occurences of string "pattern" in the command's output.) And it works fine with traditional commands. However, if the script/command overwrites lines in its output (maybe this has more to do with a terminal/console than just standard output?), e.g.:
Building project X:
CXX Building file XYZ.cpp... [123/1034]
the behavior isn't as expected. My sed will still color the output but the overwriting doesn't work anymore, i.e.:
Building project X:
CXX Building file ABC.cpp... [1/1034]
CXX Building file DEF.cpp... [2/1034]
CXX Building file GHI.cpp... [3/1034]
CXX Building file JKL.cpp... [4/1034]
CXX Building file MNO.cpp... [5/1034]
// and so on...
CXX Building file XYZ.cpp... [123/1034]
Is there a way to color the output of a script/command that overwrites lines?
I've tried several different ideas ... IFS=$'\r' + OP's sed command ... trying to use an intermediate pipe (mkfifo) for processing the output from commandWithOutput ... a few attempts at trying to unbuffer stdout and/or stdin ... but (so far) could only get a awk solution to work, so fwiw ...
NOTE: I'm assuming OP's command is generating a \r when overwriting a line; if this is not the case the OP can try piping their command's output to | od -c to see what character is at the 'end of the line', with the idea being to use said character in place of my \r references (below).
First we'll write a small script to generate some data, (re)printing over the first few lines, and then printing some 'standalone' lines:
$ cat overwrite
#!/usr/bin/bash
for (( i=1 ; i<="${1}" ; i++ ))
do
printf "this is a test ... ${i}\r"
sleep 1
done
printf "\nanother test output \t and a tab\n"
echo "X."
Running the above generates the following output:
$ overwrite 3
this is a test ... 3 << this line is actually printed 3x times with suffixes of '1', '2' and '3'
another test output and a tab
X.
Running this through od shows the \r at the end of the first 3 lines:
$ overwrite 3 | od -c
0000000 t h i s i s a t e s t .
0000020 . . 1 \r t h i s i s a t
0000040 e s t . . . 2 \r t h i s i
0000060 s a t e s t . . . 3 \r \n
0000100 a n o t h e r t e s t o u t
0000120 p u t \t a n d a t a b \n
0000140 X . \n
0000143
We'll now look at one awk solution for recoloring a specific pattern in the output from our overwrite script ...
First we'll define the start and clear/reset variables for our desired color; for this exercise I'm going to use 'red':
$ myred=$(tput setaf 1) # set our highlight color to red
$ myreset=$(tput sgr0) # disable coloring
NOTE: There are a few ways to define these colors (and the disable/reset); I'll leave that up to the reader to pick what works best in their environment.
Here's one awk solution I found that works:
$ overwrite 3 | awk -v ptn="test" -v cstart="${myred}" -v creset="${myreset}" -v RS="[\n\r]" '{ sub(ptn,cstart ptn creset) ; printf $0 RT }'
Where:
-v ptn="test" - we want to recolor all instances of the string test; we'll pass this in as awk variable ptn
-v cstart="${myred}" - assign our highlight color code (red) to our awk variable cstart
-v creset="${myreset}" - assign our color clear/reset code to the awk variable creset
-v RS="[\n\r]" - redefine our input record separator as either \r or \n
sub(ptn,cstart ptn creset) - replace all instances of test with <red> + test + <reset>
printf $0 RT - print our new line; RT allows us to make use of the same RS that was used to parse out this record
Running the above generates:
this is a test ... 3 << this line is actually printed 3x times with suffixes of '1', '2' and '3', and the 'test' string printed in red
another test output and a tab << the 'test' string is printed in red
X.
I would like to redirect output to a file how described in this StackOverflow question but with an exception: let's suppose I do not want to store all the lines which start with the character r.
Specifically, I execute:
./command | tee /tmp/output.txt
and I get all the printed lines in the file. How to modify the instruction to allow exception?
This is a case for an output process substitution:
$ printf "%s\n" {o..u} | tee >(grep -v '^r' > outputfile)
o
p
q
r
s
t
u
$ cat outputfile
o
p
q
s
t
u
After a lot of trying the past day, I can't make following command work on 1 line:
sed '/'"$var1"'/ {n;n;a '\'"$var2"\'' \\
}' tempproject.cfg
when i run this like above, it matches $var1 and replaces the 3rd line after it with $var2.
example of what the sed command should do:
var1=c
var2=hello
a a
b b
c c
d => sed '/'"$var1"'/ {n;n;a '\'"$var2"\'' \\ => d
e }' tempproject.cfg e
f 'hello' \
g g
h h
when i put the command on 1 line i get the following error:
sed: -e expression #1, char 0: unmatched `{'
Thanks in advance!
$var1=c
$var2=hello
$sed "/$var1/{n;n;n;s/.*/'$var2' \\\ /}" tempproject.cfg
should give you
a
b
c
d
e
'hello' \
g
h
i
Why use three backslash?
I have a command which outputs in this format:
A
B
C
D
E
F
G
I
J
etc
I want the output to be in this format
A B C D E F G I J
I tried using ./script | tr "\n" " " but all it does is remove n from the output
How do I get all the output in one line. (Line wrapped)
Edit: I accidentally put in grep while asking the question. I removed
it. My original question still stands.
The grep is superfluous.
This should work:
./script | tr '\n' ' '
It did for me with a command al that lists its arguments one per line:
$ al A B C D E F G H I J
A
B
C
D
E
F
G
H
I
J
$ al A B C D E F G H I J | tr '\n' ' '
A B C D E F G H I J $
As Jonathan Leffler points out, you don't want the grep. The command you're using:
./script | grep tr "\n" " "
doesn't even invoke the tr command; it should search for the pattern "tr" in files named "\n" and " ". Since that's not the output you reported, I suspect you've mistyped the command you're using.
You can do this:
./script | tr '\n' ' '
but (a) it joins all its input into a single line, and (b) it doesn't append a newline to the end of the line. Typically that means your shell prompt will be printed at the end of the line of output.
If you want everything on one line, you can do this:
./script | tr '\n' ' ' ; echo ''
Or, if you want the output wrapped to a reasonable width:
./script | fmt
The fmt command has a number of options to control things like the maximum line length; read its documentation (man fmt or info fmt) for details.
No need to use other programs, why not use Bash to do the job? (-- added in edit)
line=$(./script.sh)
set -- $line
echo "$*"
The set sets command-line options, and one of the (by default) seperators is a "\n". EDIT: This will overwrite any existing command-line arguments, but good coding practice would suggest that you reassigned these to named variables early in the script.
When we use "$*" (note the quotes) it joins them alll together again using the first character of IFS as the glue. By default that is a space.
tr is an unnecessary child process.
By the way, there is a command called script, so be careful of using that name.
If I'm not mistaken, the echo command will automatically remove the newline chars when its argument is given unquoted:
tmp=$(./script.sh)
echo $tmp
results in
A B C D E F G H I J
whereas
tmp=$(./script.sh)
echo "$tmp"
results in
A
B
C
D
E
F
G
H
I
J
If needed, you can re-assign the output of the echo command to another variable:
tmp=$(./script.sh)
tmp2=$(echo $tmp)
The $tmp2 variable will then contain no newlines.
I try to filter the system log by running the sed command
like:
error
info_a1
info_a2
info_a3
crit
info_b
error
info_c1
info_c2
warn
info_d
error
info_x
error
info_p
info
info_w
error
info_z1
info_z2
I expect to get all the error messages likeļ¼
error
info_a1
info_a2
info_a3
error
info_c1
info_c2
error
info_x
error
info_p
error
info_z1
info_z2
i use
sed -n "/error/, /[info|warn|crit|]/p"
but it does not work pretty well.
awk '
/^error/ {p=1}
/^(info|warn|crit)/ {p=0}
p
'
If your objective is to print the error message AND the next line after it, then use
cat file_name | grep 'error' -A 1 | grep -v '\-\-'
if you just want to get all lines with "error" in it,
grep "error" file
if you insist on sed
sed -n '/error/p' file
or even awk
awk '/error/' file
or Ruby
ruby -ne 'print if /error' file
Here is a sed version:
sed -n ':a;/^error/{h;:b;n;${/^[[:blank:]]/{H;x;/^error/{p;q}}};/^[^[:blank:]]/{x;p;x;ba};H;bb}' inputfile
Explanation:
:a # label a
/^error/{ # if pattern space (patt) begins with "error"
h # copy the line to hold space (hold)
:b # label b
n # read the next line
${ # if it's the last line of input
/^[[:blank:]]/{ # if it begins with a space, tab, etc.
H # append it to hold
x # exchange patt and hold
/^error/{ # if patt begins with "error"
p # print it
q # quit
}
}
}
/^[^[:blank:]]/{ # if patt starts with a non-blank (a new block of input has started
x # exchange patt and hold
p # print patt
x # exchange patt and hold
ba # branch to label a
}
H # append patt to hold (save a line of the current block)
bb # branch to b
}