I need to extract Voip log from a D-Link router, so I've setup a little python script that executes a command in this router via telnet.
My script does a "cat /var/log/calls.log" and returns the result, however...
it also sends non-important stuff, like the BusyBox banner, etc...
How can I ignore lines from 1 to 6 and the last 2 ?
This is my current output:
yaba#foobar:/stuff$ python calls.py
BusyBox v1.00 (2009.04.09-11:17+0000) Built-in shell (msh)
Enter 'help' for a list of built-in commands.
DVA-G3170i/PT # cat /var/call.log
1 ,1294620563,2 ,+351xxx080806 ,xxx530802 ,1 ,3 ,1
DVA-G3170i/PT # exit
And I just need:
1 ,1294620563,2 ,+351xxx080806 ,xxx530802 ,1 ,3 ,1
(it can have multiple lines)
So that I can save it to a CSV and later to a sql db.
Thanks, and sorry my bad english.
Why not use a pattern in AWK to match the text you want?
python calls.py | awk '/^[0-9]/{print}/'
The whole POINT of AWK is matching lines based on patterns and manipulating/printing those matched lines.
Edited to add example run.
Here's a junk data file based on your sample above.
$ cat junk.dat
BusyBox v1.00 (2009.04.09-11:17+0000) Built-in shell (msh)
Enter 'help' for a list of built-in commands.
DVA-G3170i/PT # cat /var/call.log
1 ,1294620563,2 ,+351xxx080806 ,xxx530802 ,1 ,3 ,1
DVA-G3170i/PT # exit
Here's running it through AWK with a filter.
$ cat junk.dat | awk '/^[0-9]/ {print}'
1 ,1294620563,2 ,+351xxx080806 ,xxx530802 ,1 ,3 ,1
No need for SED, no need for counting lines, no need for anything but AWK. Why make things more complicated than they need to be?
In one call to sed:
sed -n '1,6d;7,${N;$q;P;D}'
or for picky versions of sed:
sed -ne '1,6d' -e '7,${N' -e '$q' -e 'P' -e 'D}'
You could also do it based on matches:
sed -n '/^[0-9]+/p'
or something similar.
But why doesn't your Python script read the file and do the filtering (instead of calling an external utility)?
python calls.py | sed -e 1,6d -e '$d'
So that might work. It will filter out the first 6 and the last, which is what your example indicates you need. If you really want to clobber the last two lines then you could do:
python calls.py | sed -e 1,6d -e '$d' | sed -e '$d'
But wait ... you said awk, so...
python calls.py | awk '{ if(NR > 7) { print t }; t = $0 }'
This might work for you:
sed '1,6d;$!N;$d;P;D' file
I'm not sure this is the best way to do it (maybe D-Link router has FTP or SSH support) but you can do it with awk:
awk '/cat/, /exit/' | sed -e '1d' -e '$d'
awk will print everything between lines containing "cat" and "exit", unfortunately including these two lines. That's what the remaining commands are for, I couldn't figure out how to do it nicer than that...
Related
I'm trying to adapt the answer from https://stackoverflow.com/a/66365284/1236401 that adds control flow to provide match status code:
cat file.txt | sed 's/1/replaced-it/;tx;q1;:x'
It works as expected on Ubuntu and Alpine, but fails on Mac OSX (11.6), using any shell.
sed: 1: "s/1/replaced-it/;tx;q1;:x": undefined label 'x;q1;:x'
All references I could find to sed misbehaving on OSX were for in-place file edit, which is not the case here.
Commands in sed are separated primarily by newlines.
| sed 's/1/replaced-it/
tx
q1
:x
'
Alternatively:
sed -e 's/1/replaced-it/' -e 'tx' -e 'q1' -e ':x'
Additionally q1 is a GNU sed extension - it's not supported in every sed. It has to be removed, refactored, or you have to install GNU sed.
Overall, write it in awk, python or perl.
Here is an Awk refactoring.
awk '/1/ { sub("1", "replaced-it", $0); replaced=1 } 1
END { exit 1-replaced }' file.txt
Notice also how the cat is useless (with sed too, and generally any standard Unix command except annoyingly tr).
An AWK equivalent would be:
awk '!sub(/1/,"replaced-it") {print; exit 1} 1' file.txt
If a substitution is not made successfully: print and exit with a status of 1; otherwise print.
I am trying to use sed to capture a group like these examples:
123123 (i would want the first group 123)
144144 (I would want the group 144)
however sed does not seem to realize what \1 is.
Is there any way to do this using sed? I want to replace the first group with a specific string afterwards.
([0-9]+)\1
I have tried using the above regex yet, sed does not seem to realize what I am trying to do.
also tried this:
~/Desktop$ cat file
123123
23231
12323
123231
12345
144144
~/Desktop$ sed -n 's/.*\b\([[:digit:]]\{1,\}\)\1\b.*/\1/p' file
~/Desktop$
~/Desktop$ sed -n -E 's/([0-9]+)\1/specificstring\1/p' file
specificstring12323
specificstring2323
specificstring12323
specificstring14444
~/Desktop$ sed -nE 's/^([0-9]+)\1([^0-9]|$)/\1/p' file
2323
12323
Use a BRE, and avoid using + since it is not a part of POSIX REs.
$ cat file
123123
23231
12323
123231
12345
144144
$
$ sed -n 's/^\([0-9]\{1,\}\)\1$/\1/p' file
123
144
I want to replace the first group with a specific string afterwards.
With GNU sed :
sed -n -E 's/([0-9]+)\1/specificstring\1/p' file
Takeaways
-n suppresses the output which we override using the print (p ) flag of the s command.
-E enables extended regular expressions.
Note
This doesn't, however, print the lines where there no identical groups the existence of which is not mentioned in the question.
Given that the file only contains 6 digit numbers and nothing else it could be done like this:
sed -n 's/\([0-9]\{3\}\)\1/\1/p' file
I have an xml file with these lines (among others):
#Env=DEV2,DEV3,DEV5,DEV6
#Enter your required DEV environment after the ENV= in the next line:
Env=DEV6
I need to:
Verify that the text after ENV= is of the pattern DEV{1..99}
extract the number (in this case, 6) from the line ENV=DEV6 to some environment variable
I know a bit of awk and grep, and can use those to get the number, but I'm thinking of Sed, which I'm told matches patterns nicer than awk and takes less time. Also, I'm concerned about long long lines of greps matching the beginning of the line for that particular Env= .
How would I go about doing it with Sed? would I get away with a shorter line?
I'm a sed newbie, read a bunch of tutorials and examples and got my fingers twisted trying to do both things at the same time...
Can use grep also if pcre regex is available
$ cat ip.txt
#Env=DEV2,DEV3,DEV5,DEV6
#Enter your required DEV environment after the ENV= in the next line:
Env=DEV6
foo
Env=DEV65
bar
Env=DEV568
$ grep -xoP 'Env=DEV\K[1-9][0-9]?' ip.txt
6
65
-x match whole line
-o output only matching text
-P use pcre regex
Env=DEV\K match Env=DEV but not part of output
[1-9][0-9]? range of 1 to 99
I suggest with GNU sed:
var=$(sed -nE 's/^Env=DEV([0-9]{1,2})$/\1/p' file)
echo "$var"
Output:
6
awk -F'Env=DEV' '/Env=DEV[0-9]$|Env=DEV[0-9][0-9]$/{print $2}' input
Input:
echo '
Env=DEV6
Env=DEVasd
Env=DEV62
Env=DEV622'
Output:
awk -F'Env=DEV' '/Env=DEV[0-9]$|Env=DEV[0-9][0-9]$/{print $2}' input
6
62
To store it into any variable:
var=$(awk command)
In awk. First some test cases:
$ cat file
foo
Env=DEV0
Env=DEV1
Env=DEV99
Env=DEV100
$ awk 'sub(/^Env=DEV/,"") && /^[1-9][0-9]?$/' file
1
99
You can used sed as
$ sed 's/^Env=DEV\([1-9][0-9]\?\)/\1/' file
6
You can directly use the above command in export command as
export YOUR_EXPORT_VARIABLE=$(sed 's/^Env=DEV\([1-9][0-9]\?\)/\1/' file)
(or) its pretty straight forward with perl
$ perl -nle 'print $1 if /Env=DEV.*?(\d+)/' file
6
I have a file which consists of following hundred lines in addition to other data
(abc-wxyz1/2222 1234)
1234 is a random number which is different for all the 100 lines. I want to substitute all the lines with
(abc-wxyz1/2222 *)
I am using following code for it
cat input.txt | \
sed -i -e 's/\(abc\-wxyz1\/2222\ [0-9]\+\)/\(abc\-wxyz1\/2222\ \*\)/g' > output.txt
I get the output.txt blank I have no clue why. Am I doing it correctly ?
For the input sample you've shown, you could go with awk:
awk '{print $1 " *)"}' input.txt
If you're going to use the flag -i, --in place, you don't have to redirect the output. The substitutions are performed in place, as expected.
Yet, you can simplify your expression, doing:
sed -i -e 's/abc\-wxyz1\/2222\ \([0-9]\+\)/*/g' input.txt
Use this command to see what will be the result of filter:
cat input.txt | sed 's/\(abc\-wxyz1\/2222\ [0-9]\+\)/\(abc\-wxyz1\/2222\ \*\)/g'
Try using shorter patterns to toubleshoot where it is breaking.
You only need the '-e' flag if you have additional commands to run. You could also just do:
sed 'command;command2;{command3}'
I want to get the second last line from the ls -l output.
I know that
ls -l|tail -n 2| head -n 1
can do this, just wondering if sed can do this in just one command?
ls -l|sed -n 'x;$p'
It can't do third to last though, because sed only has 1 hold space, so can only remember one older line. And since it processes the lines one at a time, it does not know the line will be next to last when processing it. awk could return thrid to last, because you can have arbitrary number of variables there, but the script would be much longer than the tail -n X|head -n 1.
In a awk one-liner :
echo -e "aaa\nbbb\nccc\nddd" | awk '{v[c++]=$0}END{print v[c-2]}'
ccc
Try this to delete second-last line in file
sed -e '$!{h;d;}' -e x filename
tac filename | sed -n 2p
-- but involves a pipe, too