a
b
s
start
text
more text
end
even more text
end
I want to print the content between start and first end that follows the start (start is always unique). I also want to print between which lines the text had been printed, in this example between lines 4 and 7.
I was trying with grep and cat, but I couldn't do much.
I tried:
var=$(cat $path)
echo "$var" | grep -o -P '(?<=start).*(?=end)'
But it didn't print anything, without the grep, it prints the whole file.
Output should in this example should be:
The content is between lines 4 and 7.
start
text
more text
end
With shell variables passed to awk and then print text by range then try, mention your shell variable inside start variable of awk and we should be Good then. (Also change $0 ~ start to $0 ~ "^"start"$" in case you want to look for exact match for start value in lines.)
awk -v start="$your_shell_start_var" '
$0 ~ start,$0 ~ /^end$/{
print
if($0 ~ start){ startLine=FNR }
if($0~/^end$/){
print "The content is between lines " startLine " and " FNR
exit
}
}' Input_file
Sample output on OP's samples:
start
text
more text
end
The content is between lines 4 and 7
Simple explanation: Printing lines by range start till end in between this statements checking condition if line has end string then come out of the Input_file, we need NOT to read the complete Input_file since OP needs to print only very first set of lines.
Sample data:
$ cat -n strings.dat
1 a
2 b
3 s
4 start
5 text
6 more text
7 end of more text
8 end
9 even more text
10 end
One awk solution using a range (similar to RavinderSingh13's post) that prints out OP's textual message at the end:
startstring="start" # define start of search block
awk -v ss="${startstring}" ' # pass start of search block in as awk variable "ss"
# search for a range of lines between "ss" and "end":
$0==ss,/^end$/ { if ($0==ss && x==0 ) x=FNR # if this is the first line of the range make note of the line number
print # print the current line of the range
if ($0=="end") # if this is the last line of the range then print our textual message re: start/finish line numbers
printf "\nThe content is between lines %d and %d.\n",x,FNR
}
' strings.dat
NOTE: the $0==ss and /^end$/ tests assume no leading/trailing white space in the data file otherwise these tests will fail and there will be no range match.
With startstring="start" this generates:
start
text
more text
end of more text
end
The content is between lines 4 and 8.
With startstring="more text" this generates:
more text
end of more text
end
The content is between lines 6 and 8.
With startstring="even more text" this generates:
even more text
end
The content is between lines 9 and 10.
With startstring="water" this generates:
--no output--
NOTE: If OP uses startstring="end" the results are not as expected; while it would be possible to add more code to address this scenario I'm going to skip this scenario for the time being.
Related
Suppose I have text as:
This is a sample text.
I have 2 sentences.
text is present there.
I need to replace whole text between two 'text' words. The required solution should be
This is a sample text.
I have new sentences.
text is present there.
I tried using the below command but its not working:
sed -i 's/text.*?text/text\
\nI have new sentence/g' file.txt
With your shown samples please try following. sed doesn't support lazy matching in regex. With awk's RS you could do the substitution with your shown samples only. You need to create variable val which has new value in it. Then in awk performing simple substitution operation will so the rest to get your expected output.
awk -v val="your_new_line_Value" -v RS="" '
{
sub(/text\.\n*[^\n]*\n*text/,"text.\n"val"\ntext")
}
1
' Input_file
Above code will print output on terminal, once you are Happy with results of above and want to save output into Input_file itself then try following code.
awk -v val="your_new_line_Value" -v RS="" '
{
sub(/text\.\n*[^\n]*\n*text/,"text.\n"val"\ntext")
}
1
' Input_file > temp && mv temp Input_file
You have already solved your problem using awk, but in case anyone else will be looking for a sed solution in the future, here's a sed script that does what you needed. Granted, the script is using some advanced sed features, but that's the fun part of it :)
replace.sed
#!/usr/bin/env sed -nEf
# This pattern determines the start marker for the range of lines where we
# want to perform the substitution. In our case the pattern is any line that
# ends with "text." — the `$` symbol meaning end-of-line.
/text\.$/ {
# [p]rint the start-marker line.
p
# Next, we'll read lines (using `n`) in a loop, so mark this point in
# the script as the beginning of the loop using a label called `loop`.
:loop
# Read the next line.
n
# If the last read line doesn't match the pattern for the end marker,
# just continue looping by [b]ranching to the `:loop` label.
/^text/! {
b loop
}
# If the last read line matches the end marker pattern, then just insert
# the text we want and print the last read line. The net effect is that
# all the previous read lines will be replaced by the inserted text.
/^text/ {
# Insert the replacement text
i\
I have a new sentence.
# [print] the end-marker line
p
}
# Exit the script, so that we don't hit the [p]rint command below.
b
}
# Print all other lines.
p
Usage
$ cat lines.txt
foo
This is a sample text.
I have many sentences.
I have many sentences.
I have many sentences.
I have many sentences.
text is present there.
bar
$
$ ./replace.sed lines.txt
foo
This is a sample text.
I have a new sentence.
text is present there.
bar
Substitue
sed -i 's/I have 2 sentences./I have new sentences./g'
sed -i 's/[A-Z]\s[a-z].*/I have new sentences./g'
Insert
sed -i -e '2iI have new sentences.' -e '2d'
I need to replace whole text between two 'text' words.
If I understand, first text. (with a dot) is at the end of first line and second text at the beginning of third line. With awk you can get the required solution adding values to var s:
awk -v s='\nI have new sentences.\n' '/text.?$/ {s=$0 s;next} /^text/ {s=s $0;print s;s=""}' file
This is a sample text.
I have new sentences.
text is present there.
This question already has answers here:
sed replace last line matching pattern
(14 answers)
Closed 6 years ago.
What is the easiest way to delete the last occurrence of a pattern in a file using a grep/awk/bash/etc? For example, I have a file in which the expression "hello world" appears multiple times and I would like to delete the entire line or just the last time "hello world" occurs. Thanks!
If you happen to have GNU coreutils (or you're willing to install them) you can use the occasionally useful tac command to flip the file for processing, allowing you to treat this problem as "remove the first occurrence of the pattern", which is somewhat simpler:
tac /path/to/file | awk '!found && /Hello, world/{found=1;next}1' | tac
Otherwise, you would need to do something like buffer all the lines of the file in memory so that you can print them all out at the end. Or you could process the file twice, the first time just looking for the line number to omit, but that requires that the data be in a file, rather than a stream you're piping into the command:
awk \
-v line="$(grep -hn "hello world" /path/to/file | tail -n1 | cut -f1 -d:)" \
"{NR != line}" /path/to/file
You can do:
awk '/^hello world/ {max=NR}
{a[NR]=$0}
END{for (i=1;i<=NR;i++) {if (i!=max) print a[i]}}' file
Or, if the file size is a concern, traverse it twice and use grep to count the matches. Skip the last match with awk:
awk -v last=$(grep -c '^hello world' file) '/^hello world/ && ++cnt==last{ next } 1 ' file
You can read file twice, and can achieve this without array like below no need of tac
Input
[akshay#gold tmp]$ cat f
1 hai
2 hello
3 this
4 is
5 test
6 hello
7 this
8 is
9 test
Output
[akshay#gold tmp]$ awk 'last==FNR{next}FNR!=NR{print;next}/hello/{last=FNR}' f f
1 hai
2 hello
3 this
4 is
5 test
7 this
8 is
9 test
Explanation
NR - It gives total number of records processed.
FNR - It gives total number of records for each input file.
awk '
# while reading file first time it always evaluate to Boolean false
# This part executes always but will skip line from
# file while reading second time
last==FNR{
next
}
# while reading file first time it always evaluate to Boolean false
# This part prints line when reading file second time
# immediately after print stop processing and go to next line
FNR!=NR{
print
next
}
# (1) executes first, read first file
# search for word hello and save line number in variable last
/hello/{
last=FNR
}
' f f # Here we are reading file twice
XYZNA0000778800Z
16123000012300321000000008000000000000000
16124000012300322000000007000000000000000
17234000012300323000000005000000000000000
17345000012300324000000004000000000000000
17456000012300325000000003000000000000000
9
XYZNA0000778900Z
16123000012300321000000008000000000000000
16124000012300322000000007000000000000000
17234000012300323000000005000000000000000
17345000012300324000000004000000000000000
17456000012300325000000003000000000000000
9
I have above file format from which I want to find a matching record. For example, match a number(7789) on line starting with XYZ and once matched look for a matching number (7345) in lines below starting with 1 until it reaches to line starting with 9. retrieve the entire line record. How can I accomplish this using shell script, awk, sed or any combination.
Expected Output:
XYZNA0000778900Z
17345000012300324000000004000000000000000
With sed one can do:
$ sed -n '/^XYZ.*7789/,/^9$/{/^1.*7345/p}' file
17345000012300324000000004000000000000000
Breakdown:
sed -n ' ' # -n disabled automatic printing
/^XYZ.*7789/, # Match line starting with XYZ, and
# containing 7789
/^1.*7345/p # Print line starting with 1 and
# containing 7345, which is coming
# after the previous match
/^9$/ { } # Match line that is 9
range { stuff } will execute stuff when it's inside range, in this case the range is starting at /^XYZ.*7789/ and ending with /^9$/.
.* will match anything but newlines zero or more times.
If you want to print the whole block matching the conditions, one can use:
$ sed -n '/^XYZ.*7789/{:s;N;/\n9$/!bs;/\n1.*7345/p}' file
XYZNA0000778900Z
16123000012300321000000008000000000000000
16124000012300322000000007000000000000000
17234000012300323000000005000000000000000
17345000012300324000000004000000000000000
17456000012300325000000003000000000000000
9
This works by reading lines between ^XYZ.*7779 and ^9$ into the pattern
space. And then printing the whole thing if ^1.*7345 can be matches:
sed -n ' ' # -n disables printing
/^XYZ.*7789/{ } # Match line starting
# with XYZ that also contains 7789
:s; # Define label s
N; # Append next line to pattern space
/\n9$/!bs; # Goto s unless \n9$ matches
/\n1.*7345/p # Print whole pattern space
# if \n1.*7345 matches
I'd use awk:
awk -v rid=7789 -v fid=7345 -v RS='\n9\n' -F '\n' 'index($1, rid) { for(i = 2; i < $NF; ++i) { if(index($i, fid)) { print $i; next } } }' filename
This works as follows:
-v RS='\n9\n' is the meat of the whole thing. Awk separates its input into records (by default lines). This sets the record separator to \n9\n, which means that records are separated by lines with a single 9 on them. These records are further separated into fields, and
-F '\n' tells awk that fields in a record are separated by newlines, so that each line in a record becomes a field.
-v rid=7789 -v fid=7345 sets two awk variables rid and fid (meant by me as record identifier and field identifier, respectively. The names are arbitrary.) to your search strings. You could encode these in the awk script directly, but this way makes it easier and safer to replace the values with those of a shell variables (which I expect you'll want to do).
Then the code:
index($1, rid) { # In records whose first field contains rid
for(i = 2; i < $NF; ++i) { # Walk through the fields from the second
if(index($i, fid)) { # When you find one that contains fid
print $i # Print it,
next # and continue with the next record.
} # Remove the "next" line if you want all matching
} # fields.
}
Note that multi-character record separators are not strictly required by POSIX awk, and I'm not certain if BSD awk accepts it. Both GNU awk and mawk do, though.
EDIT: Misread question the first time around.
an extendable awk script can be
$ awk '/^9$/{s=0} s&&/7345/; /^XYZ/&&/7789/{s=1} ' file
set flag s when line starts with XYZ and contains 7789; reset when line is just 9, and print when flag is set and contains pattern 7345.
This might work for you (GNU sed):
sed -n '/^XYZ/h;//!H;/^9/!b;x;/^XYZ[^\n]*7789/!b;/7345/p' file
Use the option -n for the grep-like nature of sed. Gather up records beginning with XYZ and ending in 9. Reject any records which do not have 7789 in the header. Print any remaining records that contain 7345.
If the 7345 will always follow the header,this could be shortened to:
sed -n '/^XYZ/h;//!H;/^9/!b;x;/^XYZ[^\n]*7789.*7345/p' file
If all records are well-formed (begin XYZ and end in 9) then use:
sed -n '/^XYZ/h;//!H;/^9/!b;x;/^[^\n]*7789.*7345/p' file
I have many text files containing annotations. The original text is marked with lines containing the words:
START OF TEXT OF PASSAGE 1
END OF TEXT OF PASSAGE 1
Obviously I can search each document for the phrase START OF TEXT and delete everything up to it. Then search for END OF TEXT and start selecting text for deletion until I get to the next START OF TEXT.
I have come up with this design so far:
#!/bin/bash
a="START OF PROJECT"
b="END OF PROJECT"
while read line; do
if line contains a; do
while read line; do
'if line does not contain b'
'append the line to output.txt'; fi
done
done
fi
done
Perhaps there is an easier way using sed, awk, grep and pipes?
'for every document' 'loop through it doing this' ('find the original text between START and END' | >> output.txt)
Unfortunately I am poor at bash and ignorant of sed/awk.
The reason for this is that I am assembling a huge text document that is a concatenation of thousands of marked up documents – each of which contains some annotated passages.
In Python:
import re
with open('in.txt') as f, open('out.txt', 'w') as output:
output.write('\n'.join(re.findall(r'START OF TEXT(.*?)END OF TEXT', f.read())))
This reads the input, searches for all matches that begin and end with the necessary markers, captures the text of interest in a group, joins all those groups on a linefeed, and writes that to the result file.
Pretty easy to do with awk. You would create a script (I'll call it yank.awk) containing this:
#!/usr/bin/awk
/START OF PROJECT/ { capture = 1; next }
/END OF PROJECT/ { capture = 0 }
capture == 1 { print }
and then run it like so:
yank.awk in.txt > output.txt
Could also do with sed and grep:
sed -ne '/START OF PROJECT/,/END OF PROJECT/p' in.txt | grep -vE '(START|END) OF PROJECT' > output.txt
(Another Python solution)
You can have itertools.groupby group lines together based on a boolean value - just use a global flag to keep track of whether you are in a block or not, and then use groupby to group the lines that are in or out of blocks. Then just discard the ones that are not blocks:
sample_lines = """
lskdjflsdkjf
sldkjfsdlkjf
START OF TEXT
Asdlkfjlsdkfj
Bsldkjf
Clsdkjf
END OF TEXT
sldkfjlsdkjf
sdlkjfdklsjf
sdlkfjdlskjf
START OF TEXT
Dsdlkfjlsdkfj
Esldkjf
Flsdkjf
END OF TEXT
sldkfjlsdkjf
sdlkjfdklsjf
sdlkfjdlskjf
""".splitlines()
from itertools import groupby
in_block = False
def is_in_block(line):
global in_block
if line.startswith("END OF TEXT"):
in_block = False
ret = in_block
if line.startswith("START OF TEXT"):
in_block = True
return ret
for lines_are_text,lines in groupby(sample_lines, key=is_in_block):
if lines_are_text:
print(list(lines))
gives:
['Asdlkfjlsdkfj', 'Bsldkjf', 'Clsdkjf']
['Dsdlkfjlsdkfj', 'Esldkjf', 'Flsdkjf']
See that first group has the lines that start with A, B, and C, and the second group is made up of those lines starting with D, E, and F.
It sounds like the specific solution you need is:
awk '/END OF TEXT OF PASSAGE/{f=0} f; /START OF TEXT OF PASSAGE/{f=1}' file
See https://stackoverflow.com/a/18409469/1745001 for other ways to select text from files.
Use Perl's Flip-Flop Operator to Print Text Between Markers
Given a corpus like:
START OF TEXT OF PASSAGE 1
foo
END OF TEXT OF PASSAGE 1
START OF TEXT OF PASSAGE 2
bar
END OF TEXT OF PASSAGE 2
you can use the Perl flip-flop operator to process within a range of lines. For example, from the shell prompt:
$ perl -ne 'if (/^START OF TEXT/ ... /^END OF TEXT/) {
next if /^(?:START|END)/;
print;
}' /tmp/corpus
foo
bar
Basically, this short Perl script loops through your input. When it finds your start and end tags, it throws away the tags themselves and prints everything else in between.
Usage Notes
The line breaks between passages in the corpus are for readability. It doesn't matter if your real corpus has no line breaks between passages, so long as the text markers always start at the beginning of the line as shown in your original post. If that assumption doesn't hold true, then you will need to adjust the regular expressions used to identify the start and end of your passages.
You can pass multiple files to the Perl script. Again, it makes no practical difference as long as you don't exceed the length limit of your shell.
If you want the final output to go to somewhere other than standard output, just use shell redirection. For example:
perl -ne 'if (/^START OF TEXT/ ... /^END OF TEXT/) {
next if /^(?:START|END)/;
print;
}' /tmp/file1 /tmp/file2 /tmp/file3 > /tmp/output
You can use sed as follows:
sed -n '/^START OF TEXT/,/^END OF TEXT/{/^\(START\|END\) OF TEXT/!p}' infile
or, with extended regular expressions (-r):
sed -rn '/^START OF TEXT/,/^END OF TEXT/{/^(START|END) OF TEXT/!p}' infile
-n prevents sed from printing as a default. The rest works as follows:
/^START OF TEXT/,/^END OF TEXT/ { # For lines between these two matches
/^\(START\|END\) OF TEXT/!p # If the line does NOT match, print it
}
This works with GNU sed and might require some tweaking to run with other seds.
A program creates HTML files from a database. There are headings and stuff in between the headings.
There are not a set amount of headings.
After each heading the program places the text:
$WHITE*("5")$
$WHITE*("20")$
$HRULE$
I need every occurrence of these 4 lines to be replaced with:
$WHITE*("20")$
$HRULE$
$WHITE*("10")$
I am not fussed what program is used :)
I have tried:
sed 's:\$WHITE\*(\"5\")\$\n\n\$WHITE\*(\"20\")\$\n\$HRULE\$:\$WHITE\*(\"20\")\$\
\$HRULE$\
\$WHITE*("10")$:g'
and various other permutations
If that'S your input file, and this is the spec, you can do:
sed -n '3,$p;$a$WHITE*("10")$' INPUTFILE
But I assume that's not the case, so you might want to rephrase your question and/or giving some more detailes.
More specific solution with sed:
sed '/^\$WHITE\*("5")\$$/,/^$/d;/\$HRULE\$/ a$WHITE*("10")$' INPUTFILE
(Searches for the $WHITE*("5")$ line and deletes it till (including!) the next empty line. Then searches for the next $HRULE$ line and appends an $WHITE*("10")$ line.
awk solution:
awk '/\$WHITE\*\("5"\)\$/ { getline ; next }
/\$WHITE\*\("20"\)\$/ { print ;
getline ;
if ($0 ~ /\$HRULE\$/) { print ;
print "$WHITE*(\"10\")$" ;
}
else { print }
}
1 ' INPUTFILE
This reads the file and prints every line - that's why the 1 is there, except if it finds the $WHITE*("5") pattern it drops it, reads the next line and drops that too. if it finds the $WHITE*("20") prints it. Reads the next line and if its $HRULE$ then prints that and the appended $WHITE*("10") line. Else just prints the line.
HTH
UPDATE #2
From the sed faq, section 4.23.3
If you need to match a static block of text (which may occur any number of times throughout a file), where the contents of the block are known in advance, then this script is easy to use
UPDATE #1
Python?
$ cat input
first line
second line
3rd line
$WHITE*("5")$
$WHITE*("20")$
$HRULE$
some more lines
yet another
$WHITE*("5")$
$WHITE*("20")$
$HRULE$
THE END
the script:
#!/usr/bin/env python
## Use these 3 lines for python version < 2.5
#fd=open('input')
#text=fd.read()
#fd.close()
## Use these 2 lines for python version >= 2.5
with open('input') as fd:
text=fd.read()
old="""$WHITE*("5")$
$WHITE*("20")$
$HRULE$
"""
new="""$WHITE*("20")$
$HRULE$
$WHITE*("10")$
"""
print text.replace(old,new)
output:
first line
second line
3rd line
$WHITE*("20")$
$HRULE$
$WHITE*("10")$
some more lines
yet another
$WHITE*("20")$
$HRULE$
$WHITE*("10")$
THE END
Try something like
sed -e '${p;};/$WHITE\*("5")\$/,/$HRULE\$/{H;/$HRULE\$/{g;s/$HRULE\$//;s/20/10/;s/5/20/;s/\n/&$HRULE$/2p;s/.*//p;x;d;};d;};' white.txt
Crude, but it should work.
This might work for you:
sed '/^\$WHITE\*(\"5\")\$/{N;N;N;s/.*\n\n\(\(\$WHITE\*(\"\)20\(\")\$\s*\)\n\$HRULE\$\s*$\)/\1\n\210\3/}' file
Explanation:
Match on first string $WHITE*("5")$, read the next 3 lines and match on remainder. Use grouping and back references to formulate output lines.