i wanted to print lines between two matches using sed or awk in tcl.
since sed is fastest way to do the job, i am tring to use sed in tcl. temp is the input file.
temp file:
Hello
1
2
3
work
4 5
6
7
sed -n '/Hello/,/work /p' temp
awk 's/Hello/,/work /' temp
this is working in shell, now i want to use this in tcl file,
tclsh exec sed -n {/Hello/,/work/p} temp
tclsh exec awk {/Hello/,/work/} temp
it is giving error as:
Missing }.
excepted output:
Hello
1
2
3
work
what am i missing here ?
Just do it in pure tcl instead of trying to run an external program:
#!/usr/bin/env tclsh
# Print text in between and including lines matching the regular
# expressions `begin` and `end`
proc print_between {filename begin end} {
set inf [open $filename]
set in_block false
while {[gets $inf line] >= 0} {
if {[regexp -- $begin $line]} {
set in_block true
}
if {$in_block} {
puts $line
if {[regexp -- $end $line]} {
set in_block false
}
}
}
close $inf
}
print_between temp Hello work
Related
I have a markdown file that has snippets of code resembling the following example:
```
$ cat docs/code_sample.sh
#!/usr/bin/env bash
echo "Hello, world"
```
This means there there's a file at the location docs/code_sample.sh, whose contents is:
#!/usr/bin/env bash
echo "Hello, world"
I'd like to parse the markdown file with sed (awk or perl works too) and replace the bottom section of the code snippet with whatever the above bash command evaluates to, for example whatever cat docs/code_sample.sh evaluates to.
Perl to the rescue!
perl -0777 -pe 's/(?<=```\n)^(\$ (.*)\n\n)(?^s:.*?)(?=```)/"$1".qx($2)/meg' < input > output
-0777 slurps the whole file into memory
-p prints the input after processing
s/PATTERN/REPLACEMENT/ works similarly to a substitution in sed
/g replaces globally, i.e. as many times as it can
/m makes ^ match start of each line instead of start of the whole input string
/e evaluates the replacement as code
(?<=```\n) means "preceded by three backquotes and a newline"
(?^s:.*?) changes the behaviour of . to match newlines as well, so it matches (frugally because of the *?) the rest of the preformatted block
(?=```) means "followed by three backquotes`
qx runs the parameter in a shell and returns its output
A sed-only solution is easier if you have the GNU version with an e command.
That said, here's a quick, simplistic, and kinda clumsy version I knocked out that doesn't bother to check the values of previous or following lines - it just assumes your format is good, and bulls through without any looping or anything else. Still, for my example code, it worked.
I started by making an a, a b, and an x that is the markup file.
$: cat a
#! /bin/bash
echo "Hello, World!"
$: cat b
#! /bin/bash
echo "SCREW YOU!!!!"
$: cat x
```
$ cat a
foo
bar
" b a z ! "
```
```
$ cat b
foo
bar
" b a z ! "
```
Then I wrote s which is the sed script.
$: cat s
#! /bin/env bash
sed -En '
/^```$/,/^```$/ {
# for the lines starting with the $ prompt
/^[$] / {
# save the command to the hold space
x
# write the ``` header to the pattern space
s/.*/```/
# print the fabricated header
p
# swap the command back in
x
# the next line should be blank - add it to the current pattern space
N
# first print the line of code as-is with the (assumed) following blank line
p
# scrub the $ (prompt) off the command
s/^[$] //
# execute the command - store the output into the pattern space
e
# print the output
p
# put the markdown footer back
s/.*/```/
# and print that
p
}
# for the (to be discarded) existing lines of "content"
/^[^`$]/d
}
' $*
It does the job and might get you started.
$: s x
```
$ cat a
#! /bin/bash
echo "Hello, World!"
```
```
$ cat b
#! /bin/bash
echo "SCREW YOU!!!!"
```
Lots of caveats - better to actually check that the $ follows a line of backticks and is followed by a blank line, maybe make sure nothing bogus could be in the file to get executed... but this does what you asked, with (GNU) sed.
Good luck.
A rare case when use of getline would be appropriate:
$ cat tst.awk
state == "importing" {
while ( (getline line < $NF) > 0 ) {
print line
}
close($NF)
state = "imported"
}
$0 == "```" { state = (state ? "" : "importing") }
state != "imported" { print }
$ awk -f tst.awk file
See http://awk.freeshell.org/AllAboutGetline for getline uses and caveats.
I'd like to use Ruby's $INPUT_RECORD_SEPARATOR aka $/ to operate on a tab-separated file.
The input file looks like this (grossly simplified):
a b c
(the values are separated by tabs).
I want to get the following output:
a---
b---
c---
I can easily achieve this by using ruby -e and setting the $INPUT_RECORD_SEPARATOR alias $/:
cat bla.txt | ruby -e '$/ = "\t"; ARGF.each {|line| puts line.chop + "---" }'
This works, but what I'd really like is this:
cat bla.txt | ruby -n -e '$/ = "\t"; puts $_.chop + "---" '
However, this prints:
a b c---
Apparently, it doesn't use the provided separator - presumably because it has already read the first line before the separator was set. I tried to provide it as an environment variable:
cat bla.txt | $/="\n" ruby -n -e 'puts $_.chop + "---" '
but this confuses the shell - it tries to interpret $/ as a command (I also tried escaping the $ with one, two, three or four backslashes, all to no avail).
So how can I combine $/ with ruby -n -e ?
Use the -0 option :
cat bla.txt | ruby -011 -n -e 'puts $_.chop + "---" '
a---
b---
c---
-0[ octal] Sets default record separator ($/) as an octal. Defaults to \0 if octal not specified.
tabs have an ascii code of 9, which in octal is 11. Hence the -011
Use a BEGIN block, which is processed before Ruby begins looping over the lines:
$ echo "foo\tbar\tbaz" | \
> ruby -n -e 'BEGIN { $/ = "\t" }; puts $_.chop + "---"'
foo---
bar---
baz---
Or, more readably:
#!/usr/bin/env ruby -n
BEGIN {
$/ = "\t"
}
puts $_.chop + "---"
Then:
$ chmod u+x script.rb
$ echo "foo\tbar\tbaz" | ./script.rb
foo---
bar---
baz---
If this is more than a one-off script (i.e. other people might use it), it may be worthwhile to make it configurable with an argument or an environment variable, e.g. $/ = ENV['IFS'] || "\t".
I have this simple plat file (file.txt)
a43
test1
abc
cvb
bnm
test2
test1
def
ijk
xyz
test2
kfo
I need all lines between test1 and test2 in two forms, the firte one create two new files like
newfile1.txt :
test1
abc
cvb
bnm
test2
newfile2.txt
test1
def
ijk
xyz
test2
and the second form create only one new file like :
newfile.txt
test1abccvbbnmtest2
test1defijkxyztest2
Do you have any propositions?
EDIT
For the second form. I used this
sed -n '/test1/,/test2/p' file.txt > newfile.txt
But it give me a result like
test1abccvbbnmtest2test1defijkxyztest2
I need a return line like :
test1abccvbbnmtest2
test1defijkxyztest2
You can use this awk:
awk -v fn="newfile.txt" '/test1/ {
f="newfile" ++n ".txt";
s=1
} s {
print > f;
printf "%s", $0 > fn
} /test2/ {
close(f);
print "" > fn;
s=0
} END {
close(fn)
}' file
Perl, like sed and other languages, has the ability to select ranges of lines from a file, so it's a good fit for what you're trying to do.
This solution ended up being a lot more complicated than I thought it would be. I see no good reason to use it over #anubhava's awk solution. But I wrote it, so here it is:
#!/usr/bin/perl
use 5.010;
use strict;
use warnings;
use constant {
RANGE_START => qr/\Atest1\z/,
RANGE_END => qr/\Atest2\z/,
SUMMARY_FILE => 'newfile.txt',
GROUP_FILE => 'newfile%d.txt'
};
my $n = 1; # starting number of group file
my #wg; # storage for "working group" of lines
# Open summary file to write to.
open(my $sfh, '>', SUMMARY_FILE) or die $!;
while (my $line = <>) {
chomp $line;
# If the line is within the range, add it to our working group.
push #wg, $line if $line =~ RANGE_START .. $line =~ RANGE_END;
if ($line =~ RANGE_END) {
# We are at the end of a group, so summarize it and write it out.
unless (#wg > 2) {
# Discard any partial or empty groups.
#wg = ();
next;
}
# Write a line to the summary file.
$sfh->say(join '', #wg);
# Write out all lines to the group file.
my $group_file = sprintf(GROUP_FILE, $n);
open(my $gfh, '>', $group_file) or die $!;
$gfh->say(join "\n", #wg);
close($gfh);
printf STDERR "WROTE %s with %d lines\n", $group_file, scalar #wg;
# Get ready for the next group.
$n++;
#wg = ();
}
}
close($sfh);
printf STDERR "WROTE %s with %d groups\n", SUMMARY_FILE, $n - 1;
To use it, write the above lines into a file named e.g. ranges.pl, and make it executable with chmod +x ranges.pl. Then:
$ ./ranges.pl plat.txt
WROTE newfile1.txt with 5 lines
WROTE newfile2.txt with 5 lines
WROTE newfile.txt with 2 groups
$ cat newfile1.txt
test1
abc
cvb
bnm
test2
$ cat newfile.txt
test1abccvbbnmtest2
test1defijkxyztest2
For the second for you can add a new line after "test2" adding \n
sed -n '/test1/,/test2/p' file.txt | sed -e 's/test2/test2\n/g' > newfile.txt
sed is not useful to create multiple files so for the first one you should find another solution.
My code is:
perl -e'
use strict; use warnings;
my $a={};
eval{ test(); };
sub test{
print "11\n";
if( $a->{aa} eq "aa"){
print "aa\n";
}
else{
print "bb\n";
}
}'
Output on Terminal is:
11
Use of uninitialized value in string eq at -e line 9.
bb
If I redirect in file, the output order differ. Why?
perl -e'
...
' > t.log 2>&1
cat t.log:
Use of uninitialized value in string eq at -e line 9.
11
bb
My perl Version:
This is perl 5, version 18, subversion 4 (v5.18.4) built for x86_64-linux-thread-multi
(with 20 registered patches, see perl -V for more detail)
A simpler demonstration of the problem:
$ perl -e'print("abc\n"); warn("def\n");'
abc
def
$ perl -e'print("abc\n"); warn("def\n");' 2>&1 | cat
def
abc
This is due to differences in how STDOUT and STDERR are buffered.
STDERR isn't buffered.
STDOUT flushes its buffer when a newline is encountered if STDOUT is connected to a terminal.
STDOUT flushes its buffer when it's full otherwise.
$| = 1; turns off buffering for STDOUT[1].
$ perl -e'$| = 1; print("abc\n"); warn("def\n");' 2>&1 | cat
abc
def
Actually, the currently selected handle, which is the one print writes to if no handle is specified, which is STDOUT by default.
It's only an autoflush problem NO eval question.
Solution is:
perl -e'
use strict;
use warnings;
$|++; # <== this autoflush print output
my $a={};
test();
sub test{
print "11\n";
if( $a->{aa} eq "aa"){
print "aa\n";
}
else{
print "bb\n";
}
}' > t.log 2>&1
In some cases on terminal is the same problem:
perl -e'print("abc"); print(STDERR "def\n"); print("ghi\n");'
The only save way to get correct order, is turn on autoflush!
#dgw + ikegami ==> thank's
This question already has answers here:
Compare 2 files and remove any lines in file2 when they match values found in file1
(4 answers)
Closed 8 years ago.
I have two files not sortered which have some lines in common.
file1.txt
Z
B
A
H
L
file2.txt
S
L
W
Q
A
The way I'm using to remove common lines is the following:
sort -u file1.txt > file1_sorted.txt
sort -u file2.txt > file2_sorted.txt
comm -23 file1_sorted.txt file2_sorted.txt > file_final.txt
Output:
B
H
Z
The problem is that I want to keep the order of file1.txt, I mean:
Desired output:
Z
B
H
One solution I tought is doing a loop to read all the lines of file2.txt and:
sed -i '/^${line_file2}$/d' file1.txt
But if files are big the performance may suck.
Do you like my idea?
Do you have any alternative to do it?
You can use just grep (-v for invert, -f for file). Grep lines from input1 that do not match any line in input2:
grep -vf input2 input1
Gives:
Z
B
H
grep or awk:
awk 'NR==FNR{a[$0]=1;next}!a[$0]' file2 file1
I've written a little Perl script that I use for this kind of thing. It can do more than what you ask for but it can also do what you need:
#!/usr/bin/env perl -w
use strict;
use Getopt::Std;
my %opts;
getopts('hvfcmdk:', \%opts);
my $missing=$opts{m}||undef;
my $column=$opts{k}||undef;
my $common=$opts{c}||undef;
my $verbose=$opts{v}||undef;
my $fast=$opts{f}||undef;
my $dupes=$opts{d}||undef;
$missing=1 unless $common || $dupes;;
&usage() unless $ARGV[1];
&usage() if $opts{h};
my (%found,%k,%fields);
if ($column) {
die("The -k option only works in fast (-f) mode\n") unless $fast;
$column--; ## So I don't need to count from 0
}
open(my $F1,"$ARGV[0]")||die("Cannot open $ARGV[0]: $!\n");
while(<$F1>){
chomp;
if ($fast){
my #aa=split(/\s+/,$_);
$k{$aa[0]}++;
$found{$aa[0]}++;
}
else {
$k{$_}++;
$found{$_}++;
}
}
close($F1);
my $n=0;
open(F2,"$ARGV[1]")||die("Cannot open $ARGV[1]: $!\n");
my $size=0;
if($verbose){
while(<F2>){
$size++;
}
}
close(F2);
open(F2,"$ARGV[1]")||die("Cannot open $ARGV[1]: $!\n");
while(<F2>){
next if /^\s+$/;
$n++;
chomp;
print STDERR "." if $verbose && $n % 10==0;
print STDERR "[$n of $size lines]\n" if $verbose && $n % 800==0;
if($fast){
my #aa=split(/\s+/,$_);
$k{$aa[0]}++ if defined($k{$aa[0]});
$fields{$aa[0]}=\#aa if $column;
}
else{
my #keys=keys(%k);
foreach my $key(keys(%found)){
if (/\Q$key/){
$k{$key}++ ;
$found{$key}=undef unless $dupes;
}
}
}
}
close(F2);
print STDERR "[$n of $size lines]\n" if $verbose;
if ($column) {
$missing && do map{my #aa=#{$fields{$_}}; print "$aa[$column]\n" unless $k{$_}>1}keys(%k);
$common && do map{my #aa=#{$fields{$_}}; print "$aa[$column]\n" if $k{$_}>1}keys(%k);
$dupes && do map{my #aa=#{$fields{$_}}; print "$aa[$column]\n" if $k{$_}>2}keys(%k);
}
else {
$missing && do map{print "$_\n" unless $k{$_}>1}keys(%k);
$common && do map{print "$_\n" if $k{$_}>1}keys(%k);
$dupes && do map{print "$_\n" if $k{$_}>2}keys(%k);
}
sub usage{
print STDERR <<EndOfHelp;
USAGE: compare_lists.pl FILE1 FILE2
This script will compare FILE1 and FILE2, searching for the
contents of FILE1 in FILE2 (and NOT vice versa). FILE one must
be one search pattern per line, the search pattern need only be
contained within one of the lines of FILE2.
OPTIONS:
-c : Print patterns COMMON to both files
-f : Search only the first characters of each line of FILE2
for the search pattern given in FILE1
-d : Print duplicate entries
-m : Print patterns MISSING in FILE2 (default)
-h : Print this help and exit
EndOfHelp
exit(0);
}
In your case, you would run it as
list_compare.pl -cf file1.txt file2.txt
The -f option makes it compare only the first word (defined by whitespace) of file2 and greatly speeds things up. To compare the entire line, remove the -f.