Find lines with no matching numbers as their preceding line - bash

Question

I am trying to find the lines in a file which non of the numbers in those lines are in their preceding line. This file has around 400000 lines. This is an example of the input file:
320 5120
240 326 5120
240 326 5120
241 333 514
240 326 5120
240 326 5120
320 5120
240
100 112
240 326 5120
240 326 5120
320 5120
The expected output results is:
241 333 514
240 326 5120
240
100 112
240 326 5120
So far I could find this command:
$ awk '!seen[$1]++' file
320 5120
240 326 5120
241 333 514
100 112
which I can get the unique number of column 1 and I can do the same separately for other columns. Can I somehow get the information I want from this command? Any help would be appreciated.

Answer 1

A Perl command-line program ("one"-liner), assuming things other than numbers in the file
perl -wnE'
#n = /([0-9]+)/g;
say "#n" if not grep { exists $seen_nums{$_} } #n;
%seen_nums = map { $_ => 1 } #n
' data.txt
This prints the desired output. It also prints the very first line (correctly). Since the program parses lines for numbers it can be used for files with headers, text-only (comment?) lines, etc.
But if the data is sure to have only numbers then we can use Perl's -a switch with which words on each line are available in the #F array. Also shrunk a little to actually fit on a line
perl -wlanE'grep exists $n{$_}, #F or say; %n = map { $_=>1 } #F' data.txt
A brief explanation of switches (see docs linked above)
-w turns on warnings
-l strips the newline, and can tack it back on, with few more subtleties
-a turns on "autosplit" (when used with -n or -p), so that #F is available in the program which contains words on the line. On newer Perls this sets -n as well
-n Critical for processing files or STDIN -- opens the resource and sets up a loop over lines. Run with -MO=Deparse to see what it does
-E The -e is what makes it evaluate everything between the following quotes as Perl code. With capital (E) it also turns on features, what I use mostly for say. (Doing this has drawbacks, since it enables all features and makes things not backwards compatible anymore.)
Note: The first line can be omitted by adding condition $.!=2 to the print

Answer 2

Here is an awk solution:
$ awk 'NR>1{p=1; for (i=1;i<=NF;i++){if($i in a)p=0}} {delete a; for (i=1;i<=NF;i++)a[$i]} p' file
241 333 514
240 326 5120
240
100 112
240 326 5120
How it works
NR>1{...}
Perform the commands in braces for all except the first line. Those commands are:
p=1
Initialize p to true (nonzero)
for (i=1;i<=NF;i++){if($i in a)p=0}
If any field is a key in array a, then set p to false (zero).
delete a
Delete array a.
for (i=1;i<=NF;i++)a[$i]
Create a key in array a for every field on the current line.
p
If p is true, print the line.
Multiple line version
Or, for those who prefer their code spread over multiple lines:
awk '
NR>1{
p=1
for (i=1;i<=NF;i++){
if($i in a)p=0}
}
{
delete a
for (i=1;i<=NF;i++)
a[$i]
}
p' file

Answer 3

Here's a perl one-liner:
$ perl -M-warnings -lane 'print unless #F ~~ %prev; %prev = map { $_ => 1 } #F;' input.txt
320 512
241 333 514
240 326 512
240
100 112
240 326 512
It uses the frowned-upon smart match operator in the name of conciseness. With smartmatch, ARRAY ~~ HASH returns true if any elements of the array are keys in the hash, which is perfect for this use case. If this was a standalone script and not a one-liner I'd probably use a different approach, though.
(Is there a reason the first line of your sample input isn't in your expected output even though it meets the critera?)

Answer 4

Here is a perl solution that does that. It tests for any of the numbers were seen on the previous line.
This includes printing the first line as noted by Shawn which might be needed. If not, just exclude the print join(... line in the code.
#!/usr/bin/perl
use strict;
use warnings;
use List::Util 'any';
open my $fh, '<', 'f0.txt' or die $!;
my #nums = split ' ', <$fh>;
my %seen = map{ $_ => 1} #nums;
print join(' ', #nums), "\n"; # print the first line
while (<$fh>) {
#nums = split;
print unless any {$seen{$_}} #nums;
%seen = map{ $_ => 1} #nums;
}
close $fh or die $!;
Output is:
320 512
241 333 514
240 326 512
240
100 112
240 326 512

Answer 5

A simple awk that checks, by means of a regex-match if the number is in the previous line. The idea is:
the previous line is stored in variable t
if any of the fields is matched to the previous line, we can skip to the next line.
This is done in the following way:
$ awk '{for(i=1;i<=NF;++i) if (FS t FS ~ FS $i FS) {t=$0; next}; t=$0}1'
320 512
241 333 514
240 326 512
240
100 112
240 326 512
The trick to make it work is to ensure that the line starts and stops with a field separator. If we would do the test t ~ $i we could match the number 25 against the number 255. But by ensuring that all numbers are sandwhiched between field separators, we can just do the test FS t FS ~ FS $i FS.
note: if you don't want the first line to be printed, replace the last 1 by (FNR>1)

Answer 6

Given your updated input:
$ awk '$0 !~ p; {gsub(/ /,"|"); p="(^| )("$0")( |$)"}' file
241 333 514
240 326 5120
240
100 112
240 326 5120
The above just converts the previous line read into a regexp like (^| )(320|5120)( |$) and then does a regexp comparison to see if the current line matches it and prints the current line if it doesn't match the modified previous line. This approach would only lead to false matches if your fields contained RE metacharacters which obviously yours don't since they're all-digits