Fast way to modify a field for rows having the same values in multiple columns (in bash)? - bash

Question

(it doesn't seem complicated, so I'm sorry if it's a duplicated question; I didn't manage to formulate it well enough for a good search!)
I have a file with five columns:
dog 5 red car 10
dog 9 blue car 10
cat 1 blue car 18
owl 5 red car 15
bla 8 blue train 100
bla 2 red train 100
...
I want to find rows matching the same values in columns 1, 4 and 5 (rows (1,2) and (5,6) in the example above), and modify their value in column 2 to "0", as in:
dog 0 red car 10
dog 0 blue car 10
cat 1 blue car 18
owl 5 red car 15
bla 0 blue train 100
bla 0 red train 100
...
I am doing that by looping through the unique values of column 1 and using grep + awk to find the matches and modify their values. However, I actually have a large file (> 10000 rows, 15 columns) and I would like to find a way of finding the matches without having to read the file multiple times in the loop. Any ideas for bash or a perl one-liner to speed that up?
Thanks a lot!

Answer 1

Using awk (And assuming columns are delimited by tabs):
$ awk 'BEGIN { FS=OFS="\t"}
NR == FNR { dups[$1,$4,$5]++; next }
dups[$1,$4,$5] > 1 { $2 = 0 }
1' input.txt input.txt
dog 0 red car 10
dog 0 blue car 10
cat 1 blue car 18
owl 5 red car 15
bla 0 blue train 100
bla 0 red train 100
This does read the file twice in order to find all lines with the same three fields, but I don't think there's a good way to avoid two passes through the data one way or another while preserving order. For example, a perl version that only reads the file once, but stores all the contents in memory to iterate over at the end, so still two passes:
perl -F"\t" -lane '$dups->{$F[0]}{$F[3]}{$F[4]}++;
push #lines, [#F];
END {
for my $l (#lines) {
$l->[1] = 0 if $dups->{$l->[0]}{$l->[3]}{$l->[4]} > 1;
print join("\t", #$l);
}
}' input.txt