match columns in 2 tab-delimited text files - bash

Question

I have two tab-delimited .txt files
file1 has 20 million lines and the following structure
col1 col2 col3 col4 col5
1 x x A x
2 y y A x
3 z z A x
4 x x B x
5 x y B x
6 x y E x
7 x z F x
file2 has 3000 lines and the following structure
col1
A
B
C
D
Now I want to extract from file1 the lines where there is a match between col1 from file2 and col4 of file1
So the new file3 should look like this
col1 col2 col3 col4 col5
1 x x A x
2 y y A x
3 z z A x
4 x x B x
5 x y B x
How can I do this with perl or bash?

Answer 1

You can use standard awk command to join 2 files:
awk 'BEGIN{FS=OFS="\t"} FNR == NR { a[$1]; next } $4 in a' file2 file1

Answer 2

try this -
awk -F'[ ]+' 'NR==FNR {a[$1]++;next} $4 in a{print $0}' f2 f1
1 x x A x
2 y y A x
3 z z A x
4 x x B x
5 x y B x

Answer 3

Since you also asked about Perl, here's a reusable perl solution. You first read file 2, generate an array of lookup values, and then read file 1, printing out any line in which column 4 matches a value inside of the array we created above. Something like this might work:
#!/usr/bin/perl
use strict;
use warnings;
use autodie;
my $key_file = shift;
open(my $fh, "<", $key_file);
my $header = <$fh>; # read the header line into '$h'
my %keys = map{ chomp; $_ => 1 }<$fh>;
close $fh;
my $query_file = shift;
open(my $q_fh, "<", $query_file);
print scalar <$q_fh>;
while (<$q_fh>) {
my #fields = split;
print if $keys{$fields[3]};
}
close $q_fh;
You can run this as table_combine.pl <file2> <file1>.