Bash - More efficient way to process csv file than grep [closed] - bash

Question

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 3 years ago.
Improve this question
Updated
I have a file (file.txt) with a list of words:
apple
banana
cherry
orange
pineapples
I have a csv file (data.csv) that contains lots of data:
1,"tasty apples",3,5
23,"iphone app",5,12
1,"sour grapes",3,5
23,"banana apple smoothie",5,12
1,"cherries and orange shortage",3,5
23,"apple iphone orange cover",5,12
3,"pineapple cherry bubble gum",13,5
5,"pineapples are best frozen",22,33
I want to append the match from file like this (output.csv):
1,"tasty apples",3,5,""
23,"iphone app",5,12,""
1,"sour grapes",3,5,""
23,"banana apple smoothie",5,12,"apple+banana"
1,"cherries and orange shortage",3,5,"orange"
23,"apple iphone orange cover",5,12,"apple+orange"
3,"pineapple cherry bubble gum",13,5,"cherry"
5,"pineapples are best frozen",22,33,"pineapples"
I can this with grep, but in order to do this, I have to use a while loop with if statements and process text files.
The problem with doing this is that file.txt has about 500 lines, and data.csv has 330,000 lines. My script would work, however it may take days to complete.
I'm wondering is there a more efficient way to do this than my method?

Answer 1

Perl to the rescue!
#!/usr/bin/perl
use warnings;
use strict;
use Text::CSV_XS qw{ csv };
open my $f1, '<', 'file.txt' or die $!;
my #fruits;
chomp, push #fruits, $_ while <$f1>;
my %order;
#order{#fruits} = 0 .. $#fruits;
my $regex = join '|', sort { length $b <=> length $a } #fruits;
csv(
in => 'data.csv1',
eol => "\n",
on_in => sub {
my #matches;
push #matches, $1 while $_[1][1] =~ /\b($regex)\b/g;
push #{ $_[1] }, join '+',
sort { $order{$a} <=> $order{$b} }
#matches;
},
);
Unfortunately, Text::CSV_XS can't quote the last field if it doesn't contain a special character (or without quoting all the fields). If file.txt doesn't contain double quotes and commas, though, you add them easily:
perl ... | sed 's/,\([^,"]*\)$/,"\1"/'

Answer 2

Is there a reason you want that last field quoted? The "+" has no special meaning in CSV, so does not need quotation and neither does an empty field.
Text::CSV_XS does support quotation of empty fields or quotation of all fields, but not yet quotation of all non-numeric fields.
Based on choroba's answer, which allows the last field to be "apple+apple+orange", which is not clearly defined in the OP if that is wanted, I'd write it like this:
use 5.14.1;
use warnings;
use Text::CSV_XS qw( csv );
use Data::Peek;
chomp (my #fruits = do { local #ARGV = "file.txt"; <> });
my %order;
#order{#fruits} = 0 .. $#fruits;
my $regex = join "|", sort { length $b <=> length $a } #fruits;
csv (
in => "data1.csv",
eol => "\n",
quote_empty => 1,
on_in => sub {
push #{$_[1]}, join "+" =>
sort { $order{$a} <=> $order{$b} }
keys %{{map { $_ => 1 }
($_[1][1] =~ m/\b($regex)\b/g)}};
},
);