How to remove duplicate columns and merge their unique value [closed] - bash

Question

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
I'd like to convert my source file which has the following values:
col1|col2|col3
1|a|desc1
1|a|desc2
1|b|desc3
1|b|desc4
1|b|desc5
2|c|desc6
2|c|desc7
into:
col1|col2|col3
1|a|desc1 desc2
1|b|desc3 desc4 desc5
2|c|desc6 desc7
It's basically a column 1 and column 2 duplicate removal but their column 3 will be merged. Looking into awk or perl or sed or bash code - whichever has the minimal processing power consumption is preferred.

Answer 1

$ cat tst.awk
BEGIN { FS="|" }
{
curr = $1 FS $2
if (curr == prev) {
rec = rec " " $3
}
else {
if (rec) print rec
rec = $0
}
prev = curr
}
END { if (rec) print rec }
$ awk -f tst.awk file
col1|col2|col3
1|a|desc1 desc2
1|b|desc3 desc4 desc5
2|c|desc6 desc7

Answer 2

here is an awk-oneliner, do the merge and keep the order:
awk -F'|' '{k=$1FS$2;if(a[k])a[k]=a[k] OFS $3;else{a[k]=$0;b[++i]=k}}
END{for(x=1;x<=i;x++)print a[b[x]]}' file

Answer 3

Here's a perl solution:
open $fh, "<", "yourfile.txt";
%h = ();
$head = <$fh>;
while (<$fh>) {
if ($_ =~ /(\d\|[a-z]\|)(.*)/) {
$h{$1} .= "$2 ";
}
}
print $head;
foreach (sort keys %h) {
print "$_$h{$_}\n";
}
You can use a regex to get the combined col1 and col2 and store it as a hash key, then append the col3 values to that key whilst looping over the rest of the file.

Answer 4

Perl from command line,
perl -lne'
/(.+\|)(.+)/ or next;
$h{$1} or push #r, $1;
push #{ $h{$1} }, $2;
END { print $_, "#{$h{$_}}" for #r }
' file