Remove duplicate lines in braces - bash

Question

I a have file that contains:
l1_lololo {
abcdef
vgjhklfgkchbnvu
gfuhjfythkjbgftyhkjgyftuihgt6
deefgik
abcdef
}
l2_blabla {
123456
vgghyfthjfgtrdygfhhbnvu
gfuhjgvftdyfgvjgyftuihgt6
deiulouk
123456
}
I need to check text in braces with sed/awk/bash/etc. and remove duplicates of lines, leaving only first of recurring line in each braces, I need to get this:
l1_lololo {
abcdef
vgjhklfgkchbnvu
gfuhjfythkjbgftyhkjgyftuihgt6
deefgik
}
l2_blabla {
123456
vgghyfthjfgtrdygfhhbnvu
gfuhjgvftdyfgvjgyftuihgt6
deiulouk
}
How I can do this?

Answer 1

If you can guarantee that the blocks end with a line containing only }, it could be done as simply as:
awk '/^}$/ {delete a} !a[$0]++' input
If you need a more robust solution, perhaps just add some whitespace to the pattern to match the end of a block. But if you want a full parser and want to match braces carefully, awk is probably not suited for the task.

Answer 2

If you're open to other languages, this is really easy to do in tcl thanks to the input being in tcl list format, allowing you to use it to do all the parsing without any potentially fragile regular expressions:
#!/usr/bin/env tclsh
package require Tcl 8.5
foreach {key lst} [read stdin] {
foreach item $lst { dict set seen $item 1 }
puts "$key {\n\t[join [dict keys $seen] \n\t]\n}\n"
unset seen
}
Example:
$ ./dedup < input.txt
l1_lololo {
abcdef
vgjhklfgkchbnvu
gfuhjfythkjbgftyhkjgyftuihgt6
deefgik
}
l2_blabla {
123456
vgghyfthjfgtrdygfhhbnvu
gfuhjgvftdyfgvjgyftuihgt6
deiulouk
}

Answer 3

Desired result can be achieved with following code (data stored in a hash)
use strict;
use warnings;
use feature 'say';
my $data = do{ local $/; <DATA> }; # read whole data
my %seen;
my %records = $data =~ /(\w+)\s+\{\s*(.*?)\s*\}/sg; # split into records
while( my($k,$v) = each %records ) { # for each record split into array
my #array = map { if( not $seen{$_} ) { $seen{$_} = 1; $_ } } split '\s+', $records{$k}; # store uniq elements
pop #array; # pop out last empty element
$records{$k} = \#array; # store array in hash
}
while( my($k,$v) = each %records ) { # each record
say "$k = {"; # output hash key
say "\t$_" for #{$v}; # output each element of array
say "}\n"; # done
}
__DATA__
l1_lololo {
abcdef
vgjhklfgkchbnvu
gfuhjfythkjbgftyhkjgyftuihgt6
deefgik
abcdef
}
l2_blabla {
123456
vgghyfthjfgtrdygfhhbnvu
gfuhjgvftdyfgvjgyftuihgt6
deiulouk
123456
}
Output
l1_lololo = {
abcdef
vgjhklfgkchbnvu
gfuhjfythkjbgftyhkjgyftuihgt6
deefgik
}
l2_blabla = {
123456
vgghyfthjfgtrdygfhhbnvu
gfuhjgvftdyfgvjgyftuihgt6
deiulouk
}

Answer 4

This might work for you (GNU sed):
sed -E '/^\S+ \{/{:a;N;s/((\n[^\n]*)(\n.*)*)\2$/\1/;/\n\}$/!ba}' file
If a line begins with some text followed by a {, append the next line and remove the last line if it matches a preceding line. Repeat the latter until a line containing only a } and print the result.