Linq: Group on list inside lists - linq

Question

Let's say I have a list of lists.
For each item in this list , I have a list of custom objects.
These objects are as such:
public string Field1
public string Field2
public string Field3
What I'd like to achieve through Linq: filter out of my list of lists all the objects which have the same three fields, which are not the first element of their list and keep only the first one.
So let's say I have two lists listA and list B in my list.
listA has three objects object1, object2 and object3.
object1.Field1 = "a" object1.Field2 = "A" object1.Field3 = "1"
object2.Field1 = "a" object2.Field2 = "B" object2.Field3 = "2"
object3.Field1 = "a" object3.Field2 = "C" object3.Field3 = "3"
listB has three objects object4, object5 and object6.
object4.Field1 = "a" object4.Field2 = "A" object4.Field3 = "1"
object5.Field1 = "a" object5.Field2 = "B" object5.Field3 = "2"
object6.Field1 = "a" object6.Field2 = "D" object6.Field3 = "3"
In this example, object1 and object4 are the same, but because they are first in their respective list, they are not filtered out.
However, object2 and object5 having the same three fields values, only one of them will be kept so that at then end of my process, I'll have my two list like so:
listA has three objects object1, object2 and object3.
object1.Field1 = "a" object1.Field2 = "A" object1.Field3 = "1"
object2.Field1 = "a" object2.Field2 = "B" object2.Field3 = "2"
object3.Field1 = "a" object3.Field2 = "C" object3.Field3 = "3"
listB has now two objects object4 and object6.
object4.Field1 = "a" object4.Field2 = "A" object4.Field3 = "1"
object6.Field1 = "a" object6.Field2 = "D" object6.Field3 = "3"
I've been scratching my head for hours about this to no avail. I cannot do a foreach list of lists look into all the other lists as it would cause a performance problem (I can potentially have 1000000 lists of lists).
Anyone with an idea for this?

Answer 1

Why does it have to be LINQ? A simple iterator block solves the problem quite nicely.
The code below assumes you have overridden Equals and GetHashCode in your object to check for equality on the 3 fields. If that is not possible, use a custom equality comparer instead (to be passed in the HashSet constructor)
static IEnumerable<List<T>> GetFilteredList<T>(IEnumerable<List<T>> input)
{
var found = new HashSet<T>();
foreach (var list in input)
{
var returnList = new List<T>(list.Capacity);
foreach (var item in list)
{
// the first item is always added
// any other items are only added if they were
// never encountered before
if (list.Count == 0 || !found.Contains(item))
{
found.Add(item);
returnList.Add(item);
}
}
yield return returnList;
}
}
If you can stick with IEnumerable<IEnumerable<T>> as the return value, another approach that only sweeps the input just once could be something like this (without creating intermediary lists):
static IEnumerable<IEnumerable<T>> GetFilteredList<T>(IEnumerable<List<T>> input)
{
var encounteredElements = new HashSet<T>();
foreach (var list in input)
{
yield return Process(list, encounteredElements);
}
}
static IEnumerable<T> Process<T>(IEnumerable<T> input,
HashSet<T> encounteredElements)
{
bool first = true;
foreach (var item in input)
{
if (first) yield return item;
if (!encounteredElements.Contains(item))
{
yield return item;
}
encounteredElements.Add(item);
first = false;
}
}