Remove duplicates in O(n) by hand - algorithm

Question

I need to remove all duplicates in a list, but only if the item in list a is the same in list b aswell. This is my current code, but at 100k items it's taking literal days, is there a fast way to do this?
Any help appreciated.
List<int> ind = new List<int>();
List<int> used = new List<int>();
for (int i = 0; i < a.Count; i++)
{
for (int j = 0; j < a.Count; j++)
{
if (i != j&&!used.Contains(i))
{
if (a[j] == a[i] && b[i] == b[j])
{
ind.Add(j);
used.Add(j);
}
}
}
}
List<string> s2 = new List<string>();
List<string> a2 = new List<string>();
for (int i = 0; i < a.Count; i++)
{
if (!ind.Contains(i))
{
s2.Add(a[i]);
a2.Add(b[i]);
}
}

Answer 1

The key to many such problems is the correct data structure. To avoid duplicates, you need to use Sets, as they remove duplicates automatically.
Here is the code in Java, I hope it is similar in C#:
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Random;
import java.util.Set;
class Duplicates
{
static List<Integer> list1 = new ArrayList<>();
static List<Integer> list2 = new ArrayList<>();
static final int SIZE = 100_000;
static final int MAX_VALUE = 1000_000;
public static void main(String[] args)
{
// populate the lists with random values for testing
Random r = new Random();
for(int i=0; i<SIZE; i++)
{
list1.add(r.nextInt(MAX_VALUE));
list2.add(r.nextInt(MAX_VALUE));
}
Set<Integer> set1 = new HashSet<>(list1);
Set<Integer> set2 = new HashSet<>(list2);
// items that are in both lists
Set<Integer> intersection = new HashSet<>(set1);
intersection.retainAll(set2);
Set<Integer> notSeenYet = new HashSet<>(intersection);
List<Integer> list1Unique = new ArrayList<Integer>();
for(int n: list1)
{
if(intersection.contains(n)) // we may have to skip this one
{
if(notSeenYet.contains(n)) // no, don't skip, it's the first occurrence
{
notSeenYet.remove(n);
}
else
{
continue;
}
}
list1Unique.add(n);
}
System.out.println("list 1 contains "+list1Unique.size()+" values after removing all duplicates that are also in list 2");
}
}
It takes less than a second for 100k values.
Output
list 1 contains 99591 values after removing all duplicates that are
also in list 2

Answer 2

Create a HashSet.
First, iterate through the list b and add all elements into the HashSet.
Then, iterate through each element of the list a. When you visit an element, ask the HashSet if it already contains that element. If if doesn't, it's a new element, so just keep it. If it does, it is a duplicate and you can remove it from a.
HashSets can perform the Do you have this element? question in O(1), so for the whole list, you have O(n).
For more information, check the documentation.

Answer 3

Here is a general algorithm to consider. We can start by sorting both lists in ascending order. Using a good sorting algorithm such as merge sort, this would take O(NlgN) time, where N is the length of the list. Once we have paid this penalty, we need only maintain two pointers in each of the lists. The general algorithm would basically involve walking up both lists, searching for duplicates in the first a list, should the value in question match the pointer into the b list. If there be a match, then duplicates would be removed from the a list, otherwise we keep walking until reaching the end of the a list. This process would be only O(N), making the biggest penalty the initial sort, which is O(NlgN).

Answer 4

To "remove duplicates" I understand to mean "from n identical items, leave the first and remove the remaining n - 1". If so then this is the algorithm:
Convert list b to set B. Also introduce set A_dup. Run through list a and for each item:
if item is found in A_dup then remove it from a,
else if item is found in set B then add it to A_dup.
Repeat.
Checking for existence in sets (both A_dup and B) is O(1) operation, also to add new item in the set. So, you're left with iterating through list a, which in total gives us O(n).

Answer 5

I think what you are trying to do is find distinct pairs, right?
If so, you can do that in one line using Zip and Distinct and a C# Tuple (or use an anonymous type).
var result = a.Zip(b, (x,y) => (x, y)).Distinct();

Answer 6

import java.util.*;
import java.util.stream.Collectors;
public class Test {
public static void main(String args[]) {
List<String> dupliKhaneList = new ArrayList<>();
dupliKhaneList.add("Vaquar");
dupliKhaneList.add("Khan");
dupliKhaneList.add("Vaquar");
dupliKhaneList.add("Vaquar");
dupliKhaneList.add("Khan");
dupliKhaneList.add("Vaquar");
dupliKhaneList.add("Zidan");
// Solution 1 if want to remove in list
List<String> uniqueList = dupliKhaneList.stream().distinct().collect(Collectors.toList());
System.out.println("DupliKhane => " + dupliKhaneList);
System.out.println("Unique 1 => " + uniqueList);
// Solution 2 if want to remove using 2 list
List<String> list1 = new ArrayList<>();
list1.add("Vaquar");
list1.add("Khan");
list1.add("Vaquar");
list1.add("Vaquar");
list1.add("Khan");
list1.add("Vaquar");
list1.add("Zidan");
List<String> list2 = new ArrayList<>();
list2.add("Zidan");
System.out.println("list1 => " + list1);
System.out.println("list2 => " + list2);
list1.removeAll(list2);
System.out.println("removeAll duplicate => " + list1);
}
}
Results :
DupliKhane => [Vaquar, Khan, Vaquar, Vaquar, Khan, Vaquar, Zidan]
Unique 1 => [Vaquar, Khan, Zidan]
list1 => [Vaquar, Khan, Vaquar, Vaquar, Khan, Vaquar, Zidan]
list2 => [Zidan]
removeAll duplicate => [Vaquar, Khan, Vaquar, Vaquar, Khan, Vaquar]