Sorting two Dictionaries and finding the differences in index position of items in the sorted list - algorithm

Question

I have a challenge in front of me. Let me present the challenge which is perplexing me -
There are two dictionaries say - D1 and D2.
These dictionaries have same keys most of the time but there is no guarantee that it will be always the same.
The two Dictionaries could be represented as follows -
D1 = {["R1", 0.7], ["R2",0.73], ["R3", 1.5], ["R4", 2.5], ["R5", 0.12], ["R6", 1.9], ["R7", 9.8], ["R8", 6.5], ["R9", 7.2], ["R10", 5.6]};
D2 = {["R1", 0.7], ["R2",0.8], ["R3", 1.5], ["R4", 3.1], ["R5", 0.10], ["R6", 2.0], ["R7", 8.0], ["R8", 1.0], ["R9", 0.0], ["R10", 5.6], ["R11", 6.23]};
Here in these dictionaries, the keys are of string data type and value are of float data type.
Physically they are snapshot of a system in two different times. D1 being older than D2.
I need to sort these dictionaries independently based on the values in ascending orders. Which when done changes these dictionaries to -
D1 = {["R5", 0.12], ["R1", 0.7], ["R2",0.73], ["R3", 1.5], ["R6", 1.9], ["R4", 2.5], ["R10", 5.6], ["R8", 6.5], ["R9", 7.2], ["R7", 9.8]};
and
D2 = {["R9", 0.0], ["R5", 0.10], ["R1", 0.7], ["R2",0.8], ["R8", 1.0], ["R3", 1.5], ["R6", 2.0], ["R4", 3.1], ["R10", 5.6], ["R11", 6.23], ["R7", 8.0]};
Here the sorting of elements in the dictionary D1 is taken as reference point. Each element of the D1 is connected with the immediate next one in D1. It is expected to identify the elements in D2 which have broken the sequence as it appears in the reference dictionary D1 after sorting. While determining this addition of elements (i.e. the key not being present in D1 but is present in D2) to D2 and removal of elements (i.e. the key is present in D1 but not in D2) from D1 are ignored. i.e They should not highlighted in the result.
For example, in continuing with the example listed above, the elements which break a sequence in D2 with reference to D1 (ignoring addition and removal) are -
Breakers = {["R9", 0.0],["R8", 1.0]} since, R9 has jumped the sequence from 8th index in D1 sorted dictionary to 0th index in D2 sorted dictionary. Similarly R8 has jumped the sequence from 7th index in D1 sorted dictionary to 4th index in D2 sorted dictionary (all indexes are started from 0).
Note - ["R11", 6.23] is not expected to be in the list of Breakers since it is addition to D2.
Please suggest an algorithm to achieve this optimally, since this operation needs to be performed on data fetched from a database with 3,256,190 records.
Programming language is not a worry, if guided with logic I could take up the task of implementing it in any language.

Answer 1

I came up with this algorithm in C#. It works perfect for you example data. I also did a test with 3000000 totally random values (so a lot of breakers are detected) and it completes in 3.2seconds on my notebook (Intel Core i3 2.1GHz, 64bit).
I first put your data into temporary dictionaries, so I could copy-paste your values, before I put them into the Lists. Of course your application will put them directly in the lists.
class Program
{
struct SingleValue
{
public string Key;
public float Value;
public SingleValue(string key, float value)
{
Key = key;
Value = value;
}
public override string ToString()
{
return string.Format("{0}={1}", Key, Value);
}
}
static void Main(string[] args)
{
List<SingleValue> D1 = new List<SingleValue>();
HashSet<string> D1keys = new HashSet<string>();
List<SingleValue> D2 = new List<SingleValue>();
#if !LARGETEST
Dictionary<string, double> D1input = new Dictionary<string, double>() { { "R1", 0.7 }, { "R2", 0.73 }, { "R3", 1.5 }, { "R4", 2.5 }, { "R5", 0.12 }, { "R6", 1.9 }, { "R7", 9.8 }, { "R8", 6.5 }, { "R9", 7.2 }, { "R10", 5.6 } };
Dictionary<string, double> D2input = new Dictionary<string, double>() { { "R1", 0.7 }, { "R2", 0.8 }, { "R3", 1.5 }, { "R4", 3.1 }, { "R5", 0.10 }, { "R6", 2.0 }, { "R7", 8.0 }, { "R8", 1.0 }, { "R9", 0.0 }, { "R10", 5.6 }, { "R11", 6.23 } };
// You should directly put you values into this list... I converted them from a Dictionary so I didn't have to type over your input values :)
foreach (KeyValuePair<string, double> kvp in D1input)
{
D1.Add(new SingleValue(kvp.Key, (float)kvp.Value));
D1keys.Add(kvp.Key);
}
foreach (KeyValuePair<string, double> kvp in D2input)
D2.Add(new SingleValue(kvp.Key, (float)kvp.Value));
#else
Random ran = new Random();
for (int i = 0; i < 3000000; i++)
{
D1.Add(new SingleValue(i.ToString(), (float)ran.NextDouble()));
D1keys.Add(i.ToString());
D2.Add(new SingleValue(i.ToString(), (float)ran.NextDouble()));
}
#endif
// Sort the lists
D1.Sort(delegate(SingleValue x, SingleValue y)
{
if (y.Value > x.Value)
return -1;
else if (y.Value < x.Value)
return 1;
return 0;
});
D2.Sort(delegate(SingleValue x, SingleValue y)
{
if (y.Value > x.Value)
return -1;
else if (y.Value < x.Value)
return 1;
return 0;
});
int start = Environment.TickCount;
Dictionary<string, float> breakers = new Dictionary<string, float>();
List<SingleValue> additions = new List<SingleValue>();
// Walk through D1
IEnumerator<SingleValue> i1 = D1.GetEnumerator();
IEnumerator<SingleValue> i2 = D2.GetEnumerator();
while (i1.MoveNext() && i2.MoveNext())
{
while (breakers.ContainsKey(i1.Current.Key))
{
if (!i1.MoveNext())
break;
}
while (i1.Current.Key != i2.Current.Key)
{
if (D1keys.Contains(i2.Current.Key))
breakers.Add(i2.Current.Key, i2.Current.Value);
else
additions.Add(i2.Current);
if (!i2.MoveNext())
break;
}
}
int duration = Environment.TickCount - start;
Console.WriteLine("Lookup took {0}ms", duration);
Console.ReadKey();
}
}

Answer 2

If you could delete stuff from D2 before sorting it would be easy, right? You say you can't delete the data. However, you can make an additional data structure that simulates such deletions (e.g., add a "deleted" bit to the item, or if you can't change its type then make a set of "deleted" items). Then run the simple algorithm, but make sure to ignore items that are "deleted".

Answer 3

I have been thinking about this. As you mentioned Levenshtein distance, I am assuming you want to get those elements by moving whom from their position in D2 to some other position in D2 you will get D1 from D2 in the least number of moves (ignoring elements which don't exist in both sequences).
I wrote a greedy algorithm which may be sufficient for your needs, but it may not necessarily give the optimal result in all cases. I am honestly not sure and may come back to it later (weekend at the earliest) to check the correctness. However, if you really need to do this on sequences of 3 million elements, I believe that no algorithm which does any kind of good job at this will be fast enough, because I can't see an O(n) algorithm which does a good job and can't fail even on some trivial inputs.
This algorithm tries moving each element to its intended position and calculates the sum of errors (how far each element is from its original position) after the move. The element which results in the lowest sum of errors is proclaimed a breaker and moved. This is repeated until the sequence reverts back to D1.
I think it has O(n^3) complexity, although elements sometimes need to be moved multiple times so it could possibly be O(n^4) worst case, I'm not sure, but in 1 million random examples with 50 elements the max number of outer loop runs was 51 (n^4 would mean it can be 2500 and somehow I was lucky in all my million tests). There are just keys, no values. This is because the values are irrelevant in this step so there is no point in storing them.
edit: I wrote a counterexample generator for this and indeed it is not always optimal. The more breakers there are, the less of a chance of an optimal solution. For example, in 1000 elements with 50 randomly moved ones it will usually find a set of 55-60 breakers, when the optimal solution is at most 50.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
namespace Breakers
{
class Program
{
static void Main(string[] args)
{
//test case 1
//List<string> L1 = new List<string> { "R5", "R1", "R2", "R3", "R6", "R4", "R10", "R8", "R9", "R7" };
//List<string> L2 = new List<string> { "R9", "R5", "R1", "R2", "R8", "R3", "R6", "R4", "R10", "R11", "R7" };
//GetBreakers<string>(L1, L2);
//test case 2
//List<string> L1 = new List<string> { "R5", "R1", "R2", "R3", "R6", "R4", "R10", "R8", "R9", "R7" };
//List<string> L2 = new List<string> { "R5", "R9", "R1", "R6", "R2", "R3", "R4", "R10", "R8", "R7" };
//GetBreakers<string>(L1, L2);
//test case 3
List<int> L1 = new List<int>();
List<int> L2 = new List<int>();
Random r = new Random();
int n = 100;
for (int i = 0; i < n; i++)
{
L1.Add(i);
L2.Add(i);
}
for (int i = 0; i < 5; i++) // number of random moves, this is the upper bound of the optimal solution
{
int a = r.Next() % n;
int b = r.Next() % n;
if (a == b)
{
i--;
continue;
}
int x = L2[a];
Console.WriteLine(x);
L2.RemoveAt(a);
L2.Insert(b, x);
}
for (int i = 0; i < L2.Count; i++) Console.Write(L2[i]);
Console.WriteLine();
GetBreakers<int>(L1, L2);
}
static void GetBreakers<T>(List<T> L1, List<T> L2)
{
Dictionary<T, int> Appearances = new Dictionary<T, int>();
for (int i = 0; i < L1.Count; i++) Appearances[L1[i]] = 1;
for (int i = 0; i < L2.Count; i++) if (Appearances.ContainsKey(L2[i])) Appearances[L2[i]] = 2;
for (int i = L1.Count - 1; i >= 0; i--) if (!(Appearances.ContainsKey(L1[i]) && Appearances[L1[i]] == 2)) L1.RemoveAt(i);
for (int i = L2.Count - 1; i >= 0; i--) if (!(Appearances.ContainsKey(L2[i]) && Appearances[L2[i]] == 2)) L2.RemoveAt(i);
Dictionary<T, int> IndInL1 = new Dictionary<T, int>();
for (int i = 0; i < L1.Count; i++) IndInL1[L1[i]] = i;
Dictionary<T, int> Breakers = new Dictionary<T, int>();
int steps = 0;
int me = 0;
while (true)
{
steps++;
int minError = int.MaxValue;
int minErrorIndex = -1;
for (int from = 0; from < L2.Count; from++)
{
T x = L2[from];
int to = IndInL1[x];
if (from == to) continue;
L2.RemoveAt(from);
L2.Insert(to, x);
int error = 0;
for (int i = 0; i < L2.Count; i++)
error += Math.Abs((i - IndInL1[L2[i]]));
L2.RemoveAt(to);
L2.Insert(from, x);
if (error < minError)
{
minError = error;
minErrorIndex = from;
}
}
if (minErrorIndex == -1) break;
T breaker = L2[minErrorIndex];
int breakerOriginalPosition = IndInL1[breaker];
L2.RemoveAt(minErrorIndex);
L2.Insert(breakerOriginalPosition, breaker);
Breakers[breaker] = 1;
me = minError;
}
Console.WriteLine("Breakers: " + Breakers.Count + " Steps: " + steps);
foreach (KeyValuePair<T, int> p in Breakers)
Console.WriteLine(p.Key);
Console.ReadLine();
}
}
}