I am working on grouping of elements for large dataset in least amount of time.
Below is small dataset example of the same:
Input:
pairs = [[1,2],[8,10],[4,8],[3,4],[2,3],[5,6],[6,7],[9,4],[11,12],[13,10],[16,1],[78,79],[61,3],[93,94]]
How to derive output:
If any of the element is present in another pair then both pair can be grouped. This way traverse all pairs and create different groups.
Output: -
[{1, 2, 3, 4, 8, 9, 10, 13, 16, 61}, {5, 6, 7}, {11, 12}, {78, 79}, {93, 94}]
Question:
What is the best way to solve this problem?
e.g.: graph database(neo4j) or any specific readily available algorithm.
any other alternatives which is not listed in example solution(neo4j, algorithm) above?
Below output is expected and solution should consume less time with large dataset.
Expected Output: -
[{1, 2, 3, 4, 8, 9, 10, 13, 16, 61}, {5, 6, 7}, {11, 12}, {78, 79}, {93, 94}]
I have a list with 500+ thousand subsets each having from 1 to 500 values (integers). So i have something like:
{1, 2, 3 }
{2, 3}
{4, 5}
{3, 6, 7}
{7, 9}
{8, 4}
{10, 11}
After running the code I would like to get:
{1, 2, 3, 6, 7, 9}
{4, 5, 8}
{10, 11}
I wrote simple code [here] that compares each subset to each subset, if they intersect they are joined together, else not.
It is ok on small scale, but with big amount of data it takes forever.
Please, could you advise any improvements?
P.S. I am not strong in maths or logics, big O notation would be greek for me. I am sorry.
You're trying to find the connected components in a graph, with each of your input sets representing a set of nodes that's fully connected. Here's a simple implementation:
sets = [{1, 2, 3 },{2, 3},{4, 5},{3, 6, 7},{7, 9},{8, 4},{10, 11}]
allelts = set.union(*sets)
components = {X: {X} for X in allelts}
component = {X: X for X in allelts}
for S in sets:
comp = sorted({component[X] for X in S})
mergeto = comp[0]
for mergefrom in comp[1:]:
components[mergeto] |= components[mergefrom]
for X in components[mergefrom]:
component[X] = mergeto
del components[mergefrom]
That results in components having a list of components (keyed by their minimum element), and component storing the components for each element:
>>> print(components)
{1: {1, 2, 3, 6, 7, 9}, 4: {8, 4, 5}, 10: {10, 11}}
>>> print(component)
{1: 1, 2: 1, 3: 1, 4: 4, 5: 4, 6: 1, 7: 1, 8: 4, 9: 1, 10: 10, 11: 10}
>>>
I am trying to solve above Longest Monotonically Increasing Subsequence problem using javascript. In order to do that, I need to know about Longest Monotonically Subsequence. Current I am following wikipedia article. The thing I am not understanding this example is that the longest increasing subsequence is given as 0, 2, 6, 9, 13, 15 from 0, 8, 4, 12, 2, 10, 6, 14, 1, 9, 5, 13, 3, 11, 7, 15, … list. The question is Why the answer does not have 3 in between 2 and 6, and 8 between 6 and 9 etc? How does that answer come from that list?
Ist of all , consider the name "Longest Monotonically Increasing Subsequence" . So , from the given array you need to figure out the largest sequence where the numbers should be appeared in a strictly increasing fashion. There can be many sequence, where the sub array can be strictly increasing but you need to find the largest sub-Array.
So. lets debug this array. a[] = {0, 8, 4, 12, 2, 10, 6, 14, 1, 9, 5, 13, 3, 11, 7, 15}
In here the some monotonously increasing sub-arrays are :
{0,8,12,14,15} Length = 5
{0,4,12,14,15} Length = 5
{0,1,9,13,15} Length = 5 and so on.
But if you calculate like this , you can find the largest sub-array will be :
{0, 2, 6, 9, 13, 15} , Length = 6, so this is the answer.
Every single little time you pick any number , the next number should be large than the previous one and must be present in the array. say {0, 2, 6, 9, 13, 15} this list, when you pick 9 , then the next number should be larger than 9. the immediate sequence shows 13>9, so you can pick 13. You can also pick 11. But that will create another branch of sub-array. Like :
{0, 2, 6, 9, 11, 15} which is another solution.
Hope this explanation will help you to understand the LIS (Longest Increasing Subsequence).Thanks.
First of all, the title of your question says: Longest increasing CONTIGUOUS subsequence which is a slight variation of the original problem of LIS in which the result need not have contiguous values from original array as pointed out in above examples. Follow this link for a decent explanation on LIS algorithm which has O(n^2) solution and it can be optimized to have a O(nlogn) solution:
http://www.algorithmist.com/index.php/Longest_Increasing_Subsequence
for the contiguous variant of LIS, here is a decent solution:
http://worldofbrock.blogspot.com/2009/10/how-to-find-longest-continuous.html
Say we have a 0-indexed sequence S, take S[0] and insert it in a place in S where the next value is higher than S[0] and the previous value is lower than S[0]. Formally, S[i] should be placed in such a place where S[i-1] < S[i] < S[i+1]. Continue in order on the list doing the same with every item. Remove the element from the list before putting it in the correct place. After one iteration over the list the list should be ordered. I recently had an exam and I forgot insertion sort (don't laugh) and I did it like this. However, my professor marked it wrong. The algorithm, as far as I know, does produce a sorted list.
Works like this on a list:
Sorting [2, 8, 5, 4, 7, 0, 6, 1, 10, 3, 9]
[2, 8, 5, 4, 7, 0, 6, 1, 10, 3, 9]
[2, 8, 5, 4, 7, 0, 6, 1, 10, 3, 9]
[2, 5, 4, 7, 0, 6, 1, 8, 10, 3, 9]
[2, 4, 5, 7, 0, 6, 1, 8, 10, 3, 9]
[2, 4, 5, 7, 0, 6, 1, 8, 10, 3, 9]
[2, 4, 5, 0, 6, 1, 7, 8, 10, 3, 9]
[0, 2, 4, 5, 6, 1, 7, 8, 10, 3, 9]
[0, 2, 4, 5, 1, 6, 7, 8, 10, 3, 9]
[0, 1, 2, 4, 5, 6, 7, 8, 10, 3, 9]
[0, 1, 2, 4, 5, 6, 7, 8, 3, 9, 10]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
Got [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
Since every time an element is inserted into the list up to (n-1) numbers in the list may be moved and we must do this n times the algorithm should run in O(n^2) time.
I had a Python implementation but I misplaced it somehow. I'll try to write it again in a bit, but it's kinda tricky to implement. Any ideas?
The Python implementation is here: http://dpaste.com/hold/522232/. It was written by busy_beaver from reddit.com when it was discussed here http://www.reddit.com/r/compsci/comments/ejaaz/is_this_equivalent_to_insertion_sort/
It's a while since this was asked, but none of the other answers contains a proof that this bizarre algorithm does in fact sort the list. So here goes.
Suppose that the original list is v1, v2, ..., vn. Then after i steps of the algorithm, I claim that the list looks like this:
w1,1, w1,2, ..., w1,r(1), vσ(1), w2,1, ... w2,r(2), vσ(2), w3,1 ... ... wi,r(i), vσ(i), ...
Where σ is the sorted permutation of v1 to vi and the w are elements vj with j > i. In other words, v1 to vi are found in sorted order, possibly interleaved with other elements. And moreover, wj,k ≤ vj for every j and k. So each of the correctly sorted elements is preceded by a (possibly empty) block of elements less than or equal to it.
Here's a run of the algorithm, with the sorted elements in bold, and the preceding blocks of elements in italics (where non-empty). You can see that each block of italicised elements is less than the bold element that follows it.
[4, 8, 6, 1, 2, 7, 5, 0, 3, 9]
[4, 8, 6, 1, 2, 7, 5, 0, 3, 9]
[4, 6, 1, 2, 7, 5, 0, 3, 8, 9]
[4, 1, 2, 6, 7, 5, 0, 3, 8, 9]
[1, 4, 2, 6, 7, 5, 0, 3, 8, 9]
[1, 2, 4, 6, 7, 5, 0, 3, 8, 9]
[1, 2, 4, 6, 5, 0, 3, 7, 8, 9]
[1, 2, 4, 5, 6, 0, 3, 7, 8, 9]
[0, 1, 2, 4, 5, 6, 3, 7, 8, 9]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
If my claim is true, then the algorithm sorts, because after n steps all the vi are in order, and there are no remaining elements to be interleaved. But is the claim really true?
Well, let's prove it by induction. It's certainly true when i = 0. Suppose it's true for i. Then when we run the (i + 1)st step, we pick vi+1 and move it into the first position where it fits. It certainly passes over all vj with j ≤ i and vj < vi+1 (since these are sorted by hypothesis, and each is preceded only by smaller-or-equal elements). It cannot pass over any vj with j ≤ i and vj ≥ vi+1, because there's some position in the block before vj where it will fit. So vi+1 ends up sorted with respect to all vj with j ≤ i. So it ends up somewhere in the block of elements before the next vj, and since it ends up in the first such position, the condition on the blocks is preserved. QED.
However, I don't blame your professor for marking it wrong. If you're going to invent an algorithm that no-one's seen before, it's up to you to prove it correct!
(The algorithm needs a name, so I propose fitsort, because we put each element in the first place where it fits.)
Your algorithm seems to me very different from insertion sort. In particular, it's very easy to prove that insertion sort works correctly (at each stage, the first however-many elements in the array are correctly sorted; proof by induction; done), whereas for your algorithm it seems much more difficult to prove this and it's not obvious exactly what partially-sorted-ness property it guarantees at any given point in its processing.
Similarly, it's very easy to prove that insertion sort always does at most n steps (where by a "step" I mean putting one element in the right place), whereas if I've understood your algorithm correctly it doesn't advance the which-element-to-process-next pointer if it's just moved an element to the right (or, to put it differently, it may sometimes have to process an element more than once) so it's not so clear that your algorithm really does take O(n^2) time in the worst case.
Insertion sort maintains the invariant that elements to the left of the current pointer are sorted. Progress is made by moving the element at the pointer to the left into its correct place and advancing the pointer.
Your algorithm does this, but sometimes it also does an additional step of moving the element at the pointer to the right without advancing the pointer. This makes the algorithm as a whole not an insertion sort, though you could call it a modified insertion sort due to the resemblance.
This algorithm runs in O(n²) on average like insertion sort (also like bubble sort). The best case for an insertion sort is O(n) on an already sorted list, for this algorithm it is O(n) but for a reverse-sorted list since you find the correct position for every element in a single comparison (but only if you leave the first, largest, element in place at the beginning when you can't find a good position for it).
A lot of professors are notorious for having the "that's not the answer I'm looking for" bug. Even if it's correct, they'll say it doesn't meet their criteria.
What you're doing seems like insertion sort, although using removes and inserts seems like it would only add unnecessary complexity.
What he might be saying is you're essentially "pulling out" the value and "dropping it back in" the correct spot. Your prof was probably looking for "swapping the value up (or down) until you found it's correct location."
They have the same result but they're different in implementation. Swapping would be faster, but not significantly so.
I have a hard time seeing that this is insert sort. Using insert sort, at each iteration, one more element would be placed correctly in the array. In your solution I do not see an element being "fully sorted" upon each iteration.
The insert sort algorithm begin:
let pos = 0
if pos == arraysize then return
find the smallest element in the remaining array from pos and swap it with the element at position pos
pos++
goto 2