Best sorting algorithm [closed] - algorithm

Question

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
What is the best algorithm to sort out unique words from a list of more than 10 million words? we need the best technique in the terms of execution time.

Answer 1

There are two simple approaches I remember using:
Add all the items to a data structure that folds duplicates (generally a hash, but you can also try a balanced tree or a trie).
Sort the list, then run over it copying out all elements that are non-equal to the previous element.
Roughly speaking, and subject to the usual fudges, the hash table and the trie give you expected O(n), the balanced tree and the sort give you expected O(n log n). It is not necessarily true that the O(n) solutions are faster than the O(n log n) solutions for your particular data.
All the options in (1) may have the disadvantage of doing a lot of small memory allocations for nodes in a data structure, which can be slow unless you use a special-purpose allocator. So in my experience it's worth testing the sort on the size of data you actually care about, before embarking on anything that requires you to write significant code.
Depending what language you're using, some of these approaches might be easier to test than others. For example in Python if you have a list of strings then the hashtable approach is just set(my_strings). In C, there is no standard hashtable, so you're either writing one or looking for a library.
Of course ease of writing has no direct effect on execution time, so if (as you claim) your programmer time is immaterial and all that matters is execution speed, then you should have no problems spending a few weeks getting familiar with the best available literature on sorting and hash tables. You'd be far better able to answer the question than I am.

Answer 2

Just add them to a hash. Constant time insert. I don't believe you can do better than order n. Red black trees can be faster on small data sets (faster to traverse the tree than to compute the hash), but your data set is large.

Answer 3

Spoiler:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
struct somehash {
struct somehash *next;
unsigned hash;
char *mem;
};
#define THE_SIZE (10*1000*1000)
struct somehash *table[THE_SIZE] = { NULL,};
struct somehash **some_find(char *str, unsigned len);
static unsigned some_hash(char *str, unsigned len);
int main (void)
{
char buffer[100];
struct somehash **pp;
size_t len;
while (fgets(buffer, sizeof buffer, stdin)) {
len = strlen(buffer);
pp = some_find(buffer, len);
if (*pp) { /* found */
fprintf(stderr, "Duplicate:%s", buffer);
}
else { /* not found: create one */
fprintf(stdout, "%s", buffer);
*pp = malloc(sizeof **pp);
(*pp)->next = NULL;
(*pp)->hash = some_hash(buffer,len);
(*pp)->mem = malloc(1+len);
memcpy((*pp)->mem , buffer, 1+len);
}
}
return 0;
}struct somehash **some_find(char *str, unsigned len)
{
unsigned hash;
unsigned short slot;
struct somehash **hnd;
hash = some_hash(str,len);
slot = hash % THE_SIZE;
for (hnd = &table[slot]; *hnd ; hnd = &(*hnd)->next ) {
if ( (*hnd)->hash != hash) continue;
if ( strcmp((*hnd)->mem , str) ) continue;
break;
}
return hnd;
}
static unsigned some_hash(char *str, unsigned len)
{
unsigned val;
unsigned idx;
if (!len) len = strlen(str);
val = 0;
for(idx=0; idx < len; idx++ ) {
val ^= (val >> 2) ^ (val << 5) ^ (val << 13) ^ str[idx] ^ 0x80001801;
}
return val;
}