number of unique words in a document - bash

Question

I have a very large txt file (500GiB), and I want to get the number of its unique words. I tried this, but it seems to be very slow as it does sort:
grep -o -E '\w+' temp | sort -u -f | wc -l
Is there any better way of doing this?

Answer 1

awk to the rescue!
$ awk -v RS=" " '{a[$0]++} END{for(k in a) sum++; print sum}' file
UPDATE:
It's probably better to do preprocessing with tr and let the awk do the counting economically. You may want to delimit the words with spaces or new lines.
For example:
$ tr ':;,?!\"' ' ' < file | tr -s ' ' '\n' | awk '!a[$0]++{c++} END{print c}'

Answer 2

You can rely on awk's default behavior to split lines into words by runs of whitespace, and use its associative arrays:
awk '{ for (i=1; i<=NF; ++i) a[tolower($i)]++ } END { print length(a) }' file
Update: As #rici points out in a comment, white-space separated tokens may include punctuation other than _ and other characters, and are thus not necessarily the same as grep's \w+ construct. #4ae1e1 therefore suggests using a field separator of along the lines of '[^[:alnum:]_]'. Note that this will result in each component of a hyphenated word to be counted separately; similarly, ' separates words.
awk -F '[^[:alnum:]_]+' '{ for (i=1; i<=NF; ++i) { a[tolower($i)]++ } }
END { print length(a) - ("" in a) }' file
Associative array a is built in a way that counts the occurrence of each distinct word encountered in the input, converted to lowercase first so as to ignore differences in case - if you do NOT want to ignore case differences, simply remove the tolower() call.
CAVEAT: It seems that Mawk and BSD Awk aren't locale-aware, so tolower() won't work properly with non-ASCII characters.
On having processed all words, the number of elements of a equals the number of unique words.
NOTE: The POSIX-compliant reformulation of print length(a) is: for (k in a) ++count; print count
The above will work with GNU Awk, Mawk (1.3.4+), and BSD Awk, even though it isn't strictly POSIX-compliant (POSIX defines the length function only for strings, not arrays).

Answer 3

An important feature of sorting is that it is locale-aware, and therefore much more expensive in any locale other than C. Since you don't really care about the order here, you might as well tell sort to ignore the locale by using LC_ALL=C sort -u -f. If your locale is set to something else, that will probably cut your execution time in half.
The original version of this answer suggested that you should only do this if you don't care about non-ascii characters. However, if you are using Gnu coreutils, it turns out that none of this stuff will work in UTF-8 locales. While gnu sort will do a locale-aware string comparison in any locale (using the strxfrm standard library function), sort -f only does locale-aware case-folding in single-byte locales. Gnu uniq -i has the same problem. And tr only translates single-byte characters (by design, afaik); in theory [:alpha:] is locale-aware, but only for characters representable as single bytes.
In short, if you want to use sort -u -f, you might as well specify the C locale. That is no less broken for non-English letters, but at least the breakage doesn't waste time.
Gnu awk's tolower() function does apparently work on multibyte locales. So check out one of the awk answers if you need this to work in a UTF-8 locale.