I have tried using bloom filter for performing membership tests. I wish to perform membership tests on 80 billion entries with only allowing around 100 collisions to happen i.e., only 100 entries can be given a false positive result.
I know this can be achieved by bloom filters but using the formulas of determining the number of bits required per entry and the number of hash functions given the allowed false positive rate. I figured I would end up using 270 GB of memory and 19 hash functions.
I also had a look at Cuckoo filter but its memory requirements don't match my requirements. My Requirements are as follows :
Use at max 6 bits per element
Use no more than 7-8 Hash Functions.
Could someone suggest me a probabilistic data structure other than the one's mentioned above that can help in achieving my requirements?
The issue with the number of hash functions is not really a problem - just pick a single hash function with many bits of output and divide up the bits as if they were from separate hash functions. Your real problem here is the false positive rate tradeoff with storage space.
You have said
I wish to perform membership tests on 80 billion entries with only
allowing around 100 collisions to happen i.e., only 100 entries can be
given a false positive result.
Entries that are in the map can, by definition, not be false positives. They are true positives.
The question then is "100 false positives taken over how many entries
that you intend to test?" If the answer is also, oddly enough, 80 billion, then you are asking for a false positive rate of around 100/80,000,000,000 = 1/800,000,000, which is less than 2^-29.
The minimum space for any approximate membership data structure like Bloom filters or cuckoo filters is n lg 1/ε bits, where n is the number of elements in the structure, lg is the logarithm base 2, and ε is the false positive rate. In other words, you need more than 29 bits per element to achieve a false positive rate like 100 per 80 billion. 6 bits per element will get you 1.56% false positive rate at best. That's 1.25 billion per 80 billion, or 100 per 6400.
As far as I know there are no known practical data structures that come close to achieving this. Bloom filters don't, for instance, because they use more than lg 1/ε bits per item. Cuckoo filters don't because they use at least two additional metadata bits per item and have a bits-per-item rate that scales with lg n.
Related
You have a bloom filter, and you want to measure the rate of false positives, practically (not theoretically).
How do you go about it?
Do you insert N elements into it and count the number of hash collisions and divide by N, and that's it?
Or do you insert N elements and then do membership tests for all other elements which were not inserted (which is infinite in general)?
Or something else?
Imagine that your calculations say you'll have a false positive rate of X when there are N items in the Bloom filter.
Generate 2N unique random keys. Place half of them into the Bloom filter. Now test with the other half. You know the keys are unique, so any "positive" hit you get will be a false positive.
Compare your experimental result with the computed result.
I am looking for a shuffle algorithm to shuffle a set of sequential numbers without buffering. Another way to state this is that I’m looking for a random sequence of unique numbers that have a given period.
Your typical Fisher–Yates shuffle needs to have each element all of the elements it is going to shuffle, so that isn’t going to work.
A Linear-Feedback Shift Register (LFSR) does what I want, but only works for periods that are powers-of-two less two. Here is an example of using a 4-bit LFSR to shuffle the numbers 1-14:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
8
12
14
7
4
10
5
11
6
3
2
1
9
13
The first two is the input, and the second row the output. What’s nice is that the state is very small—just the current index. You can start of any index and get a difference set of numbers (starting at 1 yields: 8, 12, 14; starting at 9: 6, 3, 2), although the sequence is always the same (5 is always followed by 11). If I want a different sequence, I can pick a different generator polynomial.
The limitations to the LFSR are that the periods are always power-of-two less two (the min and max are always the same, thus unshuffled) and there not enough enough generator polynomials to allow every possible random sequence.
A block cipher algorithm would work. Every key produces a uniquely shuffled set of numbers. However all block ciphers (that I know about) have power-of-two block sizes, and usually a fixed or limited number of block sizes. A block cipher with a arbitrary non-binary block size would be perfect if such a thing exists.
There are a couple of projects I have that could benefit from such an algorithm. One is for small embedded micros that need to produce a shuffled sequence of numbers with a period larger than the memory they have available (think Arduino Uno needing to shuffle 1 to 100,000).
Does such an algorithm exist? If not, what things might I search for to help me develop such an algorithm? Or is this simply not possible?
Edit 2022-01-30
I have received a lot of good feedback and I need to better explain what I am searching for.
In addition to the Arduino example, where memory is an issue, there is also the shuffle of a large number of records (billions to trillions). The desire is to have a shuffle applied to these records without needing a buffer to hold the shuffle order array, or the time needed to build that array.
I do not need an algorithm that could produce every possible permutation, but a large number of permutations. Something like a typical block cipher in counter mode where each key produces a unique sequence of values.
A Linear Congruential Generator using coefficients to produce the desired sequence period will only produce a single sequence. This is the same problem for a Linear Feedback Shift Register.
Format-Preserving Encryption (FPE), such as AES FFX, shows promise and is where I am currently focusing my attention. Additional feedback welcome.
It is certainly not possible to produce an algorithm which could potentially generate every possible sequence of length N with less than N (log2N - 1.45) bits of state, because there are N! possible sequence and each state can generate exactly one sequence. If your hypothetical Arduino application could produce every possible sequence of 100,000 numbers, it would require at least 1,516,705 bits of state, a bit more than 185Kib, which is probably more memory than you want to devote to the problem [Note 1].
That's also a lot more memory than you would need for the shuffle buffer; that's because the PRNG driving the shuffle algorithm also doesn't have enough state to come close to being able to generate every possible sequence. It can't generate more different sequences than the number of different possible states that it has.
So you have to make some compromise :-)
One simple algorithm is to start with some parametrisable generator which can produce non-repeating sequences for a large variety of block sizes. Then you just choose a block size which is as least as large as your target range but not "too much larger"; say, less than twice as large. Then you just select a subrange of the block size and start generating numbers. If the generated number is inside the subrange, you return its offset; if not, you throw it away and generate another number. If the generator's range is less than twice the desired range, then you will throw away less than half of the generated values and producing the next element in the sequence will be amortised O(1). In theory, it might take a long time to generate an individual value, but that's not very likely, and if you use a not-very-good PRNG like a linear congruential generator, you can make it very unlikely indeed by restricting the possible generator parameters.
For LCGs you have a couple of possibilities. You could use a power-of-two modulus, with an odd offset and a multiplier which is 5 mod 8 (and not too far from the square root of the block size), or you could use a prime modulus with almost arbitrary offset and multiplier. Using a prime modulus is computationally more expensive but the deficiencies of LCG are less apparent. Since you don't need to handle arbitrary primes, you can preselect a geometrically-spaced sample and compute the efficient division-by-multiplication algorithm for each one.
Since you're free to use any subrange of the generator's range, you have an additional potential parameter: the offset of the start of the subrange. (Or even offsets, since the subrange doesn't need to be contiguous.) You can also increase the apparent randomness by doing any bijective transformation (XOR/rotates are good, if you're using a power-of-two block size.)
Depending on your application, there are known algorithms to produce block ciphers for subword bit lengths [Note 2], which gives you another possible way to increase randomness and/or add some more bits to the generator state.
Notes
The approximation for the minimum number of states comes directly from Stirling's approximation for N!, but I computed the number of bits by using the commonly available lgamma function.
With about 30 seconds of googling, I found this paper on researchgate.net; I'm far from knowledgable enough in crypto to offer an opinion, but it looks credible; also, there are references to other algorithms in its footnotes.
I have a need to use a Cuckoo Filter but I'm not sure how to size it. I found a calculator for Bloom Filters (https://hur.st/bloomfilter/) for which I can calculate in a few ways. I can specify the approximate number of items and the desired false positive rate and it will tell me the size and number of hash functions. I'm looking for something similar for a Cuckoo Filter but I haven't found one or other instructions on how to find those numbers.
I'm looking at a Node or Python implementation. It seems the parameters to define the filter are:
filter size or capacity
bucket size
fingerprint size
I want to specify the number of elements (eg 100k) and an FPR (eg .1%) to find out the parameters needed.
Based on information in the original paper (https://www.cs.cmu.edu/~dga/papers/cuckoo-conext2014.pdf), you need to choose bucket size first, which allows you to determine fingerprint size and capacity. Bucket size is based on the desired false positive rate:
"the space-optimal bucket size depends on the target false positive
rate ε: when ε > 0.002, having two entries per bucket yields slightly
better results than using four entries per bucket; when ε decreases to
0.00001 < ε ≤ 0.002, four entries per bucket minimizes space"1
For your suggested 0.1%, that would mean a bucket size of 4.
The fingerprint size depends on bucket size and false positive rate.
"To retain the target false positive rate ε, the filter ensures 2b/2f
≤ ε, thus the minimal fingerprint size required is approximately: f ≥ log2(1/ε) + log2(2b)"1
With b bucket size, an error rate of 0.1% would require ~10 + 3 = 13 bits for a fingerprint.
Finally, capacity is determined by the number of elements divided by the maximum allowable load, which is determined by bucket size.
"With k = 2 hash functions, the load factor α is 50% when the bucket
size b = 1 (i.e., the hash table is directly mapped), but increases to
84%, 95% or 98% respectively using bucket size b = 2, 4 or 8."1
So 100k / 0.95 gives you a capacity of 106k.
I don't know of any one formula to give you these answers, since they depend on each other, but hopefully each of those steps makes sense.
For 100k elements and 0.1% FPR, that's:
filter size of 106k
bucket size of 4
fingerprint size of 13 bits
1 Bin Fan, Dave G. Andersen , Michael Kaminsky , Michael D. Mitzenmacher, Cuckoo Filter: Practically Better Than Bloom, Proceedings of the 10th ACM International on Conference on emerging Networking Experiments and Technologies, December 02-05, 2014, Sydney, Australia [doi>10.1145/2674005.2674994]
According to https://brilliant.org/wiki/cuckoo-filter/ (scroll down to "Space Complexity"), the number of bits per entry is determined by:
bitsPerEntry = (log(1/fpp)+2)/load
fpp is your False Positive Probability. load is how full you want the table to be.
So just figure out how many items you want to put in the table, multiply by the bitsPerEntry, and divide by 8. That will tell you how many bytes to allocate for your table. By applying some simple algebra, you can structure the equation to solve for any one of the unknowns.
The article says that with a load of 95.5%, you can maintain a stable false positive rate with 7 bits per entry.
The size of the fingerprint determines your error rate for the most part. As you can see in Figure 3 in the Cuckoo paper, the bucket size does not have a major effect on the accuracy. Bucket size can reduce insert time considerably since it reduces the number of relocations of existing fingerprints in occupied buckets.
I would recommend fingerprints 7, 15, 23, 31 etc' which will maximize both accuracy and speed. The reason for (8 * n) - 1 is, one bit is used to tell whether the cell is occupied at all since 0 is legal.
To answer your question, I would recommend
Capacity - what you need plus 5-10%
FingerPrint - 15 bits
Bucket size - 4
I want to compress an array consisting of about 10^5 random integers in range 0 to 2^15. The integers are unsorted and I need to compress them lossless.
I don't care much about the amount of computation and time needed to run the algorithm, just want to have better compression ratio.
Are there any suggested algorithms for this?
Assuming you don´t need to preserve original order, instead of passing the numbers themselves, pass the count. If they have a normal distribution, you can expect each number to be repeated 3 or 4 times. With 3 bits per number, we can count up to 7. You can make an array of 2^15 * 3 bits and every 3 bits set the count of that number. To handle extreme cases that have more than 7, we can also send a list of numbers and their counts for these cases. Then you can read the 3 bits array and overwrite with the additional info for count higher than 7.
For your exact example: just encode each number as a 15-bit unsigned int and apply bit packing. This is optimal since you have stated each integer in uniformly random in [0, 2^15), and the Shannon Entropy of this distribution is 15 bits.
For a more general solution, apply Quantile Compression (https://github.com/mwlon/quantile-compression/). It takes advantage of any smooth-ish data and compresses near optimally on shuffled data. It works by encoding each integer with a Huffman code for it coarse range in the distribution, then an exact offset within that range.
These approaches are both computationally cheap, but more compute won't get you further in this case.
I have studied the working of Sieve of Eratosthenes for generating the prime numbers up to a given number using iteration and striking off all the composite numbers. And the algorithm needs just to be iterated up to sqrt(n) where n is the upper bound upto which we need to find all the prime numbers. We know that the number of prime numbers up to n=10^9 is very less as compared to the number of composite numbers. So we use all the space to just tell that these numbers are not prime first by marking them composite.
My question is can we modify the algorithm to just store prime numbers since we deal with a very large range (since number of primes are very less)?
Can we just store straight away the prime numbers?
Changing the structure from that of a set (sieve) - one bit per candidate - to storing primes (e.g. in a list, vector or tree structure) actually increases storage requirements.
Example: there are 203.280.221 primes below 2^32. An array of uint32_t of that size requires about 775 MiB whereas the corresponding bitmap (a.k.a. set representation) occupies only 512 MiB (2^32 bits / 8 bits/byte = 2^29 bytes).
The most compact number-based representation with fixed cell size would be storing the halved distance between consecutive odd primes, since up to about 2^40 the halved distance fits into a byte. At 193 MiB for the primes up to 2^32 this is slightly smaller than an odds-only bitmap but it is only efficient for sequential processing. For sieving it is not suitable because, as Anatolijs has pointed out, algorithms like the Sieve of Eratosthenes effectively require a set representation.
The bitmap can be shrunk drastically by leaving out the multiples of small primes. Most famous is the odds-only representation that leaves out the number 2 and its multiples; this halves the space requirement to 256 MiB at virtually no cost in added code complexity. You just need to remember to pull the number 2 out of thin air when needed, since it isn't represented in the sieve.
Even more space can be saved by leaving out multiples of more small primes; this generalisation of the 'odds-only' trick is usually called 'wheeled storage' (see Wheel Factorization in the Wikipedia). However, the gain from adding more small primes to the wheel gets smaller and smaller whereas the wheel modulus ('circumference') increases explosively. Adding 3 removes 1/3rd of the remaining numbers, adding 5 removes a further 1/5th, adding 7 only gets you a further 1/7th and so on.
Here's an overview of what adding another prime to the wheel can get you. 'ratio' is the size of the wheeled/reduced set relative to the full set that represents every number; 'delta' gives the shrinkage compared to the previous step. 'spokes' refers to the number of prime-bearing spokes which need to be represented/stored; the total number of spokes for a wheel is of course equal to its modulus (circumference).
The mod 30 wheel (about 136 MiB for the primes up to 2^32) offers an excellent cost/benefit ratio because it has eight prime-bearing spokes, which means that there is a one-to-one correspondence between wheels and 8-bit bytes. This enables many efficient implementation tricks. However, its cost in added code complexity is considerable despite this fortuitous circumstance, and for many purposes the odds-only sieve ('mod 2 wheel') gives the most bang for buck by far.
There are two additional considerations worth keeping in mind. The first is that data sizes like these often exceed the capacity of memory caches by a wide margin, so that programs can often spend a lot of time waiting for the memory system to deliver the data. This is compounded by the typical access patterns of sieving - striding over the whole range, again and again and again. Speedups of several orders of magnitude are possible by working the data in small batches that fit into the level-1 data cache of the processor (typically 32 KiB); lesser speedups are still possible by keeping within the capacity of the L2 and L3 caches (a few hundred KiB and a few MiB, respectively). The keyword here is 'segmented sieving'.
The second consideration is that many sieving tasks - like the famous SPOJ PRIME1 and its updated version PRINT (with extended bounds and tightened time limit) - require only the small factor primes up to the square root of the upper limit to be permanently available for direct access. That's a comparatively small number: 3512 when sieving up to 2^31 as in the case of PRINT.
Since these primes have already been sieved there's no need for a set representation anymore, and since they are few there are no problems with storage space. This means they are most profitably kept as actual numbers in a vector or list for easy iteration, perhaps with additional, auxiliary data like current working offset and phase. The actual sieving task is then easily accomplished via a technique called 'windowed sieving'. In the case of PRIME1 and PRINT this can be several orders of magnitude faster than sieving the whole range up to the upper limit, since both tasks only asks for a small number of subranges to be sieved.
You can do that (remove numbers that are detected to be non-prime from your array/linked list), but then time complexity of algorithm will degrade to O(N^2/log(N)) or something like that, instead of original O(N*log(N)). This is because you will not be able to say "the numbers 2X, 3X, 4X, ..." are not prime anymore. You will have to loop through your entire compressed list.
You could erase each composite number from the array/vector once you have shown it to be composite. Or when you fill an array of numbers to put through the sieve, remove all even numbers (other than 2) and all numbers ending in 5.
If you have studied the sieve right, you must know we don't have the primes to begin with. We have an array, sizeof which is equal to the range. Now, if you want the range to be 10e9, you want this to be the size of the array. You have mentioned no language, but for each number, you must need a bit to represent whether it is prime or not.
Even that means you need 10^9 bits = 1.125 * 10^8 bytes which is greater than 100 MB of RAM.
Assuming you have all this, most optimized sieve takes O(n * log(log n)) time, which is, if n = 10e9, on a machine that evaluates 10e8 instructions per second, will still take some minutes.
Now, assuming you have all this with you, still, number of primes till 10e9 is q = 50,847,534, to save these will still take q * 4 bytes, which is still greater than 100MB. (more RAM)
Even if you remove the indexes which are multiples of 2, 3 or 5, this removes 21 numbers in every 30. This is not good enough, because in total, you will still need around 140 MB space. (40MB = a third of 10^9 bits + ~100MB for storing prime numbers).
So, since, for storing the primes, you will, in any case require similar amount of memory (of the same order as calculation), your question, IMO has no solution.
You can halve the size of the sieve by only 'storing' the odd numbers. This requires code to explicitly deal with the case of testing even numbers. For odd numbers, bit b of the sieve represents n = 2b + 3. Hence bit 0 represents 3, bit 1 represents 5 and so on. There is a small overhead in converting between the number n and the bit index b.
Whether this technique is any use to you depends on the memory/speed balance you require.