How to efficiently store a matrix with highly-redundant values - algorithm

Question

I have a very large matrix (100M rows by 100M columns) that has a lots of duplicate values right next to each other. For example:
8 8 8 8 8 8 8 8 8 8 8 8 8
8 4 8 8 1 1 1 1 1 8 8 8 8
8 4 8 8 1 1 1 1 1 8 8 8 8
8 4 8 8 1 1 1 1 1 8 8 8 8
8 4 8 8 1 1 1 1 1 8 8 8 8
8 4 8 8 1 1 1 1 1 8 8 8 8
8 8 8 8 8 8 8 8 8 8 8 8 8
8 8 3 3 3 3 3 3 3 3 3 3 3
I want a datastructure/algorithm to store matricies like these as compactly as possible. For instance, the matrix above should only take O(1) space (even if the matrix was stretched out arbitrarily big), because there is only a constant number of rectangular regions, where each region only has one value.
The repetition happens both across rows and down columns, so the simple approach of compressing the matrix row-by-row isn't good enough. (That would require a minimum of O(num_rows) space to store any matrix.)
The representation of the matrix also needs to accessible row-by-row, so that I can do a matrix multiplication to a column vector.

Answer 1

You could store the matrix as a quadtree with the leaves containing single values. Think of this as a two-dimensional "run" of values.

Answer 2

Now for my preferred method.
Ok, as I made mention in my previous answer rows with the same entries in each column in matrix A will multiply out to the same result in matrix AB. If we can maintain that relationship then we can theoretically speed up calculations significantly (a profiler is your friend).
In this method we maintain the row * column structure of the matrix.
Each row is compressed with whatever method can decompress fast enough not to affect the multiplication speed too much. RLE may be sufficient.
We now have a list of compressed rows.
We use an entropy encoding method (like Shannon-Fano, Huffman or arithmetic coding), but we don’t compress the data in the rows with this, we use it to compress the set of rows.
We use it to encode the relative frequency of the rows. I.e. we treat a row the same way standard entropy encoding would treat a character/byte.
In this example RLE compresses a row, and Huffman compresses the entire set of rows.
So, for example, given the following matrix (prefixed with row numbers, Huffman used for ease of explanation)
0 | 8 8 8 8 8 8 8 8 8 8 8 8 8 |
1 | 8 4 8 8 1 1 1 1 1 8 8 8 8 |
2 | 8 4 8 8 1 1 1 1 1 8 8 8 8 |
3 | 8 4 8 8 1 1 1 1 1 8 8 8 8 |
4 | 8 4 8 8 1 1 1 1 1 8 8 8 8 |
5 | 8 4 8 8 1 1 1 1 1 8 8 8 8 |
6 | 8 8 8 8 8 8 8 8 8 8 8 8 8 |
7 | 8 8 3 3 3 3 3 3 3 3 3 3 3 |
Run length encoded
0 | 8{13} |
1 | 8{1} 4{1} 8{2} 1{5} 8{4} |
2 | 8{1} 4{1} 8{2} 1{5} 8{4} |
3 | 8{1} 4{1} 8{2} 1{5} 8{4} |
4 | 8{1} 4{1} 8{2} 1{5} 8{4} |
5 | 8{1} 4{1} 8{2} 1{5} 8{4} |
6 | 8{13} |
7 | 8{2} 3{11} |
So, 0 and 6 appear twice and 1 – 5 appear 5 times. 7 only once.
Frequency table
A: 5 (1-5) | 8{1} 4{1} 8{2} 1{5} 8{4} |
B: 2 (0,6) | 8{13} |
C: 1 7 | 8{2} 3{11} |
Huffman tree
0|1
/ \
A 0|1
/ \
B C
So in this case it takes one bit (for each row) to encode rows 1 – 5, and 2 bits to encode rows 0, 6, and 7.
(If the runs are longer than a few bytes then do freq count on a hash that you build up as you do the RLE).
You store the Huffman tree, unique strings, and the row encoding bit stream.
The nice thing about Huffman is that it has a unique prefix property, so you always know when you are done. Thus, given the bit string 10000001011 you can rebuild the matrix A from the stored unique strings and the tree. The encoded bit stream tells you the order that the rows appear in.
You may want to look into adaptive Huffman encoding, or its arithmetic counterpart.
Seeing as rows in A with the same column entries multiply to the same result in AB over vector B you can cache the result and use it instead of calculating it again (it’s always good to avoid 100M*100M multiplications if you can).
Links to further info:
Arithmetic Coding + Statistical Modeling = Data Compression
Priority Queues and the STL
Arithmetic coding
Huffman coding
A Comparison
Uncompressed
0 1 2 3 4 5 6 7
=================================
0 | 3 3 3 3 3 3 3 3 |
|-------+ +-------|
1 | 4 4 | 3 3 3 3 | 4 4 |
| +-----------+---+ |
2 | 4 4 | 5 5 5 | 1 | 4 4 |
| | | | |
3 | 4 4 | 5 5 5 | 1 | 4 4 |
|---+---| | | |
4 | 5 | 0 | 5 5 5 | 1 | 4 4 |
| | +---+-------+---+-------|
5 | 5 | 0 0 | 2 2 2 2 2 |
| | | |
6 | 5 | 0 0 | 2 2 2 2 2 |
| | +-------------------|
7 | 5 | 0 0 0 0 0 0 0 |
=================================
= 64 bytes
Quadtree
0 1 2 3 4 5 6 7
=================================
0 | 3 | 3 | | | 3 | 3 |
|---+---| 3 | 3 |---+---|
1 | 4 | 4 | | | 4 | 4 |
|-------+-------|-------+-------|
2 | | | 5 | 1 | |
| 4 | 5 |---+---| 4 |
3 | | | 5 | 1 | |
|---------------+---------------|
4 | 5 | 0 | 5 | 5 | 5 | 1 | 4 | 4 |
|---+---|---+---|---+---|---+---|
5 | 5 | 0 | 0 | 2 | 2 | 2 | 2 | 2 |
|-------+-------|-------+-------|
6 | 5 | 0 | 0 | 2 | 2 | 2 | 2 | 2 |
|---+---+---+---|---+---+---+---|
7 | 5 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
=================================
0 +- 0 +- 0 -> 3
| +- 1 -> 3
| +- 2 -> 4
| +- 3 -> 4
+- 1 -> 3
+- 2 -> 4
+- 3 -> 5
1 +- 0 -> 3
+- 1 +- 0 -> 3
| +- 1 -> 3
| +- 2 -> 4
| +- 3 -> 4
+- 2 +- 0 -> 5
| +- 1 -> 1
| +- 2 -> 5
| +- 3 -> 1
+- 3 -> 4
2 +- 0 +- 0 -> 5
| +- 1 -> 0
| +- 2 -> 5
| +- 3 -> 0
+- 1 +- 0 -> 5
| +- 1 -> 5
| +- 2 -> 0
| +- 3 -> 2
+- 2 +- 0 -> 5
| +- 1 -> 0
| +- 2 -> 5
| +- 3 -> 0
+- 3 +- 0 -> 0
+- 1 -> 2
+- 2 -> 0
+- 3 -> 0
3 +- 0 +- 0 -> 5
| +- 1 -> 1
| +- 2 -> 2
| +- 3 -> 2
+- 1 +- 0 -> 4
| +- 1 -> 4
| +- 2 -> 2
| +- 3 -> 2
+- 2 +- 0 -> 2
| +- 1 -> 2
| +- 2 -> 0
| +- 3 -> 0
+- 3 +- 0 -> 2
+- 1 -> 2
+- 2 -> 0
+- 3 -> 0
((1*4) + 3) + ((2*4) + 2) + (4 * 8) = 49 leaf nodes
49 * (2 + 1) = 147 (2 * 8 bit indexer, 1 byte data)
+ 14 inner nodes -> 2 * 14 bytes (2 * 8 bit indexers)
= 175 Bytes
Region Hash
0 1 2 3 4 5 6 7
=================================
0 | 3 3 3 3 3 3 3 3 |
|-------+---------------+-------|
1 | 4 4 | 3 3 3 3 | 4 4 |
| +-----------+---+ |
2 | 4 4 | 5 5 5 | 1 | 4 4 |
| | | | |
3 | 4 4 | 5 5 5 | 1 | 4 4 |
|---+---| | | |
4 | 5 | 0 | 5 5 5 | 1 | 4 4 |
| + - +---+-------+---+-------|
5 | 5 | 0 0 | 2 2 2 2 2 |
| | | |
6 | 5 | 0 0 | 2 2 2 2 2 |
| +-------+-------------------|
7 | 5 | 0 0 0 0 0 0 0 |
=================================
0: (4,1; 4,1), (5,1; 6,2), (7,1; 7,7) | 3
1: (2,5; 4,5) | 1
2: (5,3; 6,7) | 1
3: (0,0; 0,7), (1,2; 1,5) | 2
4: (1,0; 3,1), (1,6; 4,7) | 2
5: (2,2; 4,4), (4,0; 7,0) | 2
Regions: (3 + 1 + 1 + 2 + 2 + 2) * 5
= 55 bytes {4 bytes rectangle, 1 byte data)
{Lookup table is a sorted array, so it does not need extra storage}.
Huffman encoded RLE
0 | 3 {8} | 1
1 | 4 {2} | 3 {4} | 4 {2} | 2
2,3 | 4 {2} | 5 {3} | 1 {1} | 4 {2} | 4
4 | 5 {1} | 0 {1} | 5 {3} | 1 {1} | 4 {2} | 5
5,6 | 5 {1} | 0 {2} | 2 {5} | 3
7 | 5 {1} | 0 {7} | 2
RLE Data: (1 + 3+ 4 + 5 + 3 + 2) * 2 = 36
Bit Stream: 20 bits packed into 3 bytes = 3
Huffman Tree: 10 nodes * 3 = 30
= 69 Bytes
One Giant RLE stream
3{8};4{2};3{4};4{4};5{3};1{1};4{4};5{3};1{1};4{2};5{1};0{1};
5{3};1{1};4{2};5{1};0{2};2{5};5{1};0{2};2{5};5{1};0{7}
= 2 * 23 = 46 Bytes
One Giant RLE stream encoded with common prefix folding
3{8};
4{2};3{4};
4{4};5{3};1{1};
4{4};5{3};
1{1};4{2};5{1};0{1};5{3};
1{1};4{2};5{1};0{2};2{5};
5{1};0{2};2{5};
5{1};0{7}
0 + 0 -> 3{8};4{2};3{4};
+ 1 -> 4{4};5{3};1{1};
1 + 0 -> 4{2};5{1} + 0 -> 0{1};5{3};1{1};
| + 1 -> 0{2}
|
+ 1 -> 2{5};5{1} + 0 -> 0{2};
+ 1 -> 0{7}
3{8};4{2};3{4} | 00
4{4};5{3};1{1} | 01
4{4};5{3};1{1} | 01
4{2};5{1};0{1};5{3};1{1} | 100
4{2};5{1};0{2} | 101
2{5};5{1};0{2} | 110
2{5};5{1};0{7} | 111
Bit stream: 000101100101110111
RLE Data: 16 * 2 = 32
Tree: : 5 * 2 = 10
Bit stream: 18 bits in 3 bytes = 3
= 45 bytes

Answer 3

If your data is really regular, you might benefit from storing it in a structured format; e.g. your example matrix might be stored as the following list of "fill-rectangle" instructions:
(0,0)-(13,7) = 8
(4,1)-(8,5) = 1
(Then to look up the value of a particular cell, you'd iterate backwards through the list until you found a rectangle that contained that cell)

Answer 4

As Ira Baxter suggested,
you could store the matrix as a quadtree with the leaves containing single values.
The simplest way to do this is for every node of the quadtree to cover an area 2^n x 2^n,
and each non-leaf node points to its 4 children of size 2^(n-1) x 2^(n-1).
You might get slightly better compression with an adaptive quadtree that allows irregular sub-division.
Then each non-leaf node stores the cut-point (B,G) and points to its 4 children.
For example, if some non-leaf node covers an area from (A,F) in the upper-left corner to (C,H) in the lower-right corner,
then its 4 children cover areas
(A,F) to (B-1, G-1)
(A,G) to (B-1, H)
(B,F) to (C,G-1)
(B,G) to (C,H).
You would try to pick the (B,G) cut-point for each non-leaf node such that it lines up with some real division in your data.
For example, say you have a matrix with a small square in the middle filled with nines and zero elsewhere.
With the simple powers-of-two quadtree, you'll end up with at least 21 nodes: 5 non-leaf nodes, 4 leaf nodes of nines, and 12 leaf nodes of zeros.
(You'll get even more nodes if the centered small square is not precisely some power-of-two distance from the left and top edges, and not itself some precise power-of-two).
With an adaptive quadtree, if you are smart enough to pick the cut-point for the root node at the upper-left corner of that square, then for the root's lower-right child you pick a cut-point at the lower-right corner of the square, you can representing the entire matrix in 9 nodes: 2 non-leaf nodes, 1 leaf node for the nines, and 6 leaf nodes for the zeros.

Answer 5

Do you know about.... interval trees ?
Interval trees are a way to store intervals efficiently, and then query them. A generalization is the Range Tree, which can be adapted to any dimension.
Here you could effectively describe your rectangles and attach a value to them. Of course the rectangles can overlap, that's what will make it efficient.
0,0-n,n --> 8
4,4-7,7 --> 1
8,8-8,n --> 3
Then when querying for a value in one particular spot, you are returned a list of several rectangles and need to determine the innermost one: this is the value in this spot.

Answer 6

The simplest approach is to use run-length encoding on one dimension and not worry about the other dimension.
(If the dataset weren't so incredibly huge, interpreting it as an image and using a standard lossless image compression method would be very simple also--but since you'd have to work on making the algorithm work on sparse matrices, it wouldn't end up being all that simple.)
Another simple approach is to try a rectangular flood fill--start at the top-right pixel and increase it into the largest rectangle you can (breadth-first); then mark all those pixels as "done" and take the top-right most remaining pixel, repeat until done. (You'd probably want to store these rectangles in some sort of BSP or quad-tree.)
A highly effective technique--not optimal, but probably good enough--is to use a binary space partitioning tree where "space" is measured not spatially but by number of changes. You'd recursively cut so that you have equal numbers of changes on the left and right (or top and bottom--presumably you'd want to keep things square) and, as your sizes got smaller, so that you would cut as many changes as possible. Eventually, you'll end up cutting two rectangles apart from each other, each of which has all the same number; then stop. (Encoding by RLE in x and y will quickly tell you where the change points are.)

Answer 7

Your description of O(1) space for a matrix of size 100M x 100M is confusing. When you have a finite matrix, then your size is a constant (unless the program that generates the matrix doesn't alter it). So the amount of space required to store is also a constant even if you multiply it with a scalar. Definitely the time to read and write the matrix is not going to be O(1).
Sparse matrix is what I could think of to reduce the amount of space required to store such a matrix. You can write this sparse matrix to a file and store it as a tar.gz which will further compress the data.
I do have a question what does M in 100M denote? Does it mean Megabyte/million? If yes, this matrix size will be 100 x 10^6 x 100 x 10^6 bytes = 10^16 / 10^6 MB = 10^10/10^6 TB = 10^4 TB!!! What kind of a machine are you using?

Answer 8

I'm not sure why this question was made Community Wiki, but so it goes.
I'll rely on the assumption that you have a linear algebra application, and that your matrix has a rectangular type of redundancy. If so, then you can do something much better than quadtrees, and cleaner than cutting the matrix into rectangles (which is generally the right idea).
Let M be your matrix, let v be the vector that you want to multiply by M, and let
A be the special matrix
A = [1 -1 0 0 0]
[0 1 -1 0 0]
[0 0 1 -1 0]
[0 0 0 1 -1]
[0 0 0 0 1]
You'll also need the inverse matrix to A, which I'll call B:
B = [1 1 1 1 1]
[0 1 1 1 1]
[0 0 1 1 1]
[0 0 0 1 1]
[0 0 0 0 1]
Multiplying a vector v by A is fast and easy: You just take differences of consecutive pairs of elements of v. Multiply a vector v by B is also fast and easy: The entries of Bv are partial sums of the elements of v. Then you want to use the equation
Mv = B AMA B v
The matrix AMA is sparse: In the middle, each entry is an alternating sum of 4 entries of M that make a 2 x 2 square. You have to be at a corner of one of the rectangles in M for this alternating sum to be non-zero. Since AMA is sparse, you can store its non-zero entries in an associative array and use sparse matrix multiplication to apply it to a vector.

Answer 9

I do not have a specific answer for the matrix you have shown. In finite element analysis (FEA), you have matrices with redundant data. In implementing a FEA package in my under grad project, I used skyline storage method.
Some links:
Intel page for sparse matrix storage
Wikipedia link

Answer 10

The first thing to try is always the existing libraries and solutions. It is a lot of work getting custom formats working with all the operations you're going to want in the end. Sparse matrices is an old problem, so make sure you read up on the existing stuff.
Assuming you don't find something suitable, I would recommend a row-based format. Don't try to be too fancy with super-compact representations, you will end up with lots of processing needed for every little operation and bugs in your code. Instead try to compress each row separately. You know you are going to have to scan through each row for the matrix-vector multiplication, make life easy for yourself.
I would start with run-length-encoding, see how that works first. Once that is working, try adding some tricks like references to sections of the previous row. So a row might be encoded as: 126 zeros, 8 ones, 1000 entries copied directly from row above, 32 zeros. That seems like it might be very efficient with your given example.

Answer 11

Many of the above solutions are fine.
If you are working with a file consider file oriented
compression tools like compress, bzip, zip, bzip2 and friends.
They work very well especially if the data contains redundant
ASCII characters. Using an external compression tool eliminates
problems and challenges inside your code and will compress
both binary and ASCII data.
In your example you are displaying one character numbers.
The numbers 0-9 can be represented by a smaller four bit
encoding pattern. You can use the additional bits in
a byte as a count. Four bits gives you extra codes to
escape to extras... But there is a caution which reaches
back to the old Y2K bugs where two characters were used
for a year. Byte encoding from an ofset would have given
255 years and the same two bytes would span all of written
history and then some.

Answer 12

You may want to take a look at GIF format and its compression algorithm. Just think about your matrix as a Bitmap...

Answer 13

Let me check my assumptions, if for no other reason than to guide my thinking about the problem:
The matrix is highly redundant, not necessarily sparse.
We want to minimize storage (on disk and RAM).
We want to be able to multiply A[m*n] by vector B[n*1] to get to AB[m*1] without first decompressing either (at least not more than required to do the calculations).
We don’t need random access to any A[i*j] entry --all operations are over the matrix.
The multiplication is done online (as needed), and so must be as efficient as possible.
The matrix is static.
One can try all kinds of clever schemes to detect rectangles or self similarity etc, but that is going to end up hurting performance when doing the multiplication. I propose 2 relatively simple solutions.
I am going to have to work backwards a bit, so please be patient with me.
If the data is predominantly biased towards horizontal repetition then the following may work well.
Think of the matrix flattened into an array (this is really the way it is stored in memory anyway). E.g.
A
| w0 w1 w2 |
| x0 x1 x2 |
| y0 y1 y2 |
| z0 z1 z2 |
becomes
A’
| w0 w1 w2 x0 x1 x2 y0 y1 y2 z0 z1 z2 |
We can use the fact that any index [i,j] = i * j.
So, when we do the multiplication we iterate over the “matrix” array A’ with k = [0..m*n-1] and index into the vector B using (k mod n) and into vector AB with (k div n). “div” being integer division.
So, for example, A[10] = z1. 10 mod 3 = 1 and 10 div 3 = 3 A[3,1] = z1.
Now, on to the compression.
We do normal run of the mill Run Length Encoding (RLE), but against the A’, not A. With the flat array there will be longer sequences of repetition, hence better compression. Then after encoding the runs we do another process where we extract common substrings. We can either do a form of dictionary compression, or process the run data into some form of space optimized graph like a radix tree/suffix tree or a device of your own creation that merges tops and tails. The graph should have a representation of all the unique strings in the data. You can pick any number of methods to break the stream into strings: matching prefixes, length, or something else (whatever suits your graph best) but do it on a run boundary, not bytes or your decoding will be made more complicated. The graph becomes a state machine when we decompress the stream.
I’m going to use a bit stream and Patricia trie as an example, because it is simplest, but you can use something else (more bits per state change better merging, etc. Look for papers by Stefan Nilsson).
To compress the run data we build a hash table against the graph. The table maps a string to a bit sequence. You can do this by walking the graph and encoding each left branch as 0 and right branch as 1 (arbitrary choice).
Process the run data and build up a bit string until you get a match in the hash table, output the bits and clear the string (the bits will not be on a byte boundary, so you may have to buffer until you get a sequence long enough to write out). Rinse and repeat until you have processed the complete run data stream. You store the graph and the bit stream. The bit stream encodes strings, not bytes.
If you reverse the process, using the bit stream to walk the graph until you reach a leaf/terminal node you get back the original run data, which you can decode on the fly to produce the stream of integers that you multiply against the vector B to get AB. Each time you run out of runs you read the next bit and lookup its corresponding string. We don’t care that we don’t have random access into A, because we only need it in B (B which can be range / interval compressed but doesn’t need to be).
So even though RLE is biased towards horizontal runs we still get good vertical compression because common strings are stored only once.
I will explain the other method in a separate answer as this is getting too long as it is, but that method can actually speed up calculation due to the fact that repeat rows in matrix A multiplies to the same result in AB.

Answer 14

ok you need a compression algorithm try RLE (Run Length Encoding) its work very good when the data is
highly-redundant .