Pre-processing data for existing compressor - algorithm

Question

I have an existing "compression" algorithm, which compresses an association into ranges. Something like
type Assoc = [(P,C)]
type RangeCompress :: Assoc -> [(P, [(C,C)]]
Read P as "product" and C as "code". The result is a list of products each associated with a list of code-ranges. To find the product associated with a given code, one traverses the compressed data until one finds a range, the given code falls into.
This mechanism works well if consecutive codes are likely to belong to the same product. However, if the codes of different product interleave, they no longer form compact ranges and I end up with lots of ranges, where the upper bound is equal to the lower bound and nil compression.
What I am looking for is a pre-compressor, which looks at the original association and determines a "good enough" transformation of codes, such that the association expressed in terms of transformed codes can be range-compressed into compact ranges. Something like
preCompress :: Assoc -> (C->C)
or more fine-grained (by Product)
preCompress ::Assoc -> P -> (C->C)
In that case a product lookup would have to first transform the code in question and then do the lookup as before. Therefore the transformation must be expressible by a handful of parameters, which would have to be attached to the compressed data, either once for the entire association or by product.
I checked some compression algorithms, but they all seem to focus on reconstructing the original data (which is not strictly needed here), while being totally free about the way they store the compressed data. In my case however, the compressed data must be ranges, only enriched by the parameters of the pre-compression.
Is this a known problem?
Does it have a solution?
Where to look next?
Please note:
I am not primarily interested in restoring the original data
I am primarily interested in the product lookup
The number of codes is apx 7,000,000
The number of products is apx 200

Answer 1

Assuming that for each code there is only one product, you can encode the data as a list (string) of products. Let's pretend we have the following list of products and codes randomly generated from 9 products (or no product) and 10 codes.
[(P6,C2),
(P1,C4),
(P2,C10),
(P3,C9),
(P3,C1),
(P4,C7),
(P6,C8),
(P5,C3),
(P1,C5)]
If we sort them by code we have
[(P3,C1),
(P6,C2),
(P5,C3),
(P1,C4),
(P1,C5),
-- C6 didn't have a product
(P4,C7),
(P6,C8),
(P3,C9),
(P2,C10)]
We can convert this into a string of products + nothing (N). The position in the string determines the code.
{- C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 -}
[ P3, P6, P5, P1, P1, N , P4, P6, P3, P2 ]
A string of symbols in some alphabet (in this case products+nothing) puts us squarely in the realm of well-studied string compression problems.
If we run-length encode this list, we have an encoding similar to the encoding you originally presented. For each range of associations we store a single product+nothing and the run length. We only need a single small integer for the run length instead of two (possibly large) codes for the interval.
{- P3 , P6, P5, P1, P1, N , P4, P6, P3, P2 -}
[ (P3, 1), (P6, 1), (P5, 1), (P1, 2), (N, 1), (P4, 1), (P6, 1), (P3, 1), (P2, 1) ]
We can serialize this to a string of bytes and use any of the existing compression libraries on bytes to perform the actual compression. Some compression libraries, such as the frequently used zlib, are found in the codec category.
Sorted the other way
We'll take the same data from before and sort it by product instead of by code
[(P1,C4),
(P1,C5),
(P2,C10),
(P3,C1),
(P3,C9),
(P4,C7),
(P5,C3),
(P6,C2),
(P6,C8)]
We'd like to allocate new codes to each product so that the codes for a product are always consecutive. We'll just allocate these codes in order.
-- v-- New code --v v -- Old code
[(P1,C1), [(C1,C4)
(P1,C2), (C2,C5),
(P2,C3), (C3,C10),
(P3,C4), (C4,C1),
(P3,C5), (C5,C9),
(P4,C6), (C6,C7),
(P5,C7), (C7,C3),
(P6,C8), (C8,C2),
(P6,C9), (C9,C8)]
We now have two pieces of data to save. We have a (now) one-to-one mapping between products and code ranges and a new one-to-one mapping between new codes and old codes. If we follow the steps from the previous section, we can convert the mapping between new codes and old codes into a list of old codes. The position in the list determines the new code.
{- C1 C2 C3 C4 C5 C6 C7 C8 C9 -} -- new codes
[ C4, C5, C10, C1, C9, C7, C3, C2, C8 ] -- old codes
We can throw any of our existing compression algorithms at this string. Each symbol only occurs at most once in this list so traditional compression mechanisms will not compress this list any more. This isn't significantly different from grouping by the product and storing the list of codes as an array with the product; the new intermediate codes are just (larger) pointers to the start of the array and the length of the array.
[(P1,[C4,C5]),
(P2,[C10]),
(P3,[C1,C9]),
(P4,[C7]),
(P5,[C3]),
(P6,[C2,C8])]
A better representation for the list of old codes is probably the difference between old codes. If the codes already tend to be in consecutive ranges these consecutive ranges can be run-length encoded away.
{- C4, C5, C10, C1, C9, C7, C3, C2, C8 -} -- old codes
[ +4, +1, +5, -9, +8, -2, -4, -1, +6 ] -- old code difference
I might be tempted to store the difference in product and the difference in codes with the products. This should increase the opportunity for common sub-strings for the final compression algorithm to compress away.
[(+1,[+4,+1]),
(+1,[+5]),
(+1,[-9,+8]),
(+1,[-2]),
(+1,[-4]),
(+1,[-1,+6])]