Iam working on a FPGA implementation of MJPEG Encoder which uses standard quantisation and huffman tables.
Question:
Can we generate Huffman tables on fly ?.
I mean that once a frame is given as input, a block in the FPGA should generate corresponding huffman tables for the frame.
The block can though relax for sometime because a few upcoming frames will be similiar ones.
I have already gone through vitter algorithm (dynamic huffman coding). But I doubt it would be hardware efficient.
It will be helpful if I get some more ideas from people around or from those who have implemented such things already on FPGA.
Something like this :
tables = huffmantables(frame)
// wait for some frames
Thanks in advance !!
My advice is: you don't. Use standard Huffman table instead, they may not be optimal, but they are not that bad either.
Building Huffman table is a task for a processor. Unless an hardware-friendly algorithm for it exist, putting a soft-core microcontroller to build Huffman tables may be the most efficient implementation.
I suggest you do some testing to see if you really need it. How many dB of PSNR are you gaining from using optimal tables? Or how many more compression ratio? I found that standard table were good enough for generic MJPEG and chose for the smaller core over the complicated optimal one.
Related
I am trying to implement a JPEG encoder using the lowest possible operations. It is relatively doable up to Huffman encoding where most tutorials are using pointers and binary trees to make the table and encode the image. Can someone more familiar with the JPEG standard point me in the direction of the simplest compression technique that I should try to implement with low level operations (+, -, *, shifts, loops, if statements).
I have heard there are standard Huffman tables (I can't really find one), are these typically a good idea to use? After I use a standard huffman table, how simple is it to encode a 8x8 chunk with it? I stopped here, because I didn't want to go down a rabbit hole.
The standard will answer all of your questions. There are "typical" Huffman codes provided there in appendix K.3, along with the specifications for those tables. You can just hard-code those into your implementation and you will get good performance.
Since the Huffman codes are given, you would not need to implement the Huffman algorithm, which is what needs the pointers and trees you are referring to. (Not terribly complicated, so it is something you should tackle later, once you get the pre-defined codes working.)
You can just use the code table and a set of operators, which also need to include bitwise or (|) and bitwise and (&) along with the others in your list. The encoding process is simple, which is to have a bit buffer in an integer into which you accumulate code bits using the shift and or operators, and then pulling bytes from the buffer to write to the output when there are eight or more bits in the buffer.
Currently using the run length encoding for encoding bit-vectors, and the current run time is
2log(i), where is the size of the run. Is there another way of doing it to bring it down to log(i)?
Thanks.
The most efficient way of encoding a bit vector is to isolate any specific properties of the bit source. If it is totally random, there is no real noticeable gain (actually, a totally random stream of bit cannot be compressed in any way).
If you can find properties in your bit stream you could try to define a collection of vectors which will define the base of a Vector Space. In such case, the result will be very efficient.
We'll need a few more details on your bit stream.
(Edit)
Just a few more details to understand the previous statement:
"a totally random stream of bits cannot be compressed in any way"
It is not possible to compress a totally random vector of bits if by "compress" we mean the "transformed/compressed stream" plus the "vector base definition" plus the decompression program. But in most cases the decompression program (and often the vector base too) is embedded in client software. Thus, only the "compressed stream" is needed.
A good explanation (and funny story) about that is Patrick Craig 5000$ compression challenge
More scientific the theory of information, especially entropy section
And, the final one, the full story.
But whatever the solution is, if you have an unknown number of unknown streams to compress you won't be ale to do anything. You have to find a pattern.
I'm looking for compression/deompression algorithms that can give decent compression 2-4x on regular english text and yet I can decompress this data almost as fast as I can get it out of main memory (~10Gbps). Whats the current state of the art in terms of fast decompression algorithms (perhaps vectorized code that uses multiple cores)
In particular, I'm looking at this paper Fast Integer compression using SIMD instructions
and wondering if similar algorithms have been used in any system.
Look at LZO and lz4. Try them on your data and see how they perform.
A golomb code can be good like a huffman and is very simple and fast.
BWT + entropy coding (for instance Huffman coding) is quite fast (compexity O(n)) but needs two pass.
Are there any good compression algorithms for a large sequence of integers (A/D converter data). There is similar question
But the data is different in my case. It can be negarive or positive and changing like wave data.
EDIT1:sample data added
Please refer to this file for a data sample
Generally if you have some knowledge about the signal, use it to predict next value basing on previous ones. Then - compress difference between predicted and real value.
If prediction is good, differences will be small and their compressing will be good.
Anything more specific is unlikely possible without seeing the data and knowing about its physical nature.
update:
If the prediction is really well and uses all knowledge about dependencies, the differences are likely to be independent and something like arithmetic encoding would work for them.
You want a Delta Encode and then you want to apply a RLE or a Golomb Code. The Golomb Code can be as good as a Huffman Code.
Nearly any standard compression algorithm for byte strings can be applied; after all, any file of data can be interpreted as a sequence of signed integers. Is there something special about your particular integers that you think will make them amenable to some more-specific algorithm? You mention wave data; maybe take a look at FLAC which is designed for audio data; if your data has similar characteristics those techniques may be valuable.
You could diff the data then apply RLE on suitable subregions (i.e. between inflection points).
Does anyone of you know a lossless compression algorithm, which produces headerless outputs?
For example do not store the huffman tree used to compress it? I do not speak about hard coded huffman trees, but I like to know if there is any algorithm that can compress and decompress input without storing some metadata in its output. Or is this even theoretically impossible?
Of course it is posible. Among others, the LZ family of compressors don't need to output anything apart from the compressed data itself, as the dictionary is built on-line as compression (or decompression) progress. You have a lot of reference implementations for those LZ-type algorithms. For example, LZMA, component of 7zip.
Adaptive Huffman coding does exactly that. More generally, the term adaptive coding is used to describe entropy codes with this property. Some dictionary codes have this property too, e.g. run-length encoding (RLE) and Lempel-Ziv-Welch (LZW).
Run Length Encoding would be one example
lzo springs to mind. it's used in OpenVPN, with great results
Why are you looking for compression algorithms with headerless compressed output?
Perhaps (a) you have a system like 2-way telephony that needs low-latency streaming compression/decompression.
The adaptive coding category of compression algorithms mentioned by Zach Scrivena
and the LZ family of dictionary compression algorithms mentioned by Diego Sevilla and Javier
are excellent for this kind of application.
Practical implementations of these algorithms usually do have a byte or two of metadata
at the beginning (making them useless for (b) applications), but that has little or no effect on latency.
Perhaps (b) you are mainly interested in cryptography, and you hear that compress-before-encrypt gives some improved security properties, as long as the compressed text does not have fixed metadata header "crib".
Modern encryption algorithms aren't (as far as we know) vulnerable to such "cribs", but if you're paranoid you might be interested in
"bijective compression" (a, b, c, etc.).
It's not possible to detect errors in transmission (flipped bits, inserted bits, deleted bits, etc.) when a receiver gets such compressed output (making these algorithms not especially useful for (a) applications).
Perhaps (c) you are interested in headerless compression for some other reason. Sounds fascinating -- what is that reason?