I was trying to read some files like images, but when I try to open them with the notepad I found weird codes like this:
ÿH‹\$0H‹t$8HƒÄ _ÃÌÌÌÌÌÌH‰\$H‰l$H‰t$ WAVAWHƒì ·L
Click here to see the image
So I have the following questions:
Why do I find those weird symbols instead of zeros and ones?
Does programmers do this for security or optimization?
Is this an encoding such as ASCII that every symbol has an unique decimal and binary number associated?
Can anyone with the correspondent decoder read this information?
Thank you
Most data files like images are stored as hexadecimal. If you know the format of the file, you can use a hexadecimal editor (I use HexEdit) to look at the data.
A colour is often stored as RGB, meaning Red, Green, or Blue, so for instance, this is a dark red:
80 00 00 // (there are no spaces in the real file format, but hex editors add them.)
The format of an image depends on how it's stored. Most image formats have ways of encoding the difference between pixels rather than the actual pixels themselves, because there's a lot of information redundancy between the different pixels.
For instance, if I have a picture of the night sky with a focus on the moon, there's probably a big area in one corner that's all much the same shade of grey; encoding that without optimization would mean a hell of a lot of file that just read:
9080b09080b09080b09080b09080b09080b09080b59080b59080b5...
In this case, the grey is slightly bluish-purple, tending towards a brighter blue at the end. I've stored it as RGB here - R:90, G:80, B:b0 - but there are other formats for that storage too. Try them out here.
Instead of listing every pixel, I could equally say instead "6 lots of bluish-gray then it gets brighter in blue":
=6x9080b0+3x000005+...
This reduces the amount of information I would need to transmit. Most optimizations aren't quite that human-readable, but they operate on similar lines (this is a general information principle used in all kinds of things like .zip files too, not just images).
Note that this is still a lossless format; I could always get back to the actual pixel-perfect image. Bitmaps (.bmp) are lossless (though obviously still digital; they will never capture everything a human sees).
A number of formats use the frequency of images to encode the information. It's a bit like looking at a wave form of music, except it's two-dimensional. Depending on the sampling frequency, information could easily be lost here (and often is). JPEGs (.jpg) use lossy compression like this.
The reason you see ASCII characters is because some of the values just happen to coincide with ASCII text codes. It's pure coincidence; Notepad is doing its best to interpret what's essentially gibberish. For instance this colour sequence:
4e4f424f4459
happens to coincide with the letters "NOBODY", but also represents two pixels next to each other. Both are grey, especially the left (R:4e, G:4f, B:42) with the right-most one being a bit more blue (R:4f, G:44, B:59).
But that's only if your format is storing raw pixel information... which is expensive, so it probably isn't the case.
Image formats are a pretty specialist area. The famous XKCD cartoon "Digital Data" showcases the optimizations being made in some of them. This is why, generally speaking, you shouldn't use JPEG for text, but use something like PNG (.png) instead.
Related
I am working on a project I wanted to do for quite a while. I wanted to make an all-round huffman compressor, which will work, not just in theory, on various types of files, and I am writing it in python:
text - which is, for obvious reasons, the easiet one to implement, already done, works wonderfully.
images - this is where I am struggling. I don't know how to approach images and how to read them in a simple way that it'd actually help me compress them easily.
I've tried reading them pixel by pixel, but somehow, it actually enlarges the picture instead of compressing it.
What I've tried:
Reading the image pixel by pixel using Image(PIL), get all the pixels in a list, create a freq table (for each pixel) and then encrypt it. Problem is, imo, that I am reading each pixel and trying to make a freq table out of that. That way, I get way too many symbols, which leads to too many lengthy huffman codes (over 8 bits).
I think I may be able to solve this problem by reading a larger set of pixels or anything of that sort because then I'd have a smaller code table and therefore less lengthy huffman codes. If I leave it like that, I can, in theory, get 255^3 sized code table (since each pixel is (0-255, 0-255, 0-255)).
Is there any way to read larger amount of pixels at a time (>1 pixel) or is there a better way to approach images when all needed is to compress?
Thank you all for reading so far, and a special thank you for anyone who tries to lend a hand.
edited: If huffman is a real bad compression algorithm for images, are there any better ones you can think off? The project I'm working on can take different algorithms for different file types if it is neccessary.
Encoding whole pixels like this often results in far too many unique symbols, that each are used very few times. Especially if the image is a photograph or if it contains many coloured gradients. A simple way to fix this is splitting the image into its R, G and B colour planes and encoding those either separately or concatenated, either way the actual elements that are being encoded are in the range 0..255 and not multi-dimensional.
But as you suspect, exploiting just 0th order entropy is not so great for many images, especially photographs. As example of what some existing formats do, PNG uses filters to take some advantage of spatial correlation (great for smooth gradients), JPG uses quantized discrete cosine transforms and (usually) a colour space transformation to YCbCr (to decorrelate the channels, and to crush Chroma more mercilessly than Luma) and (usually) Chroma subsampling, JPEG2000 uses wavelets and colour space transformation both in its lossy and lossless forms (though different wavelets, and a different colour space transformation) and also supports subsampling though dropping a wavelet scale achieves a similar effect.
I'm new to GhostScript. Can you let me know the Ghostscript command for finding the number of colors used for each page in pdf file. I need to parse the results of this command from java program
There is no such Ghostscript command or device. It would also be difficult to figure out; so much depends on what you mean. Do you intend to count the colour of each pixel in every image for example ? Which colour spaces are you interested in ? What about ICCBased colour spaces, do you want the component values, or the CIE values ?
[edit]
Yeah there's no Ghostscript equivalent, I did say that.
You wuold have to intercept every call to the colour operators, examine the components being supplied and see if they were no black and white. For example, if you set a CMYK colour with C=M=Y=0 and K!=0 then its still black and white. Similar arguments apply for RGB, CIE and ICC colour spaces.
Now I bet ImageMagick doesn't do that, I suspect it simply uses Ghostscript to render a bitmap (probably RGB) and then counts the number of pixels of each colour in the output. Image manipulation tools pretty much all have to have a way to do that counting already, so its a low cost for them.
Its also wrong.
It doesn't tell you anything about the original colour. If you render a colour object to a colour space that is different to the one it was specified in, then the rendering engine has to convert it from the colour space it was in, to the expected one. This often leads to colour shifts, especially when converting from RGB to CMYK but any conversion will potentially have this problem.
So if this is what ImageMagick is doing, its inaccurate at best. It is possible to write PostScript to do this accurately, with some effort, but exactly what counts as 'colour' and 'black and white' is still a problem. You haven't said why you want to know if an input file is 'black and white' (you also haven't said if gray counts as black and white, its not the same thing)
I'm guessing you intend to either charge more for colour printing, or need to divert colour input to a different printer. In which case you do need to know if the PDF uses (eg) R=G=B=1 for black, because that often will not result in C=M=Y=0 K=1 when rendered to the printer. Not only that, but the exact colour produced may not even be the same from one printer to another (colour conversion is device-dependent), so just because Ghostscript produced pure black doesn't mean that another printer would.
This is not a simple subject.
I would like to know why we need to decode let's say a png to a bitmap in order to show the image.
Why not just show the png like that (encoded).
I'm asking here a moron type of question on purpose. It's clear to me it's impossible to show an encoded image just like that but I want to know why, and how an image is shown on a screen because it's easy just to do :
canvas.drawBitmap(((AndroidImage)Image).bitmap, x, y, null);
I want to understand the full of it. I'm guessing we need to show every pixels one by one, but I want more details.
It's easy to know how to do, it's a bit harder to understand why.
If someone has a course/tuto/article/explanation that explains it... I would appreciate
Thanks in advance
PS : Please don't respond "you need to decode/convert png to bitmap" I know that... And that's not my question
There are lots of reasons. There is not really a direct relation between 'a value in a file' and 'a pixel on a screen'.
You need to know the width and height of the bitmap. You cannot infer this from the image size -- it has to be stored somewhere inside the image file. (Or anywhere else. Point is, you have to know its size.)
You need to know the bit depth and color model of the bitmap. You cannot meaningfully copy an 8-bit indexed image directly onto a screen that accepts 32-bit BGR ordering with an unused byte, for example.
Your example, the PNG file format, specifies that all image data is compressed. This is for a sane reason: the PNG format was designed for use on web pages, in a time period where every byte still counted. But even the lowly simple BMP file format uses a very specific form of 'encoding': in its 24-bit format, every line consists of sets of BGR values for each pixel and is padded at the end with enough bytes to make its total length evenly divisible by 4.
JPEG uses an even more advanced encoding scheme (which is too difficult to explain in a few short words) so it can compress images even more. The advanced encoding scheme allows far more compression than regular methods (which in turn means there is only the tiniest relation between 'values in the file' and 'pixels on the screen').
The full question from our sample exam paper:
Explain by highlighting the relevant parts of the algorithm, why the GIF format is not the most compact format for representing images with natural content.
I understand that using GIFs wouldn't be great for natural images because it's limited to 256 colours, but why wouldn't it provide a sufficiently compact file? If anything, you'd think that less colours would imply smaller file-sizes.
In our notes we're told that the LZW compression used is better than Huffman for a few reasons (incl. the fact it only does one pass). Would Huffman encoding/compression result in a smaller file?
According to Wikipedia, the PNG format provides better compression than GIF. LZW is most likely the culprit then, but why? Which "parts of the algorithm" support the argument that it "is not the most compact format for representing images", particularly for natural images?
I'm not precisely sure on the details of LZW coding, but I believe it compresses data by building a dictionary of common bit sequences (which must be identical each time they appear). This means that line-drawings and so forth compress very well, because they contain many areas of "solid" colour (i.e. the same pixel colours repeated identically, many times in a row.) If you have an area of 100 white pixels, you can compress that by saying it's '100 of the same white pixel in a row'.
"Natural images", such as those produced by digital cameras, don't contain areas of solid colour. A photo of a blue wall will actually contain many different shades of blue - camera noise, at the very least, will make every pixel slightly different from its neighbours. LZW will not be able to find many repeated sequences, so it won't be able to compress the data much.
JPEG achieves smaller file sizes than GIF because it's not afraid to lose a bit of information from the picture - it's lossy. The trick is that JPG is designed to only lose information that humans are bad at seeing anyway. (See note 1.) We're good at seeing artifacts in "smooth" areas of pictures, but not good at seeing artifacts near sharp transitions in the image.
It's also an entirely different breed of compression algorithm, based on encoding the image as a 2-D frequency domain representation (digital signals processing type stuff) rather than trying to find repeating subsequences of bits.
Note 1: JPEG is a "perceptual" compression method. It plays to human visual strengths and weaknesses - small errors in "smooth" regions of colour are very easy to see, but small errors in "busy" areas of the image are quite easy to miss amongst the details.
MP3 audio works on the same principle. It throws away information that we can't hear; for example, if there's a loud sound followed by a quiet sound, we generally can't hear the quiet sound - our ears have been overwhelmed by the loud sound, and won't pick up the quiet sound that comes after it. The MP3 encoding throws away the quiet sound and concentrates on getting the loud sounds right.
Note 2:
I just realised I haven't actually addressed PNG.
PNG achieves better compression than GIF because it applies a pre-filtering step before the lossless compression (i.e. DEFLATE, roughly equivalent to LZW.) See Wikipedia's explanation of PNG filtering.
The idea is that neighbouring pixels may be slightly different from each other (due to noise), but they will be roughly the same value.
Say we had the data:
132 133 134 135 136 137 138
LZW looks at this and says "these values are all different; I can't compress that."
PNG looks at this in terms of the differences between values:
132 +1 +1 +1 +1 +1 +1 +1
That is, the first data element is 132, and the following elements are obtained by adding 1. This representation compresses quite well with LZW.
How can I write some information inside a photo file like jpg or gif without destroying the image? and of course without showing it on the photo since the whole idea is to send information in the file of photo undetected by anyone (to provide security/privacy to some extent)!
You can concatenate a gif and a zip (the information you want to hide) into one file. Gifs are read from the start of the file, while zips are read from the end of the file.
To create such a file in linux:
$ cat file1.gif >> outfile.gif
$ cat file2.zip >> outfile.gif
The resulting file should have the size of file1.gif and file2.zip together and should be openable by any gif viewer and zip file handler.
I'm sure there are many ways. Here's one:
In a photograph, minor variations in color would often be unnoticable to the naked eye, or even if noticed, might easily be mistaken for flaws in the quality of the picture.
So to take a simple example, suppose you had a gray-scale GIF image where the pallette is arranged in order from white to black with a smooth range of grays in between. I'm not sure how much you know about graphic file formats, but in GIF you have one byte per pixel, with each possible byte value mapping to some specific color. So in this case we could say pallette #0=RGB(0,0,0), pallette #1=RGB(1,1,1), ... palette #255=RGB(255,255,255).
Then you take an ordinary, real photograph. Break your secret message into individual bits. Set the last bit of each pallette index number to successive bits of your message.
For example, suppose the first eight pixels of the original photo are, say, 01 00 C9 FF FF C8 42 43. Your message begins with the letter "C", ascii code 0110 0111. So you change the last bit of the first byte to 0, changing the byte from 01 to 00. You change the last bit of the second byte to 1, changing the byte from 00 to 01. You change the last bit of the third byte to 1. It's already 1, so that makes no difference. Etc. You end up with the coded 8 bytes being 00 01 C9 FE FF C9 43 43.
The changes to the colors would be so subtle that it's unlikely that anyone looking at the picture would notice. Even if they did notice, unless they had a reason to be suspicious, they would likely just conclude that the picture was of less-than-perfect quality.
Of course nothing says you have to use 1 bit per byte for the secret message. Depending on how much degradation in quality you think you can get away with, you could use 2 bits per byte, or just change 1 bit in every other byte, etc.
Of course the same technique can be used with color photos: change the last bit in each of the RGB components to encode 3 bits per pixel, etc.
Hey that method is called as Steganography. With that we can hide messages in not just images but also in audio,vedeo and other formats.
Here is an opensouce Steganography software called steganotool This project is an open source steganography tool that can be used to hide and extract text to/ from Bitmap images.
About Steganography Mediums
Steganography in images
This type of steganography is very effective against discovery and can serve a variety of purposes. These purposes can include authentication, concealing of messages, and transmission of encryption keys. The most effective method for this type of steganography is normally the least significant bit method. This simply means that the hidden message will alter the last bit of a byte in a picture. By altering that last bit, there will be relatively no change to the color of that pixel within the carrier image. This keeps the message from being easily detected. The best type of image file to hide information inside of is a 24 bit Bitmap. This is due the large file size and high quality.
Steganography in Audio
In audio files, the most prominent method for concealing information is the low bit encoding method. The low bit encoding method is somewhat similar to the least significant bit method used in image files. The secret information is attached to the end of the file. One of the issues with low bit encoding is that it can be noticeable to the human ear. If someone is trying to hide information, this could be risky, since it is so easily detectable. The spread spectrum method is another method that has been used in the concealment of information in audio files. What this method does, is it adds random noise to the audio broadcast. This method enables for the information to be spread accross the frequency spectrum and remain hiddden under the random noise. The last method seen in audio steganography is echo hiding data. This method seeks to hide information by using the echos that occur naturally within sound files. Then, extra sound can be added to these echos, extra sound being the concealed message. This is a sufficient way to hide information, expecially since it even improves the sound of the original audio file in some cases.
Steganography In Video
Steganography in Videos is basically hiding of information in each frame of video. Only a small amount of information is hidden inside of video it generally isn’t noticeable at all, however the more information that is hidden the more noticeable it will become. This method is effective as well, but must be done right or else reveal more information instead of hiding.
Steganography In Documents
This is basically adding white space and tabs to the ends of the lines of a document. This type of
Steganography is extremely effective, because the use white space and tabs is not visible to the human eye in most text/document editors.
You can also refer to this open source project
This article can be very useful.
You can store some information in image metadata. In fact that's how man digital cameras 'tag' the photos their making (camera model, date and time, GPS coords etc.).
This data format is called EXIF (Exchangeable Image File Format). There are a lot of examples how to use it in programming languages. Here's the example in Java.
If you want to prevent users from reading this data you can encrypt them somehow, but they will always be able to remove it from your picture (by opening in Photoshop and using Save As for example).
If you want to hide data (text, another image, whatever) in a jpeg file, you can simply append it to the end of the file. When the image is viewed, you'll only see the original image and your added data will be ignored.
While not a super-duper way of hiding data, this is a good way of hiding another jpeg, as if anyone opens the file in notepad or a hex editor, they will probably not notice that there are two jpegs and not one because the end of the second image will just look like the first anyway.
In windows you can use simple command to hide archive in image : copy /b cat.jpg + Documents.rar cat_new.jpg.
And then use for example winrar to extract data ftom image as from archive.
But better way is to use Steganography.
simple program for it for linux and windows : http://linux01.gwdg.de/~alatham/stego.html
using this program you will use pass phrase, without it nobody even will know, that you hided some data in picture ))
compressed data in gif files is in variable sized chunks. each chunk starts with a length byte. Usually these chunks are 255 bytes of data (and the length byte says 255) except for the last two chunks (the last one is 0)
But you could re-code the gif with chunks whose size spells out the message
eg using characters for ASCII text or bytes 1 2 3 4 representing 00 01 10 11 binary data,
byte-frequency analysis on the gif would reveal fewer than expected 255 bytes.
and the size would be larger than the original, but the image would look exactly the same.
You don't need any type of advanced tool to hide a file in an image –– you just you need to know some basic DOS commands… just follow the link below, to see how to do this (it’s a short description of image steganography):
https://www.youtube.com/watch?v=ox9ArqXtaWw&list=PLqEKOAKK4IbSTfDJZE_lH-DiOjOzZUczx&index=2