I have a format for signatures in a database that an uncooperative vendor has been using for a client of ours. We are replacing their system.
A signature begins:
0A000500010002000100020001000100010001000100010001000100D100010001004F0001000100
01000100010001000100010001000100010001000100010001000100FF00FF00FF00010001000100
01000100010001000100010001000100010001000100010001000100010001000100010001000100
01000100010001000100010001000100010001000100010001000100010001000100010001000100
01000100010001000100010001000100010001000100010001000100010001000100010001000100
01000100010001000100010001000100010001000100010001000100010001000100010001000100
010001000100010002001C0001000100010001000100010001000100010001000100010001000100
01000100010001000100010001000100010001000100010001000100010001000100010001000100
01000100010001000100010001000100010001000100010001000100010001000100010001000100
01000100010001000100010001000100010001000100010001000100010001000100010001000100
01000100010001000100010001000100010001000100010001000100010001000100010001000100
01000100010001000100010001000100010001000100010001000100010001000100010001000100
01000100010001000100010001000100DA00FF00C100C000C100FF00DA00FF00C100C000C100FF00
DA00FF00C100C000C100FF00DA00FF00C100C000C100FF00DA00FF00C100C000C100FF00DA00FF00
C100C000C100FF00DA00FF00C100C000C100FF00DA00FF00C100C000C100FF00DA00FF00C100C000
C100FF00DA00FF00C100C000C100FF00DA00FF00C100C000C100FF00DA00FF00C100C000C100FF00
DA00FF00C100C000C100FF00DA00FF00C100C000C100FF00DA00FF00C100C000C100FF00DA00FF00
C100C000C100FF00DA00FF00C100C000C100FF00DA00FF00C100C000C100FF00DA00FF00C100C000
C100FF00DA00FF00C100C000C100FF00DA00FF00C100C000C100FF00DA00FF00C100C000C100FF00
DA00FF00C100C000C100FF00DA00FF00C100C000C100...
continues with more image data and ends with a long series of 0100's.
Any ideas on what the file format is?
Thanks.
Taking the first 16 or so bytes of data and putting them into a file, the linux 'file' command says:
$ file test.file
test.file: PCX ver. 2.5 image data
Looks like it might be a really simple format. 0A seems to be a header. Then it looks like pairs of darkness values, although it seems like 50% of the space is wasted. If you post a file, I'd be happy to try to write a converter.
The whole file is necessary because it looks like there's a fixed image size, and it might take a little fiddling. Do you have an image that's not confidential, but still has data in it?
That looks like raw pixel values from an 8-bit A/D converter (scanner), embedded in 16-bit words, little-Endian (x86) format.
Are the files all the same size? That would give you a strong clue to the image size.
It looks like RAW, un-encoded image data. Have you tried loading it straight into a two dimensional buffer and seeing what you get? Copy the file and name it foo.raw and try loading it in Photoshop, for example. If my guess is correct and it is just raw 16bit samples, you will have to supply the width and height yourself. The number of channels may be 1x16bit. As tfinniga says, the first two bytes may be a header you will have to skip.
I never seen this file format... maybe is a 'private format', and looks like easy to decode.
Edit: looks like the bytes are arranged in group of 2 (2 'xx' hexadecimal numbers)
Related
I started to work on a PNG encoding/decoding library for learning purposes so I want to implement every part of it by hand.
I got pretty long with it but now I'm a bit stuck. Here are the things I succesfully implemented already:
I can load a PNG binary and go through its bytes
I can read the signature and the IHDR chunk for metadata
I can read the IDAT chunks and concatenate the image data into a buffer
I can read and interpret the zlib headers from the above mentioned image data
And here is where I got stuck. I vaguely know the steps from here which are:
Extract the zlib compressed data according to its headers
Figure out the filtering methods used and "undo" them to get the raw data
If everything went correctly, now I have raw RGB data in the form of [<R of 1st pixel>, <G of 1st pixel>, <B of 1st pixel>, <R of 2nd pixel>, <G of 2nd pixel>, etc...]
My questions are:
Is there any easy-to-understand implementation (maybe with examples) or guide on the zlib extraction as I found the official specifications hard to understand
Can there be multiple filtering methods used in the same file? How to figure these out? How to figure out the "borders" of these differently filtered parts?
Is my understanding of the how the final data will look like correct? What about the alpha channel or when a palette is used?
Yes. You can look at puff.c, which is an inflate implementation written with the express purpose of being a guide to how to decode a deflate stream.
Each line of the image can use a different filter, which is specified in the first byte of the decompressed line.
Yes, if you get it all right, then you will have a sequence of pixels, where each pixel is a grayscale value, G, that with an alpha channel, GA, RGB (red-green-blue, in that order), or RGBA.
So I am developing a type of tamagotchi (virtual pet) for my microprocessors final. I made my own images 128x64 pixels long, as I am using a display with that resolution, so each image weighs 1Kbytes. I am using an at89s52 (8052) microcontroller and it doesn't have enough memory to store all the animations I want. My plan (and I kind of want to keep it that way) is to use an EPROM to save all my images with intel hex format (the programmer I am using is SUPERPRO and it imports that type of files). Of course the assembly code will be easy for me after the point where I have the data in the ROM. I am not that good of a programmer to develop a code that does exaclty what I want (convert images to intel hex), and all the software I have tried doesn't generate them correctly (inserts hex values that aren't supposed to be there, for example: in a blank space there is supposed to be only zeroes, and there is another value). I have tried with png with transparent background, with white background and jpg. The images I have are such:
http://imgur.com/a/yiCOb
(it seems I am not allowed to post images here)
I don´t see much help in other places of the internet, so the answer to this question would be of great help for future MCU-based programmers. Thank you.
It's about 30 years since I last made an EPROM :-)
Anyway, you need 2 things...
Part One
Firstly, your files are PNG format which means they have dates, times, palettes, gamma chunks and a bunch of zlib compressed data and you can't just copy that to a screen buffer. So, you need to convert the PNGs to a simple binary format where 0 is off and 1 is on and there is nothing else in the file. The easiest way to do that is with ImageMagick which is installed on most Linux platforms and is available for free on macOS and Windows. Let's say one of your frames is called anim.png and we want to get it to a simple format, like PGM (Portable GreyMap - see Wikipedia description) we can use ImageMagick like this at the console:
convert anim.png -compress none anim.pgm
The first few lines will be:
P2
128 64
255
255 255 255 255 255 255 255 ...
...
...
because the image is 128x64 and the maximum brightness in the file is 255. Then all the data follows in ASCII (because I put -compress none). In there, 255 represents white and 0 represents black.
As that is too big for the screen, here is an image of how it looks - hopefully you can see your black box as a bunch of zeroes in the middle at the bottom:
Now, if you run that same command again, but remove the -compress none, the same header will be produced but the data will follow in binary.
convert anim.png anim.pgm
And we can also use sed to delete the 3 lines of header:
convert anim.png anim.pgm | sed '1,3d' > anim.bin
Now you have a binary file of just pure pixels that is free of dates/times, author and copyrights, palettes and compressed data, you can pass to the next part.
Part 2
Secondly, once you have got your data in a sensible binary format you need to convert it to Intel Hex, and for that you need srec_cat which is available for Linux daily and via homebrew on a Mac.
Then, I haven't tested this and have never used it, I think you will want something like:
srec_cat anim.bin -binary -output -intel
:020000040000FA
:20000000323535203235352032353520323535203235352032353520323535203235352000
:200020003235352032353520323535203235352032353520323535203235352032353520E0
:200040003235352032353520323535203235352032353520323535203235352032353520C0
:200060003235352032353520323535203235352032353520323535203235352032353520A0
:20008000323535203235352032353520323535203235352032353520323535203235352080
...
:207E8000353520323535203235352032353520323535203235352032353520323535203202
:207EA0003535203235352032353520323535203235352032353520323535203235352032E2
:147EC000353520323535203235352032353520323535200A2A
:00000001FF
Summary
You can abbreviate and simplify what I am suggesting above - I will leave it there so folks can understand it in future though!
convert YourImage.png gray: | srec_cat - -binary -output -intel
The gray: is a very simple ImageMagick format equivalent to just the binary part of a PGM file without any header. Like PGM it uses one byte per pixel so it will be somewhat inefficient for your pure black and white needs. You can see that by looking at the file size - the PGM file is 8192 bytes, so 1 byte per pixel. If you really, really want 1 bit per pixel, you could use PBM format like this:
convert YourImage.png pbm: | sed '1,3d' | srec_cat - -binary -output -intel
Note:
From v7 of ImageMagick onwards, you should replace convert by magick so as to avoid clashing with Windows' built-in convert command that converts filesystems to NTFS.
ImageMagick is quite a large package, you could do this equally well just with the NetPBM suite, and use the tool called pngtopnm in place of convert.
I'm creating a program that "hides" an encrypted file at the end of a JPEG. The problem is, when retrieving this encrypted file again, I need to be able to determine when the JPEG it was stored in ends. At first I thought this wouldn't be a problem because I can just run through the file checking for 0xFF and 0xD9, the bytes JPEG uses to end the image. However... I'm noticing that in quite a few JPEGs, this combination of bytes is not exclusive... So my program thinks the image has ended randomly half way through it.
I'm thinking there must be a set way of expressing that a JPEG has finished, otherwise me adding a load of bytes to the end of the file would obviously corrupt it... Is there a practical way to do this?
You should read the JFIF file format specifications
Well, there are always two places in the file that you can find with 100% reliability. The beginning and the end. So, when you add the hidden file, add another 4 bytes that stores the original length of the file and a special signature that's always distinct. When reading it back, first seek to the end - 8 and read that length and signature. Then just seek to that position.
You should read my answer on this question "Detect Eof for JPG images".
You're likely running into the thumbnail in the header, when moving through the file you should find that most marked segments contain a length indicator, here's a reference for which do and which don't. You can skip the bytes within those segments as the true eoi marker will not be within them.
Within the actual jpeg compressed data, any FF byte should be followed either by 00 (the zero byte is then discarded), or by FE to mark a comment (which has a length indicator, and can be skipped as described above).
Theoretically the only way you encounter a false eoi reading in the compressed data is within a comment.
I have an idea for an app that would take some flash content which contains graphics and images like various geometric shapes and polygons and some random images and convert them to PDF.
Also, since I envision this app to be used my multiple users I want this process to be quick and scalable. One possible solution I could think of is have a small flash client with the capability of assembling the above mentioned graphics and images. Generate some sort of XML, send it to a server running a Java process which could render the PDF using iText.
I was wondering what are the other possible ways to do it or the best practices. Technology isn't an issue; open source or commercial.
I understand that image uploads etc will take variable amount of time so consider that images are readily available. Here are the criterias in terms of what I am looking for in a solution for PDF rendering:
No constraint on the flash client because the PDF render engine.
Scalable to multiple users
Speed and Efficiency
Least amount of serialization / deserialization
I would appreciate if you could share your tech stack idea. Thanks a lot!
PS: I would appreciated if you don't get bogged down my Flash >> XML >> Java approach.
I believe it to be one of the many approaches that could be taken.
If generating the PDF in the browser using Flash is an option, then consider using AlivePdf. If not, then check out XSL:FO, we use it for server side conversion to PDF.
I believe iText generates PDFs in Java code. It may or may not use XML as its data source; POJOs will do just as well.
Another way is XSL-FO. It requires an XML data source and an XSL-FO stylesheet to transform the XML and generate a PDF. Apache's Xalan (or any other XSL-T library) can do it for you.
"Quick" and "scalable" may require more than these. Uploading a lot of images is a process that has its own timescale and optimizations that have nothing to do with PDFs.
There's pdflib for PHP, and FPDF (also for PHP).
So you're also willing to consider other clients? It sounds like you've got a kids drawing app and want to generate something that'll preserve the state of their drawing at the time.
Lets face it, XML isn't that efficient. That's not its purpose. It's both machine and human readable, validatable, etc etc.
Instead, how about a <Canvas> based web page that submitted the state of that canvas to the server in JSON (fewer bytes, and less work to build them). The server can then work in whatever the hell library/language it wants. Lots of JSON->my-language libraries floating around out there.
Your choice in PDF libraries is then limited only by what is you have installed on your server. You also said you wanted to do as little reading/writing as possible.
The most efficient possible setup would be to have a read-only partial PDF already loaded into memory tailored to minimize the impact of canvas changes (including images). Each session would dupe that partial PDF, convert the JSON to PDF graphic commands, and save the PDF.
To minimize structural changes to the PDF you'd want to use Inline Images. No new objects in the PDF means you don't need to change your cross reference table at all (until you add fonts or want to reuse an existing image). You could build the "doc info" dictionary padded with a specific amount of spaces between objects so you could fill it in without changing any byte offsets (which would force you to recompute the xref table).
You may or may not need to mess with the page size... we are just talking about one page here, right?
So the PDF would look something like...
%%PDF-1.6
<3-4 random high order bytes to convince folks that we're a binary stream>
1 0 obj
<</Type/Catalog/Pages 2 0 R>>
endobj
2 0 obj
<</Type/Pages/Count 1/Kids[3 0 R]>>
endobj
3 0 obj
<</Type/Page/Contents 4 0 R/MediaBox[0 0 612 792]/Parent 2 0 R>>
endobj
5 0 obj
<</Type/DocInfo/Author() --<insert big whitespace gap here>--
/Title() --<ditto>--
/Subject() --<ditto>--
/Keywords() --<ditto>--
/Creator(My app's Name)
/Producer(My pdf library's name)
/CreationDate(encodedDateWhenThisTemplateWasBuilt) D:YYYYMMDDHHMMSS-timeZoneOffset
/ModDate() --<another, smaller whitespace gap>--
>>
4 0 obj
<</Filter/SeveralDifferentFiltersAvailable/Length --<byte length of the stream in this file>-->>
stream
And your template stops there. You'd have a similar "end of the PDF" template that would look something like this:
endstream
endobj
xref
0 6
0000000000 65535 f
0000000010 00000 n
0000000025 00000 n
0000000039 00000 n
0000000097 00000 n
0000000050 00000 n
trailer
<</Root 1 0 R/Size 6/Info 5 0 R>>
startxref
--<some white space>--
%%EOF
The columns of numbers at the end are all wrong. The first column is the byte offset of that particular object (and I'm not up for counting bytes just now thank you). The second column is largely irrelevant.
PDF filling app will need to know:
The byte offsets of everything you intend to fill in within the first template.
All the "doc info" fields, which are all optional by the way. The /Info key and the dictionary it points to are optional for that matter. You could yank 'em if you cared to.
the /Length key of the content stream. That needs to be the post-filter byte length of the stream itself.
How to convert the JSON into pdf drawing commands. If you wanted to cheat a bit you could use iText[Sharp]'s PdfContentByte class, use its drawing commands, and then get the finished byte stream and slap that into your PDF. Be sure you use Inline Images or this whole scheme goes right out the window. There are probably other libraries you could gut similarly if you felt the need. Or you could just read up on the PDF spec and roll your own. You'll be sticking to a fairly limited subset of PDF's content syntax.
The byte offset of the word "xref" from the start of the file. You can calculate this: LengthOfInitialTemplate + LengthOfContentStream + OffsetFromStartOf2ndTemplateTo'xref'.
The byte offset of the line below "startxref", which is where you write the aforecalculated byte offset of 'xref'
You're not going to get much more efficient than that. You'd read in your templates once. Read/calculate the byte offsets you needed once.
I'm trying to open an image file and store a list of pixels by color in a variable/array so I can output them one by one.
Image type: Could be BMP, JPG, GIF or PNG. Any of them is fine and only one needs to be supported.
Color Output: RGB or Hex.
I've looked at a couple libraries (RMagick, Quick_Magick, Mini_Magick, etc) and they all seem like overkill. Heroku also has some sort of difficulties with ImageMagick and my tests don't run. My application is in Sinatra.
Any suggestions?
You can use Rmagick's each_pixel method for this. each_pixel receives a block. For each pixel, the block is passed the pixel, the column number and the row number of the pixel. It iterates over the pixels from left-to-right and top-to-bottom.
So something like:
pixels = []
img.each_pixel do |pixel, c, r|
pixels.push(pixel)
end
# pixels now contains each individual pixel of img
I think Chunky PNG should do it for you. It's pure ruby, reasonably lightweight, memory efficient, and provides access to pixel data as well as image metadata.
If you are only opening the file to display the bytes, and don't need to manipulate it as an image, then it's a simple process of opening the file like any other, reading X number of bytes, then iterating over them. Something like:
File.open('path/to/image.file', 'rb') do |fi|
byte_block = fi.read(1024)
byte_block.each_byte do |b|
puts b.asc
end
end
That will merely output bytes as decimal. You'll want to look at the byte values and build up RGB values to determine colors, so maybe using each_slice(3) and reading in multiples of 3 bytes will help.
Various image formats contain differing header and trailing blocks used to store information about the image, data format and EXIF information for the capturing device, depending on the type. Probably going with a something that is uncompressed would be good if you are going to read a file and output the bytes directly, such as uncompressed TIFF. Once you've decided on that you can jump into the file to skip headers if you want, or just read those too to see or learn what's in them. Wikipedia's Image file formats page is a good jumping off place for more info on the various formats available.
If you only want to see the image data then one of the high-level libraries will help as they have interfaces to grab particular sections of the image. But, actually accessing the bytes isn't hard, nor is it to jump around.
If you want to learn more about the EXIF block, used to describe a lot of different vendor's Jpeg and TIFF formats ExifTool can be handy. It's written in Perl so you can look at how the code works. The docs nicely show the header blocks and fields, and you can read/write values using the app.
I'm in the process of testing a new router so I haven't had a chance to test that code, but it should be close. I'll check it in a bit and update the answer if that didn't work.