It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.
Closed 11 years ago.
I have been around the world twice and a half trying to fix a strange issue with a large sum of images I have.
What I need is to know how to read and write the first 4 and 18th bytes in the header of a JPEG file. If they match certain properties, I need to do some certain work and tweak these bytes to reflect something else. I'm basically fixing the format of these pictures to a more standard format, as Delphi doesn't recognize the format they're in.
So how do I read/write these bytes? What do I use? I know nothing about reading the raw data of an image file, let alone tweaking it.
NOTE
Deleted most of the question as I had put way too much information from the start.
The mechanics of actually changing the JPG header are very easy. That doesn't mean fixing the JPG is easy, it's just that if you know what to change, you can easily do it. Since that's what you're asking, here's how to change the JPG header in-memory, load the modified JPG into a TJpgImage and convert to TBitmap.
Please note I don't think this is a good approach. Fixing the resulting bitmap is so much easier! See my fresh answer to your other question for how to fix the bitmap here.
And here's the code you asked for (how to change the JPG header):
function LoadJpegIntoBitmap_HeaderFix(const FileName:string): TBitmap;
var M: TMemoryStream;
P: PByteArray;
Jpg: TJPEGImage;
begin
M := TMemoryStream.Create;
try
M.LoadFromFile(FileName);
if M.Size < 18 then raise Exception.Create('File too short.');
P := M.Memory;
// Here you can manipulate the header of the JPG file any way you want.
if P[0] = $07 then P[3] := P[17] - P[0]*3;
// Now load the modified JPG into a TJpgImage
Jpg := TJPEGImage.Create;
try
Jpg.LoadFromStream(M);
// Convert to bitmap
Result := TBitmap.Create;
Result.Assign(Jpg);
finally Jpg.Free;
end;
finally M.Free;
end;
end;
Don't even think you can tweak manually the first x bytes of your jpeg files.
As it has been explained already, the jpeg format is multiform and very complex. So much that some commonly used libraries do not handle all the acceptable formats. Open a libjpeg.h and see the bunch of header structures you can find in a jpeg file.
If you want to modify your header without changing the data, you still have to open the jpeg with a library that will hand over the proper structures for you to modify the relevant flags and save it back with the wanted format.
Either find a library written in Delphi or with a Delphi interface and use it to build your batch converter or to transform the format on the fly when opening a jpeg in your application.
If you know the exact combination of properties/flags defining your problem files, you can tweak the jpeg.pas unit to suit your need as I've shown in the previous answer.
Here's some pseudo code really quick. I'll improve it later. I have to run.
Assign the Adobe JPEG image file to a TFileStream
Read first 18 bytes and use CompareMem to see if signature matches Adobe RGB JPEG
If RGB JPEG then
We'll either:
Load from stream and tweak the header as we load
OR
Load from stream, copy to TBitmap, and use ScanLine to fix RGB
Else
Load from stream, normally
Hint, see LoadFromStream() instead of LoadFromFile().
I don't know what you're doing with the image afterward, so there may be some more work to be done.
Right, so your options here are (since we've completely ruled out any 3rd party code whatsoever):
Some Delphi guru (which I am certainly not) coming out of the woodwork and illuminating us with the existence of standard Delphi library code which handles this, or,
You write your own JPEG metadata decoding library.
Since I can't help you with the first option, here are some references you'll need while pursuing the second:
Wikipedia's JPEG article - one of the better written articles on the wiki. Great introductory material to this hairy subject.
ITU T.81 - The original JPEG specification. The most important bit is in Annex B, but this actually describes a format known as JIF which isn't actually in use. However, both JFIF and EXIF are based on it.
ICC - This specifies ICC (International Color Consortium) profiles, which are the bit that seems to be wrong with your headers.
ICC 2010 - New version of above specification, with Errata.
JFIF - The format most JPEG files actually use.
EXIF - Another format JPEG files use, especially those originating from digital cameras.
ITU T.84 - Less useful, this describes JPEG extensions. I'm including it here for completeness.
Sadly, none of the public domain JPEG implementations that I'm aware of (in any language) actually handle the relevant bits of the standards, namely EXIF and ICC profiles, so I can't give you any code you could draw inspiration from. The best place to look would probably be libexif, but that's GPL'ed.
What you're going to want to do is to read in the JPEG file as a regular file, parse the input according to information gleaned from the above documents, make the necessary changes, and write it back out. I don't think this is going to be easy, but apparently this is the only solution you'll accept.
Good luck!
Related
I already did a lot of research and realized that clear information about "How to generate PDF/A-1a" or "...convert to PDF/A-1a" is really rare. I found some information to convert to PDF/A-1a via GhostScript, but I didn't make it to get it working. So, maybe there are some necessary conditions for the data missing in the first place. Conditions like propper metadata of the PDF, structured data for readability by a screen reader, alternative text for pictures, and a declaration of the given language of the text. I need a proper working GhostScript command with the corresponding gs version and the mandatory file conditions to generate or even convert to PDF/A-1a. PDF/A-1b means nothing to me because I'm already able to convert to that.
Thanks for any help.
How to convert image.png or image.bmp to integer array? (do not use any non-standard library)
Please ignore chunks that are not directly related to image data.(IHDR、IEND...etc.)
thank you very much.
SOLVED: I should use binary I/O function in stdio.h to read image file. thanks
If you have to read images into arrays without any image processing libraries you need two things:
You need means to read files in general.
You need to know the internal structure of the file formats you want to read.
So for png refer to https://www.w3.org/TR/2003/REC-PNG-20031110/
This document will tell you where to find the image dimensions, pixel data and other features. It's basically a manual for software developers on how to use this standard format properly.
Some image formats will require additional work like decrompression.
I have a collection of images in the CBM file format.
What is this format? How to read it? How to convert it to, say, a BMP bitmap?
Searching the net, I found that it could mean one of those three things:
1) Fuzzy Bitmap, as per Michael Mauldin's library, FBM (1989). The softwares XnView and Graphics Convert Pro 2011 should be able to open it, but they don't. Most likely it means that the file I have is not really a Fuzzy Bitmap.
2) Compiled bitmap, as per XLib (wikipedia)
How to proceed in order to test that my file is such a Compiled Bitmap? Where to download XLib / what should I build?
3) Compressed Bitmap. Do you have more info's on this format?
I found CreateDIBitmap function. If you pass correct parameters, you should be able to save it into some more known file format.
Here you can find something about Bitmap Compression.
If you don't know image details like resolution, bitsperpixel, compression type,... It will be hard to read it.
I believe CBMs are just Zip files renamed. Try renaming and unzipping.
try to open it with konvertor (www.konvertor.net); it does read several CBMs formats
I have a very large PDF File (200,000 KB or more) which contains a series of pages containing nothing but tables. I'd like to somehow parse this information using Ruby, and import the resultant data into a MySQL database.
Does anyone know of any methods for pulling this data out of the PDF? The data is formatted in the following manner:
Name | Address | Cash Reported | Year Reported | Holder Name
Sometimes the Name field overflows into the address field, in which case the remaining columns are displayed on the following line.
Due to the irregular format, I've been stuck on figuring this out. At the very least, could anyone point me to a Ruby PDF library for this task?
UPDATE: I accidentally provided incorrect information! The actual size of the file is 300 MB, or 300,000 KB. I made the change above to reflect this.
I assume you can copy'n'paste text snippets without problems when your PDF is opened in Acrobat Reader or some other PDF Viewer?
Before trying to parse and extract text from such monster files programmatically (even if it's 200 MByte only -- for simple text in tables that's huuuuge, unless you have 200000 pages...), I would proceed like this:
Try to sanitize the file first by re-distilling it.
Try with different CLI tools to extract the text into a .txt file.
This is a matter of minutes. Writing a Ruby program to do this certainly is a matter of hours, days or weeks (depending on your knowledge about the PDF fileformat internals... I suspect you don't have much experience of that yet).
If "2." works, you may halfway be done already. If it works, you also know that doing it programmatically with Ruby is a job that can in principle be solved. If "2." doesn't work, you know it may be extremely hard to achieve programmatically.
Sanitize the 'Monster.pdf':
I suggest to use Ghostscript. You can also use Adobe Acrobat Distiller if you have access to it.
gswin32c.exe ^
-o Monster-PDF-sanitized ^
-sDEVICE=pdfwrite ^
-f Monster.pdf
(I'm curious how much that single command will make your output PDF shrink if compared to the input.)
Extract text from PDF:
I suggest to first try pdftotext.exe (from the XPDF folks). There are other, a bit more inconvenient methods available too, but this might do the job already:
pdftotext.exe ^
-f 1 ^
-l 10 ^
-layout ^
-eol dos ^
-enc Latin1 ^
-nopgbrk ^
Monster-PDF-sanitized.pdf ^
first-10-pages-from-Monster-PDF-sanitized.txt
This will not extract all pages but only 1-10 (for proof of concept, to see if it works at all). To extract from every page, just leave off the -f 1 -l 10 parameter. You may need to tweak the encoding by changing the parameter to -enc ASCII7 (or UTF-8, UCS-2).
If this doesn't work the quick'n'easy way (because, as sometimes happens, some font in the original PDF uses "custom encoding vector") you should ask a new question, describing the details of your findings so far. Then you need to resort bigger calibres to shoot down the problem.
At the very least, could anyone point
me to a Ruby PDF library for this
task?
If you haven't done so, you should check out the two previous questions: "Ruby: Reading PDF files," and "ruby pdf parsing gem/library." PDF::Reader, PDF::Toolkit, and Docsplit are some of the relatively popular suggested libraries. There is even a suggestion of using JRuby and some Java PDF library parser.
I'm not sure if any of these solutions is actually suitable for your problem, especially that you are dealing with such huge PDF files. So unless someone offers a more informative answer, perhaps you should select a library or two and take them for a test drive.
This will be a difficult task, as rendered PDFs have no concept of tabular layout, just lines and text in predetermined locations. It may not be possible to determine what are rows and what are columns, but it may depend on the PDF itself.
The java libraries are the most robust, and may do more than just extract text. So I would look into JRuby and iText or PDFbox.
Check whether there is any structured content in the PDF. I wrote a blog article explaining this at http://www.jpedal.org/PDFblog/?p=410
If not, you will need to build it.
Maybe the Prawn ruby library? link text
Is there a good way to see what format an image is, without having to read the entire file into memory?
Obviously this would vary from format to format (I'm particularly interested in TIFF files) but what sort of procedure would be useful to determine what kind of image format a file is without having to read through the entire file?
BONUS: What if the image is a Base64-encoded string? Any reliable way to infer it before decoding it?
Most image file formats have unique bytes at the start. The unix file command looks at the start of the file to see what type of data it contains. See the Wikipedia article on Magic numbers in files and magicdb.org.
Sure there is. Like the others have mentioned, most images start with some sort of 'Magic', which will always translate to some sort of Base64 data. The following are a couple examples:
A Bitmap will start with Qk3
A Jpeg will start with /9j/
A GIF will start with R0l (That's a zero as the second char).
And so on. It's not hard to take the different image types and figure out what they encode to. Just be careful, as some have more than one piece of magic, so you need to account for them in your B64 'translation code'.
Either file on the *nix command-line or reading the initial bytes of the file. Most files come with a unique header in the first few bytes. For example, TIFF's header looks something like this: 0x00000000: 4949 2a00 0800 0000
For more information on the TIFF file format specifically if you'd like to know what those bytes stand for, go here.
TIFFs will begin with either II or MM (Intel byte ordering or Motorolla).
The TIFF 6 specification can be downloaded here and isn't too hard to follow