I am converting some largish images from a multi-image (pyramidal) tif to png format. The salient parts of the report from "identity -verbose" on the largest image are here:
Image:
Format: TIFF (Tagged Image File Format)
Class: DirectClass
Geometry: 72224x64080+0+0
Resolution: 72x72
Print size: 1003.11x890
Units: PixelsPerInch
Type: TrueColor
Base type: TrueColor
Endianess: MSB
Colorspace: RGB
Depth: 8-bit
Channel depth:
red: 8-bit
green: 8-bit
blue: 8-bit
...
Page geometry: 72224x64080+0+0
...
Scene: 2 of 12
Compression: JPEG
Orientation: TopLeft
Properties:
...
Filesize: 1.389GBB
Number pixels: 4.6281GB
Pixels per second: 5.516MB
User time: 218.277u
Elapsed time: 13:60.020
Version: ImageMagick 6.7.1-0 2011-07-06 Q16 http://www.imagemagick.org
I am intending to use deepzoom composer to produce input for the Silverlight multiscaleimage control with this image. My question is how do I bring my system to its knees while processing these images with ImageMagick - it is taking too long to convert them. I have looked at a few articles, but I can't seem to get anywhere.
Some system and other related information:
OS: Windows 7 64 bit.
CPU: Intel Core2 Duo E7300 # 2.66, 2.67
RAM: 4.0 GB
PAGEFILE: 8-12GB on non-OS disk
"MAGICK_TMPDIR": Yet another empty, non-os disk with 140GB available.
Here is the result of "identify -list resource":
File Area Memory Map Disk Thread
------------------------------------------------------------------
1536 4.1582GB 15.491GiB 30.981GiB unlimited 2
I am running this command to extract the image referenced above:
convert "myFN.tif[2]" -limit file 8192GB -limit thread 32 "myFN%d.png"
Adding the two limit values did not seem to make a difference. When I run this, I average about 10% CPU utilization and have a pagefile commit size of 3BG. I can barely tell that it is running.
Q1) Is there anything else I can do to get ImageMagick to use more system resources? Most of the "large image" links I have found are asking the opposite question.
Q2) Changing "policy.xml" values (such as files) located here:
C:\Program Files\ImageMagick-6.7.1-Q16\www\source
did not seem to affect anything - the changes did not show up in the next "identify -list resource." Is there a trick to this?
Q3) Any other hints or ideas for this task?
Thanks,
David
libvips can convert pyramidal tiff directly into deepzoom pyramids. It's free, very fast and doesn't need much memory.
For example, I see:
$ vipsheader vips-pyr.tif
vips-pyr.tif: 18008x22764 uchar, 3 bands, srgb, tiffload
$ time vips dzsave vips-pyr.tif x.zip
real 0m9.763s
user 0m19.700s
sys 0m4.644s
peak memory: 180mb
That's a 20,000 x 20,000 pyramidal tiff converted to deepzoom in under 10 seconds on a small laptop. It's writing a zip file containing the pyramid, so you can upload to a server immediately. Memory use scales with image width, so it'll do very large images --- I regularly process 250,000 x 250,000 pixel slides.
There's a chapter in the docs introducing dzsave.
For your (my) image, the limiting factor is the size of the pixel cache, which is limited by the setting "MAGICK_AREA_LIMIT". The default of 4GB is not large enough for 72224 x 64080 - that would require a setting of at least 4.4GB - try "MAGICK_AREA_LIMIT=8GB."
If you want to control the impact that ImageMagick has on system RAM and the system page file, then you can limit that using "MAGICK_MEMORY_LIMIT." In truth, there isn't much need to use a large limit there since the fallback location for the pixel cache is mapped memory files, which are on the same order of magnitude of efficiency as the system page file. Try "MAGICK_MEMORY_LIMIT=2GB", to keep the pixel cache out of there (not that it would go there anyways - it is way bigger than 12GB.)
You want the pixel cache to go to mapped memory, so try "MAGICK_MAP_LIMIT=100GB" to take advantage of that space you have. The memory mapped files will end up, not in the system temp directory, but in the directory specified by "MAGICK_TMPDIR".
For extra credit, you also might experiment with the Q8 version, since you don't need 16 bit color channels. You can expect roughly half the disk io with that version.
Good luck!
David
The Q8 version uses half the disk space and time to complete a conversion compared to the Q16 version! Also, if you are going to end up breaking up the image into tiles, you can do that in a single step with a command like:
convert.exe" "WRL_15_1A.tif[2]" -crop 14409x15396 +repage
-scene 0 "temp\WRL_15_1A%d.tif"
The "[2]" calls out the third image (the one with the highest
resolution.)
The -crop parameters are 1/4 of the width and height
respectively, giving us 16 tiles.
The +repage sets all of the tiles at origin (0,0)
The "%d" numbers the files, starting at the # set by
"-scene".
Imagemagick has a format for handling large files (mpc). Basically trading disk space for ram. Two files are generated on convert, .mpc and .cache, and you can run imagemagick commands on the smaller .mpc file. These files may only work on your current build of imagemagick, so they aren't suitable for archive.
Related
I wonder why the jpeg compression in recent MATLAB and Octave versions has gone so strong that it causes noticable compression artifacts:
Octave 3 jpeg image with size of 41 KB with no artifacts:
MATLAB 9 jpeg image with size of 26 KB with artifacts:
Octave 5 jpeg image with size of 23 KB with artifacts:
Here is the code to plot:
description = strcat('version-', num2str(version));% find out MATLAB/Octave version
x=1:2; % simple var
figure; % plot
plot(x, x);
title(description);
print(strcat("test_jpeg_size_", description ,'.jpg'), '-djpeg'); % write file
Do you know a possibility to tell MATLAB and Octave to do a weaker jpeg compression. I cannot find anything like this on https://de.mathworks.com/help/matlab/ref/print.html.
I know that I could plot png files and use imagemagick to convert it to jpeg with a given quality but this would be a workaround with additional tools. Or I could png files in the first place but the real images have no compression advantages for png (like like simple one here) and I would have to change a lot of other stuff.
This used to be documented*, I was surprised to not find it in the documentation pages. I tested it with the latest version of MATLAB (R2019b) and it still works:
The -djpeg option can take a quality value between 0 and 100, inclusive. The device option becomes -djpeg100 or -djpeg80, or whatever value you want to use.
print(strcat("test_jpeg_size_", description ,'.jpg'), '-djpeg100');
* Or at least I remember it being documented... The online documentation goes back to R13 (MATLAB 6.5), and it's not described in that version of the documentation nor in a few random versions in between that and the current version.
However, I strongly recommend that you use PNG for line drawings. JPEG is not intended for line drawings, and makes a mess of them (even at highest quality setting). PNG will produce better quality with a much smaller file size.
Here I printed a graph with -djpeg100 and -dpng, then cut out a small portion of the two files and show them side by side. JPEG, even at 100 quality, makes a mess of the lines:
Note that, in spite of not having any data loss, the PNG file is about 10 times smaller than the JPEG100 file.
You can go for
f = getframe(gcf);
imwrite(f.cdata, 'Fig1.jpg')
where imwrite takes the following options
Compression (compression scheme)
Quality (quality of JPEG-compressed file from 0 to 100)
See the doc of imwrite.
I am searching for a library (in any language) that is capable of reading a region of an image file (any format) without having to initially read that entire image file.
I have come across a few options such as vips, which does indeed not keep the entire image in memory, but still seems to need to read it entirely to begin with.
I realize this may not be available for compressed formats such as jpegs, but in theory it sounds like bmps or tiffs should allow for this type of reading.
libvips will read just the part you need, when it can. For example, if you crop 100x100 pixels from the top-left of a large PNG, it's fast:
$ time vips crop wtc.png x.jpg 0 0 100 100
real 0m0.063s
user 0m0.041s
sys 0m0.023s
(the four numbers are left, top, width, height of the area to be cropped from wtc.png and written to x.jpg)
But a 100x100 pixel region from near the bottom is rather slow, since it has to read and decompress the pixels before the pixels you want to get to the right point in the file:
$ time vips crop wtc.png x.jpg 0 9000 100 100
real 0m3.063s
user 0m2.884s
sys 0m0.181s
JPG and strip TIFF work in the same way, though it's less obvious since they are much faster formats.
Some formats support true random-access read. For example, tiled TIFF is fast everywhere, since libvips can use libtiff to read only the tiles it needs:
$ vips copy wtc.png wtc.tif[tile]
$ time vips crop wtc.tif x.jpg 0 0 100 100
real 0m0.033s
user 0m0.013s
sys 0m0.021s
$ time vips crop wtc.tif x.jpg 0 9000 100 100
real 0m0.037s
user 0m0.021s
sys 0m0.017s
OpenSlide, vips, tiled OpenEXR, FITS, binary PPM/PGM/PBM, HDR, RAW, Analyze, Matlab and probably some others all support true random access like this.
If you're interested in more detail, there's a chapter in the API docs describing how libvips opens a file:
http://libvips.github.io/libvips/API/current/How-it-opens-files.md.html
Here's crop plus save in Python using pyvips:
import pyvips
image = pyvips.Image.new_from_file(input_filename, access='sequential')
tile = image.crop(left, top, width, height)
tile.write_to_file(output_filename)
The access= is a flag that hints to libvips that it's OK to stream this image, in case the underlying file format does not support random access. You don't need this for formats that do support random access, like tiled TIFF.
You don't need to write to a file. For example, this will make a buffer object containing the file encoded as a JPG:
buffer = tile.write_to_buffer('.jpg', Q=85)
Or this will write directly to stdout:
target = pyvips.Target.new_from_descriptor(0)
tile.write_to_target('.jpg', Q=85)
The Q=85 is an optional argument to set the JPG Q factor. You can set any of the file save options.
ITK can do it with some formats. There is a method CanStreamRead which returns true for formats which support streaming, such as MetaImageIO. An example can be found here. You can ask more detailed questions on ITK's forum.
If have control over the file format, I would suggest you use tiled TIFF files. These are typically used in digital pathology whole slide images, with average sizes of 100kx30k pixels or so.
LibTiff makes it easy to read the tiles corresponding to a selected ROI. Tiles can be compressed without making it less efficient to read a small region (no need to decode whole scan lines).
The BMP format (uncompressed) is simple enough that you can write the function yourself.
TIFF is a little less easy, as there are so many subformats. But the TIFF library (TIFFlib) supports a "tile-oriented" I/O mode. http://www.libtiff.org/libtiff.html#Tiles
I don't know of such a library solution.
Low level, file-read access is format specific and in particular, file mapping is OS specific.
If you have access to the raw bytes then assuming you know the width, height, depth and number of channels etc. then calculating file offsets is trivial so just roll your own.
If you're transferring the extracted data over a network you might consider compressing the extracted ROI in-memory if it's relatively big before sending it over the network.
I'm having problems opening up certain jpeg files (ones from Facebook and Instagram, and some Samsung phones) in Photoshop. I've read that if I use mogrify -comment test insert-image-here.jpg, it will "handle" the file and somehow it'll open up in Photoshop. And surprisingly, it works very well.
However, I recently "mogrified" an image using the same above command, only to have the filesize go down by 0.71mb, which was alarming as I don't want to recompress my images in jpeg. I then mogrified it ten more times, but I didn't see any obvious visual losses. I tried "mogrifying" a small 170kb image 20 times, and the filesize initially decreased, then increased every subsequent iteration. I compared the files by swapping between them quickly, but didn't see any quality loss.
What is mogrify doing that is decreasing the filesize, seemingly without reducing the quality? Is there a quality option that I can add to make mogrify not reduce the filesize?
This is a similar question from another user Why does the size of my image decrease when I add a comment to an image? but I cannot discern any quality loss whatsoever, so running my image through at 95 or 70% quality 20 times would be immediately noticeable.
Here is the link to the image that I am using as a test: http://ocicat.wildrain.tripod.com/sitebuildercontent/sitebuilderpictures/aragon.jpg
Edit:
I ran two more images through the mogrify command 1000 times (one of white noise which I didn't include.) I still don't see any quality loss--is JPEG compression that unnoticeable (maybe my eyes are failing me.) Interestingly, the final file size and the original file size of these two images are the same.
Zero iterations:
1000 iterations:
I do not understand why you would have trouble opening a JPG from those sources. But some viewers will not handle CMYK jpg files properly. I would be surprised if Photoshop has that problem. If you use Imagemagick to add a comment, then it will decompress your file and recompress it. Imagemagick will use the -quality value in the file if it can find it and make the output the same. However, if it cannot find the quality value in the file, then it will compress at value 92. That could cause a decrease in the file size if it was at 100 as input but was recompressed at 92. The next time you do the same it will continue to use 92. However, there might be some loss of effective quality because JPG is lossy. But at 92 it probably will not be visually noticeable. You could try convert in place of mogrify and see if that is any different. Also there is no need to add a comment. Just reading the input and saving it again will decompress and recompress it in Imagemagick. See http://www.imagemagick.org/script/command-line-options.php#quality
Your image is sRGB and has a quality of 75 according to Imagemagick
identify -verbose Aragon.jpg
Format: JPEG (Joint Photographic Experts Group JFIF format)
Mime type: image/jpeg
Class: DirectClass
Geometry: 9600x14400+0+0
Resolution: 2400x2400
Print size: 4x6
Units: PixelsPerInch
Type: TrueColor
Endianess: Undefined
Colorspace: sRGB
Depth: 8-bit
...
Compression: JPEG
Quality: 75
Orientation: Undefined
Properties:
date:create: 2017-08-18T21:26:37-07:00
date:modify: 2017-08-18T21:26:37-07:00
jpeg:colorspace: 2
jpeg:sampling-factor: 2x2,1x1,1x1
signature: bad15aa674dc45312d47627b620c895ee76b1fa4457b55bf1acca85883de5963
Artifacts:
filename: aragon.jpg
verbose: true
So the CMYK issue is not present.
Is this file before or after you processed it with Imagemagick mogrify?
If this file was originally quality 75 and was recompressed at 92 or some higher value than 75, then it might increase in file size.
If you do not want the file size to decrease then recompress at 75 or higher. 100 would give you the least compression, but may increase the file size.
Other factors could be a change is -sampling-factor for the JPG. See http://www.imagemagick.org/script/command-line-options.php#sampling-factor. Also there could be a difference in the compression tables. Imagemagick use libjpg to read and decompress JPGs. The original JPG may have been compressed using other tools.
Another factor might be the introduction or removal of a color profile. Imagemagick should not be changing that automatically.
My best suggestion is to check the quality of the input and what quality is assigned to the output. Also check the input colorspace. Use identify -verbose yourimage to see what may have changed.
Unfortunately, I do not know exactly what is happening. I can only tell you some of the factors that may be involved.
I used Imagemagick 6.9.9.7 Q16 Mac OSX to convert your file.
convert aragon.jpg aragon2.jpg
The input has Filesize: 5.94157MiB. The output has Filesize: 5.23264MiB. Both files have the same quality 75. So there is a slight change in file size due to decompression and recompression, probably due to actual loss in quality due to the fact that JPG has a lossy compression. Or perhaps due to a change in compression tables used. Doing it once more yields a Filesize: 5.23289MiB. So a very slight increase. Doing
convert aragon.jpg -quality 100 aragon4.jpg
Yields a Filesize: 13.968MiB, since we have asked it to use a larger compression quality than the input, so the file size will increase dramatically even though it is a lossy compression.
I am building a map system that requires a large image (native 13K pixels wide by 20K pixels tall) to be overlayed onto an area of the US covering about 20 kilometers or so. I have the file size of the image in jpg format down to 23 MB and it loads onto the map fairly quickly. I can zoom in and out and it looks great. It's even located exactly where I need it to be (geographically). However, that 25 MB file is causing Firefox to consume an additional 1GB of memory!!! I am using Memory Restart extension on Firefox and without the image overlay, the memory usage is about 360 MB to 400 MB, which seems to be about the norm for regular usage, browsing other websites etc. But when I add the image layer, the memory usage jumps to 1.4 GB. I'm at a complete loss to explain WHY that is and how to fix it. Any ideas would be greatly appreciated.
Andrew
The file only takes up 23 MB as a JPEG. However, the JPEG format is compressed, and any program (such as FireFox) that wants to actually render the image has to uncompress it and store every pixel in memory. You have 13k by 20k pixels, which makes 260M pixels. Figure at least 3 bytes of color info per pixel, that's 780 MB. It might be using 4 bytes, to have each pixel aligned at a word boundary, which would be 1040 MB.
As for how to fix it, well, I don't know if you can, except by reducing the image size. If the image contains only a small number of colors (for instance, a simple diagram drawn in a few primary colors), you might be able to save it in some format that uses indexed colors, and then FireFox might be able to render it using less memory per pixel. It all depends on the rendering code.
Depending on what you're doing, perhaps you could set things up so that the whole image is at lower resolution, then when the user zooms in they get a higher-resolution image that covers less area.
Edit: to clarify that last bit: right now you have the entire photograph at full resolution, which is simple but needs a lot of memory. An alternative would be to have the entire photograph at reduced resolution (maximum expected screen resolution), which would take less memory; then when the user zooms in, you have the image at full resolution, but not the entire image - just the part that's been zoomed in (which likewise needs less memory).
I can think of two approaches: break up the big image into "tiles" and load the ones you need (not sure how well that would work), or use something like ImageMagick to construct the smaller image on-the-fly. You'd probably want to use caching if you do it that way, and you might need to code up a little "please wait" message to show while it's being constructed, since it could take several seconds to process such a large image.
Consider an application handling uploading of potentially very large PNG files.
All uploaded files must be stored to disk for later retrieval. However, the PNG files can be up to 30 MB in size, but disk storage limitations gives a maximum per file size of 1 MB.
The problem is to take an input PNG of file size up to 30 MB and produce an output PNG of file size below 1 MB.
This operation will obviously be lossy - and reduction in image quality, colors, etc is not a problem. However, one thing that must not be changed is the image dimension. Hence, an input file of dimension 800x600 must produce an output file of dimension 800x600.
The above requirements outlined above are strict and cannot be changed.
Using ImageMagick (or some other open source tool) how would you go about reducing the file size of input PNG-files of size ~30 MB to a maximum of 1 MB per file, without changing image dimensions?
PNG is not a lossy image format, so you would likely need to convert the image into another format-- most likely JPEG. JPEG has a settable "quality" factor-- you could simply keep reducing the quality factor until you got an image that was small enough. All of this can be done without changing the image resolution.
Obviously, depending on the image, the loss of visual quality may be substantial. JPEG does best for "true life" images, such as pictures from cameras. It does not do as well for logos, screen shots, or other images with "sharp" transitions from light to dark. (PNG, on the other hand, has the opposite behavior-- it's best for logos, etc.)
However, at 800x600, it likely will be very easy to get a JPEG down under 1MB. (I would be very surprised to see a 30MB file at those smallish dimensions.) In fact, even uncompressed, the image would only be around 1.4MB:
800 pixels * 600 pixels * 3 Bytes / color = 1,440,000 Bytes = 1.4MB
Therefore, you only need a 1.4:1 compression ratio to get the image down to 1MB. Depending on the type of image, the PNG compression may very well provide that level of compression. If not, JPEG almost certainly could-- JPEG compression ratios on the order of 10:1 are not uncommon. Again, the quality / size of the output will depend on the type of image.
Finally, while I have not used ImageMagick in a little while, I'm almost certain there are options to re-compress an image using a specific quality factor. Read through the docs, and start experimenting!
EDIT: Looks like it should, indeed, be pretty easy with ImageMagick. From the docs:
$magick> convert input.png -quality 75 output.jpg
Just keep playing with the quality value until you get a suitable output.
Your example is troublesome because a 30MB image at 800x600 resolution is storing 500 bits per pixel. Clearly wildly unrealistic. Please give us real numbers.
Meanwhile, the "cheap and cheerful" approach I would try would be as follows: scale the image down by a factor of 6, then scale it back up by a factor of 6, then run it through PNG compression. If you get lucky, you'll reduce image size by a factor of 36. If you get unlucky the savings will be more like 6.
pngtopng big.png | pnmscale -reduce 6 | pnmscale 6 | pnmtopng > big.png
If that's not enough you can toss a ppmquant in the middle (on the small image) to reduce the number of colors. (The examples are netpbm/pbmplus, which I have always found easier to understand than ImageMagick.)
To know whether such a solution is reasonable, we have to know the true numbers of your problem.
Also, if you are really going to throw away the information permanently, you are almost certainly better off using JPEG compression, which is designed to lose information reasonably gracefully. Is there some reason JPEG is not appropriate for your application?
Since the size of an image file is directly related to the image dimensions and the number of colours, you seem to have only one choice: reduce the number of colours.
And ~30MB down to 1MB is a very large reduction.
It would be difficult to achieve this ratio with a conversion to monochrome.
It depends a lot on what you want at the end, I often like to reduce the number of colors while perserving the size. In many many cases the reduced colors does not matter. Here is an example of reducing the colors to 254.
convert -colors 254 in.png out.png
You can try the pngquant utility. It is very simple to install and to use. And it can compress your PNGs a lot without visible quality loss.
Once you install it try something like this:
pngquant yourfile.png
pngquant --quality=0-70 yourfile.png
For my demo image (generated by imagemagick) the first command reduces 350KB to 110KB, and the second one reduces it to 65KB.
Step 1: Decrease the image to 1/16 of its original size.
Step 2: Decrease the amount of colors.
Step 3: Increase the size of the image back to its original size.
I know you want to preserve the pixel size, but can you reduce the pixel size and adjust the DPI stored with the image so that the display size is preserved? It depends on what client you'll be using to view the images, but most should observe it. If you are using the images on the web, then you can just set the pixel size of the <img> tag.
It depends on they type of image, is it a real life picture or computer generated image,
for real life images png will do very little it might even not compress at all, use jpg for those images, it the image has a limited number of different colors (it can have a 24 bit image depth but the number of unique images will be low) png can compress quite nicely.
png is basicly an implementation of zip for images so if a lot of pixels are the same you can have a rather nice compression ratio, if you need lossless compression don't do resizing.
use optipng it reduce size without loss
http://optipng.sourceforge.net/
Try ImageOptim https://imageoptim.com/mac it is free and open source
If you want to modify the image size in ubuntu, you can try "gimp".
I have tried couple of image editing apps in ubuntu and this seemed to be the best among them.
Installation:
Open terminal
Type: sudo apt install gimp-plugin-registry
Give admin password. You'll need net connection for this.
Once installed, open the image with GIMP image editor. Then go to: File > Export as > Click on 'Export' button
You will get a small window, where check box on "Show preview in image window". Once you check this option, you will get to see the current size of the file along with Quality level.
Adjust the quality level to increase/decrease the file size.
Once adjusting is done, click on 'Export' button finally to save the file.
Right click on the image. Select open with paint. Click on resize. Click on pixel and change the horizontal to 250 or 200.
That's the only thing. It is the fastest way for those who are using Windows XP or Windows 7.