I have two thousand large (20mb) image files. Each file has the same height, width, and resolution. The files are currently JPGs, though they are available in other formats.
For each image, I need to extract 6 cropped images from the larger file, each of which are in exactly the same location across the larger files.
I am looking to provide the coordinates of the regions I need extracted (or that I need each larger image cropped to) and then to loop through all 2,000 of the larger images and extract the size sub-images (specified by coordinates), from the larger images.
Is there a program, software package, or straight forward way to do this in python, Java, or some other language?
It seems that ImageMagick is exactly what you need.
ImageMagick has interfaces for Python, C and C++, Perl and Java. See full list of APIs on the page: http://www.imagemagick.org/script/api.php
Also uou can use mogrify utility (often comes with ImageMagick package), which can process images massively. Then your command line might look like this:
mogrify -crop 420x400+365+25 -path '/home/user/images' *.jpg
Related
I am searching for a library (in any language) that is capable of reading a region of an image file (any format) without having to initially read that entire image file.
I have come across a few options such as vips, which does indeed not keep the entire image in memory, but still seems to need to read it entirely to begin with.
I realize this may not be available for compressed formats such as jpegs, but in theory it sounds like bmps or tiffs should allow for this type of reading.
libvips will read just the part you need, when it can. For example, if you crop 100x100 pixels from the top-left of a large PNG, it's fast:
$ time vips crop wtc.png x.jpg 0 0 100 100
real 0m0.063s
user 0m0.041s
sys 0m0.023s
(the four numbers are left, top, width, height of the area to be cropped from wtc.png and written to x.jpg)
But a 100x100 pixel region from near the bottom is rather slow, since it has to read and decompress the pixels before the pixels you want to get to the right point in the file:
$ time vips crop wtc.png x.jpg 0 9000 100 100
real 0m3.063s
user 0m2.884s
sys 0m0.181s
JPG and strip TIFF work in the same way, though it's less obvious since they are much faster formats.
Some formats support true random-access read. For example, tiled TIFF is fast everywhere, since libvips can use libtiff to read only the tiles it needs:
$ vips copy wtc.png wtc.tif[tile]
$ time vips crop wtc.tif x.jpg 0 0 100 100
real 0m0.033s
user 0m0.013s
sys 0m0.021s
$ time vips crop wtc.tif x.jpg 0 9000 100 100
real 0m0.037s
user 0m0.021s
sys 0m0.017s
OpenSlide, vips, tiled OpenEXR, FITS, binary PPM/PGM/PBM, HDR, RAW, Analyze, Matlab and probably some others all support true random access like this.
If you're interested in more detail, there's a chapter in the API docs describing how libvips opens a file:
http://libvips.github.io/libvips/API/current/How-it-opens-files.md.html
Here's crop plus save in Python using pyvips:
import pyvips
image = pyvips.Image.new_from_file(input_filename, access='sequential')
tile = image.crop(left, top, width, height)
tile.write_to_file(output_filename)
The access= is a flag that hints to libvips that it's OK to stream this image, in case the underlying file format does not support random access. You don't need this for formats that do support random access, like tiled TIFF.
You don't need to write to a file. For example, this will make a buffer object containing the file encoded as a JPG:
buffer = tile.write_to_buffer('.jpg', Q=85)
Or this will write directly to stdout:
target = pyvips.Target.new_from_descriptor(0)
tile.write_to_target('.jpg', Q=85)
The Q=85 is an optional argument to set the JPG Q factor. You can set any of the file save options.
ITK can do it with some formats. There is a method CanStreamRead which returns true for formats which support streaming, such as MetaImageIO. An example can be found here. You can ask more detailed questions on ITK's forum.
If have control over the file format, I would suggest you use tiled TIFF files. These are typically used in digital pathology whole slide images, with average sizes of 100kx30k pixels or so.
LibTiff makes it easy to read the tiles corresponding to a selected ROI. Tiles can be compressed without making it less efficient to read a small region (no need to decode whole scan lines).
The BMP format (uncompressed) is simple enough that you can write the function yourself.
TIFF is a little less easy, as there are so many subformats. But the TIFF library (TIFFlib) supports a "tile-oriented" I/O mode. http://www.libtiff.org/libtiff.html#Tiles
I don't know of such a library solution.
Low level, file-read access is format specific and in particular, file mapping is OS specific.
If you have access to the raw bytes then assuming you know the width, height, depth and number of channels etc. then calculating file offsets is trivial so just roll your own.
If you're transferring the extracted data over a network you might consider compressing the extracted ROI in-memory if it's relatively big before sending it over the network.
I would like to split large pictures, but they are two big to be opened for my GPU. Max JPEG size i can completly open is over 35Mo and 150,000,000px, with any program tried, using almost all resources. Windows affords opening heavier images after resizing them, so actually there is no more difference between 20Mo and 200Mo pictures on screen. Usual image editors do not try to resize them but they get my computer crash. The only way i found to open completly and safely is into.. a browser.
I also found two useful websites for online splitting but none of them handles pictures larger than 20Mo.
how can one split an image when it is too large to be opened, even on specialized websites ?
Not sure why you would use Chrome or anything web-based, as your uncompressed image is going to require 3+GB of RAM to decompress and get started on.
I would use ImageMagick like this:
convert -crop 10x10# input.png +repage out%02d.png
that will get you 100 tiles, each 2,160 pixels square, called out00.png to out99.png
In the past i have used PDF images of vector files in an NSImage, the advantage being that i can scale them without losing quality. I know that people usually use jpg and png files, why is this? Do PDF files significantly reduce performance or is there some other reason?
Thank you in advance,
Ben
It depends on what's in your PDF file. If there's enough going on in it, then yeah, a raster image may be faster. The trade-off is, of course, scalability—you end up needing to create 1x and 2x variants for every destination size, or create an icon family (if appropriate), instead of just using one image for everything.
But I think most people create raster resources because that's the sort of tool they're used to: Photoshop, Pixelmator, or Acorn. Not many people use vector editors or write their art in PostScript. (And the field of vector editors available on the Mac is pretty weak.)
My recommendation for a few years now has been an app called Opacity. It's vector-focused, but can export raster images in multiple sizes, PDFs, and even source code.
I use PDF files too, for precisely the same reason that they scale automatically. Apple do the same (look inside the Xcode.app bundle - you won't find much other than .pdf files).
There is no reason to use .jpg or .png files at all.
I need a batch process for making mobile images and decided to use ImageMagick, but unfortunately one of my requirements is that the images produced are the same across OS's, since I'm sending them back and forth between my local system (Windows) and the server (Linux). It seems however whenever I call
convert test.jpg -resize 25% test-small.jpg
the process creates different images on both machines. I know this because when I use checksum the values aren't exactly the same.
Does anyone know of any reason why this would happen? And maybe some way around it, either via using a different executable or passing in a parameter that would produce the same images across OS's?
The files have more than the pixels in them -- If you are going to compare the images, write a checksum that works on just the decoded pixel data. That will at least tell you if the images would look the same. The internals of the file could be different because of a lot of factors.
Resizing is dependent on float arithmetic and you can't count on that to be the same across machines. So instead of using just a checksum, you might want to see if each pixel is within a tolerance from the associated one in the other file.
Take a look at these links:
http://pdiff.sourceforge.net/
http://www.phpied.com/image-diff/
JPEG algorithms are non-deterministic. There is no way to ensure that the same image will be generated across two systems, or even between two invocations on the same system.
Relying on 'checksum' or 'md5sum' or similar to compare two images isn't a wise choice. This can only verify if the files are indeed identical. However, if you have different results, this could be caused by just one byte in some random meta data value being different (like a simple timestamp), while there's no pixel difference at all.
To discover the pixel differences between two images, you can use ImageMagick's compare like this:
compare image1.jpg image2.jpg delta.jpg
For colored input images, the resulting delta.jpg willl use image1.jpb as a light-gray background and display the differences in red color. To get a red+white delta image without the light-gray background, use
compare image1.jpg image2.jpg -compose src delta.jpg
Examples images of this techniq can be found here:
ImageMagick: “Diff” an Image
Left: Image with text Center: Original image Right: Differences (=text) in red pixels.
I have a small script that I use to resize all of the images in a directory. I run this script in cygwin and it uses "convert" to do the image resizing. The images change their resolution just fine, but I am having problems with file sizes after the script is run.
I typically use this script to resize images dumped out from a Powerpoint presentation to use in a little web presentation app that I wrote. When I dump out gif's and run the script, the files more than double in size (ex. 8KB to 18KB; 14KB to 50KB)
The pertinent lines of the script are as follows:
/usr/bin/convert $holdfile -thumbnail x480 temp.GIF
mv temp.GIF $i
Is there a switch to prevent the file sizes from growing so much? I know that the file sizes are not huge, but when I have a good number of people connecting to a presentation or the unavoidable dialup users, I just want to make their experience as nice as possible.
Edit: I should have specified that the files start at a 960px x 720px resolution and are being resized to 640px x 480px.
Well, this can happen if convert compresses worse than the input files. Since the exact same compression scheme might yield different results depending how good the compressing code is this can happen.
Another, more likely option here would probably be that you are resizing the images which will probably be done with bicubic resizing. This causes the edges of text or drawings to become a little bit blurry. This means they use up more colors and compress worse.
Also likely would be that your original images use an optimized color palette, maybe just with a few colors and after resizing they need the full 256 colors which are supported by a single GIF frame, due to smoothing done by the resizing.
In any case, you probably should see better performance using PNG instead of GIF. PNG was designed as a modern replacement for GIF and no (graphical) browser in use today has problems displaying PNGs (without an alpha channel). PNG compresses much better than GIF, and allows more colors at the same time. Also there are tools like optipng which will compress PNG images even further.
Convert automatically optimizes the palette however the palette might be growing due to colors being blended during resize. You should be able to inspect the source and resultant images in a graphics program and see the number of colors.
GIF only supports LZW compression but due to patent restraints that have since expired (the last was 2004) it was once necessary to manually enable LZW compression. I'm not sure if that is still the case however it's worth looking into.
If LZW compression is specified but
LZW compression has not been enabled,
the image data is written in an
uncompressed LZW format that can be
read by LZW decoders. This may result
in larger-than-expected GIF files.
- imagemagick.org