passing set of images as input to mapreduce - hadoop

Question

I have a system where I get images (jpg) from some module. I get images for 10 objects (1000 images for single object) at a time (total 10000 images at a time). I need to do some processing on these images using Hadoop cluster.
I am wondering how should I go about this. Like how should I form the input. I would like to process one object (and its images = 1000) completely in one mapper or reducer. For ex: first object in first mapper, second object in second mapper etc.
Some of the approaches that come to my mind are:
1. For each object create a directory and place all its images in that. Then tar, compress the directory and this will go as one input to a single mapper.
Do the same thing as mentioned above, but just tar the file (dont compress). Implement InputFormat interface and make "isSplittable()" return false.
Create sequencefile for each object. Sequensfile would contain a key - value pair for each object image. Here I am not sure how to tell MapReduce to give the sequencefile to just one mapper.

Answer 1

Here I am not sure how to tell MapReduce to give the sequencefile to just one mapper.
FileInputFormat#isSplitable is your friend here for all the file input formats. SequenceFileInputFormat extends FileInputFormat.

Answer 2

PROCESSING IMAGES IN HADOOP USING MAPREDUCE
HIPI:
Hipi is Hadoop Image Processing Interface. This provides set of tools and Inputformat to process bulk amount of images using Hadoops Distributes File System (HDFS) and Mapreduce.
STPES INVOLVED:
In hipi the entire process can be categorized in to 2 parts.
1) Converting all images into a bulk file(HIPI Image Bundle).
2) Processing the created bulk file of image using HIPI's image input formats.
The cull (culler class) is use to filter out images with low clarity or defects
ISSUES WITH HIPI:
To simulate my bulk image processing scenario , I used a java program to create multiple copies of same image with different names in a single directory. then by using HIPI's utility I converted all images into a bulk file (known as HIP file in HIPI ).
To check whether all images are exist in the bulk file I done the reverse process (Converted HIP file into multiple images). There is also another utility to do the same. But I didn't got all images back, and found that using HIPI I am loosing some images.
I couldn't proceed with my POC by using HIPI, and thought of creating a new framework to process bulk images using mapreduce.
NEW IMAGE PROCESSING FRAMEWORK:
In order to avoid spawning multiple maps (each per file) we must do as like HIPI does, that is , to convert all images into a single bundle file.
This bundle file is given as the input to map-reduce. The image input format parse the bundle file and create Buffered Image Object corresponding to each image.
IMAGE INPUT FORMAT-IMPORTANT CLASSES:
Image Combiner:
Merges multiple images in to a single bundle file.
ImageInputFormat :
Return ImageRecordRreader, and manages splits
ImageRecordReader:
Manage reading each splits.
Perform initial seek of the file pointer to the start of each split.
nextKeyValue() method reads each Image from the split and convert to BufferedImage.
BufferedImageWritable:
Since the key value classes of map reduce should be a writable serializable type we cannot keep BufferedImage directly as value in the map method.
This is a wrapper class that just holds the BufferedImage in it.
BufferedImageWritable
{
BufferedImage img;
#Override
public void readFields(DataInput arg0) throws IOException {
}
#Override
public void write(DataOutput arg0) throws IOException {
}
#Override
public int compareTo(byte[] o) {
return 0;
}
}
Not implemented readFiled(),write(),compareTo() methods since in my scenario I dont want to store the mages back to HDFS.
If you want to write back any images in HDFS(In map or reduce), you may have to implement all these methods. in write() you may need to write the logic to store the image as like we used to write images while creating the bulk file. The readFiled() should contain the opposite logic of write(). compareTo() is no need to implement since we never keep this image as key in map-reduce (compareTo() is invoked during the sort face of map-reduce).
Since you are getting image as BufferedImages (a common java class for image processing) it is easy to do most of the operations easily on it. But in the case of HIPI the image is available in map's value as Hipi's FloatImage class, and you might feel difficulty in doing manipulations on top of it.
I have successfully implemented facedetection program using this custom input format and OpenCV.
The code that I used to develop the same will be shared soon on GitHub
http://worldofbigdata-inaction.blogspot.in/2017/02/processing-images-in-hadoop-using.html