How can i emmit key values in the end of the whole file processing? - hadoop

Question

Mapper reads lines from file... How can i emmit key values in the end, after the whole scanning of the file and not per line?

Answer 1

Using the new mapreduce API, you can override the Mapper.cleanup(Context) method and use Context.write(K, V) as you normally would in the map method.
#Override
protected void cleanup(Context context) {
context.write(new Text("key"), new Text("value"));
}
The old mapred API you can override the close() method - but you'll need to store a reference to the OutputCollector given to the map method:
private OutputCollector cachedCollector = null;
void map(Longwritable key, Text value, OutputCollector outputCollector, Reporter reporter) {
if (cachedCollector == null) {
cachedCollector = outputCollector;
}
// ...
}
public void close() {
cachedCollector.collect(outputKey, outputValue);
}

Answer 2

Do you have one Key value for whole file or multiple?
If it is case #1:
Use WholeFileInputFormat. You will receive full file content as a single record. You can split this into records, process all the records and emit final Key/Value at the end of your processing
Cae #2:
Use same fileInputFormat. Store all key values in a temp storage. At the end, access your temp storage and emit whatever the Key/values you want and suppress those you don't want

Answer 3

Another alternative to Chris's answer could be that you can achieve this by overriding the run() of the Mapper class (New API)
public static class Map extends Mapper<IntWritable, IntWritable, IntWritable, IntWritable> {
//map method here
// Override the run()
#override
public void run(Context context) throws IOException, InterruptedException {
setup(context);
while (context.nextKeyValue()) {
map(context.getCurrentKey(), context.getCurrentValue(), context);
}
// Have your last <key,value> emitted here
context.write(lastOutputKey, lastOutputValue);
cleanup(context);
}
}
And in order to ensure that each mapper gets one file to process, you have to create your own version of FileInputFormat and override isSplittable(), like this:
Class NonSplittableFileInputFormat extends FileInputFormat{
#Override
public boolean isSplitable(FileSystem fs, Path filename){
return false;
}
}