Sampling Records from Hadoop Mapper - hadoop

Question

I have a dataset whose key consists of 3 parts: a, b and c. In my mapper, I would like to emit records with the key as 'a' and the value as 'a,b,c'
How do I emit 10% of the total records for each 'a' that is detected from the mapper in Hadoop? Should one consider saving the total number of records seen for each 'a' from a previous Map-Reduce job in a temp file?

Answer 1

If you want close to 10%, you can use Random. Here is an example of Mapper:
public class Test extends Mapper<LongWritable, Text, LongWritable, Text> {
private Random r = new Random();
#Override
public void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
if (r.nextInt(10) == 0) {
context.write(key, value);
}
}
}

Answer 2

Use this java code to select 10% randomly:
import java.io.IOException;
import java.util.*;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.*;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
public class RandomSample {
public static class Map extends Mapper<LongWritable, Text, Text, Text> {
private Text word = new Text();
public void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
if (Math.random()<0.1)
context.write(value,null);
else
context.write(null,null);
context.write(value,null);
}
}
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = new Job(conf, "randomsample");
job.setJarByClass(RandomSample.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
job.setNumReduceTasks(0);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);
}
}
And use this bash script to run it
echo "Running Job"
hadoop jar RandomSample.jar RandomSample $1 tmp
echo "copying result to local path (RandomSample)"
hadoop fs -getmerge tmp RandomSample
echo "Clean up"
hadoop fs -rmr tmp
For example, if we name the script random_sample.sh, to select 10% from folder /example/, simply run
./random_sample.sh /example/