Search for a String in 1000 files and each file size is 1GB - bash

Question

I am working on SunOS (which is slightly brain-dead). And below is the Disk Throughput for the above Solaris Machine-
bash-3.00$ iostat -d 1 10
sd0 sd1 sd2 sd3
kps tps serv kps tps serv kps tps serv kps tps serv
0 0 0 551 16 8 553 16 8 554 16 8
0 0 0 701 11 25 0 0 0 1148 17 33
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
Problem Statement
I have around 1000 files and each file is of the size of 1GB. And I need to find a String in all these 1000 files and also which files contains that particular String. I am working with Hadoop File System and all those 1000 files are in Hadoop File System.
All the 1000 files are under real-time folder, so If I do like this below, I will be getting all the 1000 files. And I need to find which files contains a particular String.
bash-3.00$ hadoop fs -ls /apps/technology/b_dps/real-time
So for the above problem statement, I am using the below command that will find all the files which contains the particular string-
hadoop fs -ls /apps/technology/b_dps/real-time | awk '{print $8}' | while read f; do hadoop fs -cat $f | grep cec7051a1380a47a4497a107fecb84c1 >/dev/null && echo $f; done
So in the above case it will find all the files which contains this string cec7051a1380a47a4497a107fecb84c1. And it is working fine for me and I am able to get the file names which contains the particular string.
My Question is-
But the problem with above command is, it is very very slow. So is there any way we can parallelize the above command or make the above command to search the files a lot faster?
Any suggestions will be appreciated.

Answer 1

You could write a simple MapReduce job to achieve this if you want. You don't actually need any reducers though, so the number of reducers would be set to zero. This way you can make use of the parallel processing power of MapReduce and chunk though the files much faster than a serial grep.
Just set up a Mapper that can be configured to search for the string you want. You will probably read in the files using the TextInputFormat, split the line and check for the values you are searching for. You can then write out the name of the current input file for the Mapper that matches.
Update:
To get going on this you could start with the standard word count example: http://wiki.apache.org/hadoop/WordCount. You can remove the Reducer and just modify the Mapper. It reads the input a line at a time where the line is contained in the value as a Text object. I dont know what format your data is, but you could even just convert the Text to a String and hardcode a .contains("") against that value to find the String you're searching for (for simplicity, not speed or best practice). You just need to workout which file the Mapper was processing when you get a hit and then write out the files name.

Answer 2

You can get a hint from grep class. It comes with the distribution in the example folder.
./bin/hadoop jar hadoop-mapred-examples-0.22.0.jar grep input output regex
For details source on the implementation of this class you can go to the directory. "src\examples\org\apache\hadoop\examples" that comes with the distribution
So you can do this in your main class:
Job searchjob = new Job(conf);
FileInputFormat.setInputPaths("job Name", "input direcotory in hdfs");
searchjob.setMapperClass(SearchMapper.class);
searchjob.setCombinerClass(LongSumReducer.class);
searchjob.setReducerClass(LongSumReducer.class);
In your SearchMapper.class you can do this.
public void map(K key, Text value,
OutputCollector<Text, LongWritable> output,
Reporter reporter)
throws IOException {
String text = value.toString();
Matcher matcher = pattern.matcher(text);
if(matcher.find()) {
output.collect(key,value);
}

Answer 3

If you have 1000 files, is there any reason to use a finely-grained parallelized technique? Why not just use xargs, or gnu parallel, and split the work over the files, instead of splitting the work within a file?
Also it looks like you are grepping a literal string (not a regex); you can use the -F grep flag to search for string literals, which may speed things up, depending on how grep is implemented/optimized.
I haven't worked with mapReduce specifically, so this post may or may not be on point.