HDFS distributed reads without Map/Reduce - hadoop

Question

Is it possible to achieve distributed reads from HDSF cluster using an HDFS client on one machine?
I have carried out an experiment with a cluster consisting of 3 data nodes (DN1,DN2,DN3). Then I run 10 simultaneous reads from 10 independent files from a client program located on DN1, and it appeared to be only reading data from DN1. Other data nodes (DN2,DN3) were showing zero activity (judging from debug logs).
I have checked that all files' blocks are replicated across all 3 datanodes, so if I shut down DN1 then data is read from DN2 (DN2 only).
Increasing the amount of data read did not help (tried from 2GB to 30GB).
Since I have a need to read multiple large files and extract only a small amount of data from it (few Kb), I would like to avoid using map/reduce since it requires settings up more services and also requires writing the output of each split task back to HDFS. Rather it would be nice to have the result streamed directly back to my client program from the data nodes.
I am using SequenceFile for reading/writing data, in this fashion (jdk7):
//Run in thread pool on multiple files simultaneously
List<String> result = new ArrayList<>();
LongWritable key = new LongWritable();
Text value = new Text();
try(SequenceFile.Reader reader = new SequenceFile.Reader(conf,
SequenceFile.Reader.file(filePath)){
reader.next(key);
if(key.get() == ID_I_AM_LOOKING_FOR){
reader.getCurrentValue(value);
result.add(value.toString());
}
}
return result; //results from multiple workers are merged later
Any help appreciated. Thanks!

Answer 1

I'm afraid the behavior you see is by-design. From Hadoop document:
Replica Selection
To minimize global bandwidth consumption and read latency, HDFS tries
to satisfy a read request from a replica that is closest to the
reader. If there exists a replica on the same rack as the reader node,
then that replica is preferred to satisfy the read request. If angg/
HDFS cluster spans multiple data centers, then a replica that is
resident in the local data center is preferred over any remote
replica.
It can be further confirmed by corresponding Hadoop source code:
LocatedBlocks getBlockLocations(...) {
LocatedBlocks blocks = getBlockLocations(src, offset, length, true, true);
if (blocks != null) {
//sort the blocks
DatanodeDescriptor client = host2DataNodeMap.getDatanodeByHost(
clientMachine);
for (LocatedBlock b : blocks.getLocatedBlocks()) {
clusterMap.pseudoSortByDistance(client, b.getLocations());
// Move decommissioned datanodes to the bottom
Arrays.sort(b.getLocations(), DFSUtil.DECOM_COMPARATOR);
}
}
return blocks;
}
I.e., all available replicas are tried one after another if former one fails but the nearest one is always the first.
On the other hand, if you access HDFS files through HDFS Proxy, it does pick datanodes randomly. But I don't think that's what you want.

Answer 2

In addition to what Edwardw said note that your current cluster is very small (just 3 nodes) and in this case you see the files on all the nodes. This happens because the default replication factor of Hadoop is also 3. In a larger cluster your files will not be available on each node and so accessing multiple files is likely to go to different nodes and spread the load.
If you work with smaller datasets you may want to look at HBase which lets you work with smaller chunks and spread the load between nodes (by splitting regions)

Answer 3

I would tell that your case sounds good for MR. If we put aside particular MR computational paradigm we can tell that hadoop is built to bring code to the data, instead of opposite. Moving code to the data is essential to get scalable data processing.
In other hand - setting up MapReduce is easier then HDFS - since it store no state between jobs.
In the same time - MR framework will care about parallel processing for you - something it will take time to do properly.
Another point- if results of data processing is so small - there will be no significant performance impact if you will combine them together in reducer.
In other words - I would suggest to reconsider use of the MapReduce.