Hadoop Map Task Fails When Using Compression In 2 Node Cluster. But Both Node Works Fine When Running as Single Node - hadoop

Question

Node1 : hadoop2.5.2 RedhatLinux.el6 64bit
build 64bit native library and it's working
Node2 : hadoop2.5.2 RedhatLinux.el5 32bit
build 32bit native library and it's working
when running map reduce task as single node it works(with compression)
as multinode also it's working (without compression)
but as multinode with compression it's not working....
map task only finishing in one of the node(somtimes in node1, sometime in node2) in other node it is failed with error and job got failed.
Error: java.io.IOException: Spill failed at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.checkSpillException(MapTask.java:1535)
at . . Caused by: java.lang.RuntimeException: native lz4 library not
available at
org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:124)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
at
i tried
<name>mapreduce.admin.user.env</name>
<value>LD_LIBRARY_PATH=$HADOOP_HOME/lib/native</value>
in mapred-site.xml
but still not working...
please suggest a solution...

Answer 1

Adding these properties in mapred-site.xml of the Hadoop node, in which the job is submitting solved the problem.
<property>
<name>yarn.app.mapreduce.am.admin.user.env</name>
<value>LD_LIBRARY_PATH={{HADOOP_COMMON_HOME}}/lib/native</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>LD_LIBRARY_PATH={{HADOOP_COMMON_HOME}}/lib/native</value>
</property>
<property>
<name>mapreduce.admin.user.env</name>
<value>LD_LIBRARY_PATH={{HADOOP_COMMON_HOME}}/lib/native</value>
</property>

Answer 2

Enable debug logs for hadoop in the machine where the exception is thrown.
restart hadoop process, post that you should be able to figure out based on logs of NativeCodeLoader as why native library is not loaded.
you can use below command to verify if native libraries are loaded or not.
hadoop checknative -a