Hadoop Crontab Put - bash

Question

Im trying to program with crontab a simple task, copy some files from local to HDFS. My code is this:
#!/bing/ksh
ANIO=$(date +"%Y")
MES=$(date +"%m")
DIA=$(date +"%d")
HORA=$(date +"%H")
# LOCAL AND HDFS DIRECTORIES
DIRECTORIO_LOCAL="/home/cloudera/bicing/data/$ANIO/$MES/$DIA/stations"$ANIO$MES$DIA$HORA"*"
DIRECTORIO_HDFS="/bicing/data/$ANIO/$MES/$DIA/"
# Test if the destination directory exist and create it if it's necesary
echo "hdfs dfs -test -d $DIRECTORIO_HDFS">>/home/cloudera/bicing/data/logFile
hdfs dfs -test -d $DIRECTORIO_HDFS
if [ $? != 0 ]
then
echo "hdfs dfs -mkdir -p $DIRECTORIO_HDFS">>/home/cloudera/bicing/data/logFile
hdfs dfs -mkdir -p $DIRECTORIO_HDFS
fi
# Upload the files to HDFS
echo "hdfs dfs -put $DIRECTORIO_LOCAL $DIRECTORIO_HDFS">>/home/cloudera/bicing/data/logFile
hdfs dfs -put $DIRECTORIO_LOCAL $DIRECTORIO_HDFS
As you can see is quite simple, it only define the folders variables, create the directory in HDFS (if it doesn't exists) and copies the files from local to HDFS.
The script works if I launch it directly on the Terminal but when I schedule it with Crontab it doesn't "put" the files in HDFS.
Moreover, the script creates a "logFile" with the commands that should have been executed. When I copy them to the Terminal them work perfectly.
hdfs dfs -test -d /bicing/data/2015/12/10/
hdfs dfs -mkdir -p /bicing/data/2015/12/10/
hdfs dfs -put /home/cloudera/bicing/data/2015/12/10/stations2015121022* /bicing/data/2015/12/10/
I have checked the directories and files, but I cant find the key to solve it.
Thanks in advance!!!

Answer 1

When you execute these commands on the console, they run fine, because "HADOOP_HOME" is set. But, when the Cron job runs, most likely, "HADOOP_HOME" environment variable is not available.
You can resolve this problem in 2 ways:
In the script, add the following statements at the beginning. This will add the paths of all the Hadoop jars to your environment.
export HADOOP_HOME={Path to your HADOOP_HOME}
export PATH=$PATH:$HADOOP_HOME\etc\hadoop\;$HADOOP_HOME\share\hadoop\common\*;$HADOOP_HOME\share\hadoop\common\lib\*;$HADOOP_HOME\share\hadoop\hdfs\*;$HADOOP_HOME\share\hadoop\hdfs\lib\*;$HADOOP_HOME\share\hadoop\mapreduce\*;$HADOOP_HOME\share\hadoop\mapreduce\lib\*;$HADOOP_HOME\share\hadoop\tools\*;$HADOOP_HOME\share\hadoop\tools\lib\*;$HADOOP_HOME\share\hadoop\yarn\*;$HADOOP_HOME\share\hadoop\yarn\lib\*
You can also update your .profile (present in $HOME/.profile) or .kshrc (present in $HOME/.kshrc) to include the HADOOP paths.
That should solve your problem.