Cloudera's CDH4 WordCount hadoop tutorial - issues - hadoop

Question

I am following this tutorial:
http://www.cloudera.com/content/cloudera-content/cloudera-docs/HadoopTutorial/CDH4/Hadoop-Tutorial/ht_topic_5_2.html
It says the following:
javac -cp classpath -d wordcount_classes WordCount.java
where classpath is:
CDH4 - /usr/lib/hadoop/*:/usr/lib/hadoop/client-0.20/*
CDH3 - /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u4-core.jar
I have downloaded the "cloudera-quickstart-demo-vm-4.2.0-vmware" .
Running as user cloudera.
[cloudera#localhost wordcount]$ javac -cp /usr/lib/hadoop/*:/usr/lib/hadoop/client-0.20/* -d wordcount_classes WordCount.java
incorrect classpath: /usr/lib/hadoop/*
incorrect classpath: /usr/lib/hadoop/client-0.20/*
----------
1. ERROR in WordCount.java (at line 8)
import org.apache.hadoop.fs.Path;
^^^^^^^^^^
When checking the cp folder: .
[cloudera#localhost wordcount]$ ls -l /usr/lib/hadoop
total 3500
drwxr-xr-x. 2 root root 4096 Apr 22 14:37 bin
drwxr-xr-x. 2 root root 4096 Apr 22 14:33 client
drwxr-xr-x. 2 root root 4096 Apr 22 14:33 client-0.20
drwxr-xr-x. 2 root root 4096 Apr 22 14:36 cloudera
drwxr-xr-x. 2 root root 4096 Apr 22 14:30 etc
-rw-r--r--. 1 root root 16536 Feb 15 14:24 hadoop-annotations-2.0.0-cdh4.2.0.jar
lrwxrwxrwx. 1 root root 37 Apr 22 14:30 hadoop-annotations.jar -> hadoop-annotations-2.0.0-cdh4.2.0.jar
-rw-r--r--. 1 root root 46855 Feb 15 14:24 hadoop-auth-2.0.0-cdh4.2.0.jar
lrwxrwxrwx. 1 root root 30 Apr 22 14:30 hadoop-auth.jar -> hadoop-auth-2.0.0-cdh4.2.0.jar
-rw-r--r--. 1 root root 2266171 Feb 15 14:24 hadoop-common-2.0.0-cdh4.2.0.jar
-rw-r--r--. 1 root root 1212163 Feb 15 14:24 hadoop-common-2.0.0-cdh4.2.0-tests.jar
lrwxrwxrwx. 1 root root 32 Apr 22 14:30 hadoop-common.jar -> hadoop-common-2.0.0-cdh4.2.0.jar
drwxr-xr-x. 3 root root 4096 Apr 22 14:36 lib
drwxr-xr-x. 2 root root 4096 Apr 22 14:33 libexec
drwxr-xr-x. 2 root root 4096 Apr 22 14:31 sbin
What am I doing wrong?
This is directly from the Cloudera Quickstart VM with CDH4 installed.
Following the "Hadoop Tutorial" .
It even says
**Prerequisites**
Ensure that CDH is installed, configured, and running. The easiest way to get going quickly is to use a CDH4 QuickStart VM
Which is exactly from where I am running this tutorial from - the CDH4 QuickStart VM.
What am I doing wrong?
*update
Version information;
[cloudera#localhost cloudera]$ cat cdh_version.properties
# Autogenerated build properties
version=2.0.0-cdh4.2.0
git.hash=8bce4bd28a464e0a92950c50ba01a9deb1d85686
cloudera.hash=8bce4bd28a464e0a92950c50ba01a9deb1d85686
cloudera.base-branch=cdh4-base-2.0.0
cloudera.build-branch=cdh4-2.0.0_4.2.0
cloudera.pkg.version=2.0.0+922
cloudera.pkg.release=1.cdh4.2.0.p0.12
cloudera.cdh.release=cdh4.2.0
cloudera.build.time=2013.02.15-18:39:29GMT
cloudera.pkg.name=hadoop
CLASSPATH ENV:
[cloudera#localhost bin]$ echo $CLASSPATH
:/usr/lib/hadoop/*:/usr/lib/hadoop/client-0.20/*
EDIT!!
So I think I figured it out.
This is a new issue possibly with the Cloudera CD4 VM quickstart VM:
from: This Post dated yesterday
Another person was having the exact same problem.
It seems the javac program does not accept wildcards properly on exported paths.
I had to do the following:
export CLASSPATH=/usr/lib/hadoop/client-0.20/\*:/usr/lib/hadoop/\*
Then
javac -d [Without a -cp override]
javac -d wordcount_classes/ WordCount.java
Only warnings will appear.
I wonder if Cloudera has to fix their quickstart VM.

Answer 1

You need to have a classpath variable set that includes those directories in /usr/lib/hadoop if you want javac to find them. You can set this env var as follows
$: export CLASSPATH=$CLASSPATH:/usr/lib/hadoop/*:/usr/lib/hadoop/client-0.20/*
javac will now find those libs. If you have any additional complaining regarding classpath variables you can just append them to the above line using a colon (:) as a delimiter
You could include this in a bash script, but it is best practice to set the correct env variables at runtime, then you get exactly what you want. In this case it could be word count or the CDH4 env is setting it, but it is best to just set it yourself.

Answer 2

I spent some time searching for a response to the same issue (also using VM with CDH4) so I shall leave my solution here in hopes that it might help others.
Unfortunately, neither of the solutions above worked in my case.
However, I was able to successfully compile the example by closing my terminal and opening a new one. My issue was having previously switched to the 'cloudera' user with 'sudo su cloudera' as mentioned in the tutorial.
Reference:
http://community.cloudera.com/t5/Apache-Hadoop-Concepts-and/Classpath-Problem-on-WordCount-Tutorial-QuickStart-VM-4-4-0-1/td-p/3613