Pig without Hadoop on Windows 7 - windows

Question

I am trying to run PigUnit tests on a Windows 7 machine before running the actual pig script on a Ubuntu cluster and I start to think that my understanding of "withouthadoop" is not correct.
Do I need to install Hadoop to locally run a PigUnit test on a Windows 7 machine?
I installed:
eclipse Juno & ant
cygwin
I set up:
JAVA_HOME=C:\Program Files\Java\jdk1.6.0_39
PIG_HOME=C:\Users\john.doe\Java\eclipse\pig
PIG_CLASSPATH=%PIG_HOME%\bin
I created using eclipse's Ant builder jar-all and pigunit-jar:
pig.jar
pig-withouthadoop.jar
pigunit.jar
Still when I type pig -x local in cygwin I get:
$./pig -x local
cygpath: can't convert empty path
Exception in thread "main" java.io.IOException: Error opening job jar: /usr/lib/pig/pig-withouthadoop.jar
at org.apache.hadoop.util.RunJar.main(RunJar.java:135)
Caused by: java.io.FileNotFoundException: \usr\lib\pig\pig-withouthadoop.jar (the systen cannot find the given path)
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:127)
at java.util.jar.JarFile.<init>(JarFile.java:136)
at java.util.jar.JarFile.<init>(JarFile.java:73)
at org.apache.hadoop.util.RunJar.main(RunJar.java:133)
When I try to run the test from http://pig.apache.org/docs/r0.10.0/test.html#pigunit from within eclipse using the option "Run as JUnit", I get:
java.io.IOException
at org.apache.pig.pigunit.pig.PigServer.registerScript(PigServer.java:62)
at org.apache.pig.pigunit.PigTest.registerScript(PigTest.java:171)
at org.apache.pig.pigunit.PigTest.assertOutput(PigTest.java:267)
at org.apache.pig.pigunit.PigTest.assertOutput(PigTest.java:262)
at da.utils.pigunit.PigUnitExample.testTop2Queries(PigUnitExample.java:72)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
I am starting to think that I missed some crucial basic information to run Pig on Windows, I have to say also that I am no experienced user with Windows 7 and cygwin, I come from the Unix world.

Answer 1

Don't fight it. Install Hadoop HDInsight server on Windows from the Web Platform installer:
http://www.microsoft.com/web/downloads/platform.aspx
It doesn't take long or take up that much space, and the whole shebang is just set up and running for you. I can't get Pig scripts to take parameters, and there's no HBase, but you get HDFS, Pig, Hive. You can even get a whole local cluster going if just follow:: http://social.msdn.microsoft.com/Forums/en-US/hdinsight/thread/885efc22-fb67-4df8-8648-4ff38098dac6/

Answer 2

I have installed pig 0.12 in cygwin (I run windows 7 64-bit) without installing hadoop. As far as I can see, the steps I followed where:
Install Cygwin64 (with Perl package)
Download pig-0.12.1.tar.gz, copy to home folder
Extract to home folder in cygwin:
$ tar xzf pig-0.12.1.tar.gz
Export JAVA_HOME:
$ export JAVA_HOME=/cygdrive/c/Program\ Files/Java/jre6/
Add pig to path:
$ export PATH=~/pig-0.12.1/bin/:$PATH
Rename pig-0.12.1.jar to pig.jar:
$ cp pig-0.12.1.jar pig.jar
Export PIG_CLASSPATH:
$ export PIG_CLASSPATH=~/pig-0.12.1/pig.jar
Run pig in local mode (start Grunt):
$ pig -x local
There will be a warning:
"cygpath: cannot create short name of C:\cygwin64\home\xxx\pig-0.12.1\logs"
We can remove it simply by running:
$ mkdir logs

Answer 3

Like you, I'm trying at the moment to get a functioning Pig installation on a Windows PC using cygwin in order to learn Pig Latin using small datasets on a single JVM. Not a huge ask, you would have thought, but the pain is almost unbearable. I come from a Windows background and the UNIX part is the steep learning curve for me. The pig-withouthadoop jar doesn't contain hadoop, so hadoop needs to be already installed on your machine to use it; the pig.jar contains pig's own version of hadoop and so is the one to use if hadoop is not already installed on your machine. This is the way I understand it, and it seems to be born out by dumping a list of the contents of each .jar to a text file and viewing the results in Notepad++ .
When you type pig -x local at cygwin's dollar prompt, the bash command script 'pig' is invoked and run. Have a look at it (from your PIG_HOME) with $cd bin $ cat pig. I've been right through it these last few days with vim (!) and near the end of the code is a little fork for cygwin users, in order to cast environment variables, that up until now have been in Unix format, into a form that the Windows version of java.exe will understand when 'exec java ...' is called right at the end of the script. Without this conversion, the Windows java.exe won't understand its parameters:
'#' cygwin path translation
if $cygwin; then
CLASSPATH=cygpath -p -w "$CLASSPATH"
PIG_HOME=cygpath -d "£PIG_HOME"
PIG_LOG_DIR=cygpath -d "$PIG_LOG_DIR"
fi
Cygpath is a cygwin utility that converts UNIX-style file paths into Windows-style file paths, and vice versa. The error message: "cygpath: can't convert empty path" must come from here, I think. Check that CLASSPATH, PIG_HOME and PIG_LOG_DIR aren't empty, perhaps by placing your own echo commands in the script.
On my machine and installation, there was an error generated here, but not the same as yours. I found replacing -w and -d with -m, which makes cygpath use the C:/Program Files/Java... syntax conversion, worked. But then other problems appear, which I shall leave for my own question.

Answer 4

According to this note [1], it is not possible to use the Hadoop native libraries on windows 7 directly using Cygwin:
Hadoop native library is supported only on *nix platforms only. Unfortunately it is known not to work on Cygwin [...]
I have traced down the error message cygpath: can't convert empty path to the line JAVA_LIBRARY_PATH=cygpath -w "$JAVA_LIBRARY_PATH" in hadoop-config.sh which I commented out following the advice from [2]
Comment some translation in hadoop-config.sh.
#if $cygwin; then
#HADOOP_PREFIX=`cygpath -w "$HADOOP_PREFIX"`
#HADOOP_LOG_DIR=`cygpath -w "$HADOOP_LOG_DIR"`
#JAVA_LIBRARY_PATH=`cygpath -w "$JAVA_LIBRARY_PATH"`
#fi
Now I get the following error:
Error before Pig is launched -- ERROR 2999: Unexpected internal error.
java.lang.UnsupportedOperationException: Not implemented by the DistributedFileSystem FileSystem implementation
So the conclusion I draw from this, is that Pig, even in local mode, requires the HDFS. And the HDFS requires the Hadoop native libraries. And the native libraries are known not to work on Cygwin. Hence: IMHO, Pig cannot run using Cygwin as it is.

Answer 5

To run PigUnit on Windows 7. I don't install Cygwin.
Thanks for Konstantin Kudryavtsev i use his FixHadoopOnWindows.runFix()
http://simpletoad.blogspot.com/2013/05/pigunit-issue-on-windows.html
I call the runFix in my setUp, for example
private static PigTest test;
#BeforeClass
public static void setUp() throws IOException, ParseException {
try {
FixHadoopOnWindows.runFix();
// TODO: load pig script properly
test = new PigTest("src/pig/example.pig");
//test.override(...);
}
catch (Exception e) {
}
}
Use maven, need the following dependency
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>1.2.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.pig</groupId>
<artifactId>pig</artifactId>
<version>0.15.0</version>
</dependency>
<dependency>
<groupId>org.jboss.forge</groupId>
<artifactId>forge-javassist</artifactId>
<version>2</version>
</dependency>
<dependency>
<groupId>org.apache.pig</groupId>
<artifactId>pigunit</artifactId>
<version>0.15.0</version>
<scope>test</scope>
</dependency>