I am trying to follow the v 1.2.3 tutorial for apache storm.
Trying to get the ExclamationTopology example from storm starter running.
I installed storm on my macbook using brew.
Which meant I ended up with apache storm v 1.2.3
(And that would suit me fine cos I'm trying to get to a situation where I understand stormcrawler which seems to use 1.2.3 anyway.)
So, I spent a long time trying to see if I could view the output from the example and finally think I found the output in:
/usr/local/Cellar/storm/1.2.3/libexec/logs/worders-artifacts
First question.. is that the right place to find the output?
Second Question:
the log file there is just full of messages as follows:
0.003s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:artifacts/gc.log instead. Unrecognized VM option 'PrintGCDateStamps' Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
I had tried to get this whole lot running with jdk 1.8 as 1.7 was giving errors... so would that explain why this error is occurring ?
Any help or advice would be greatly appreciated...
The directory layout for the brew install of Storm looks to be a little different than a standard Storm zip install. You are probably looking in the right place, yes.
The error message you're seeing is because you're running with some JDK version greater than 9. You can either use an older JDK (e.g. 8), or upgrade Storm to 2.0.0. Finally you can also work around the issue by editing/adding the worker.childopts option in your storm.yaml, which is somewhere in your storm install directory. Something like:
worker.childopts: "-Xmx%HEAP-MEM%m -Xlog:gc*:artifacts/gc.log:time,uptime:filecount=10,filesize=1M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=artifacts/heapdump"
should work (this is the default in Storm 2.0.0).
What is the difference between
giraph-dist-1.2.0-hadoop2-bin.tar.gz and giraph-dist-1.2.0-bin.tar.gz.
Is there any documentation about that?
The only documentation that I found is the following one:
Apache Hadoop 2 (latest version: 2.5.1)
This is the latest version of Hadoop 2 (supporting YARN in addition
to MapReduce) Giraph could use. You may tell maven to use this version
with "mvn -Phadoop_2 ".
After compiling Hadoop 2.5.1 with maven
hadoop version
Hadoop 2.5.1, I tried to compile apache spark using the following command:
mvn -Pyarn -Phadoop-2.5 -Dhadoop.version=2.5.1 -Pdeb -DskipTests clean package
But apparently there is no 2.5 profile.
My question is : what should I do?
rebuild hadoop 2.4
or compile spark with profile 2.4
or any other solution ?
Looks like this was asked after the poster inquired:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-1-0-with-Hadoop-2-5-0-td15827.html
"The hadoop-2.4 profile is really intended to be "Hadoop 2.4+". It
should compile and run fine with Hadoop 2.5 as far as I know. CDH 5.2
is Hadoop 2.5 + Spark 1.1, so there is evidence it works."
Just changing the profile name worked for me.
Thx for the answers.
Hadoop 2.2.0 and HBase 0.98.0-hadoop2 are compatible ?
EDIT: http://hbase.apache.org/book/configuration.html already read.
No, they are not compatible.
Read this release.
I installed Hadoop (1.0.2) for a single node on Windows 7 with Cygwin, and it is working. However, I cannot get PIG (0.10.0) to see the Hadoop.
1) "Error: JAVA_HOME is not set."
I added this line to pig (under bin): export JAVA_HOME=/cygdrive/c/PROGRA~1/Java/jdk1.7.0_05
2) which: no hadoop in (/usr/local/b.....)
cygpath: cannot create short name of C:\pig-0.10.0\logs
Cannot locate pig.jar. do 'ant jar', and try again
I tried adding below lines to pig and it is still not finding hadoop. What should i do?
export PIG_HOME="/cygdrive/c/pig-0.10.0"
export PATH=$PATH:$PIG_HOME/bin
export PIG_CLASSPATH=/cygdrive/hadoop/hadoop-1.0.2/conf
You might need to add your Hadoop install to your path as well. e.g.
export HADOOP_INSTALL=/Users/yourname/dev/hadoop-0.20.203.0
export PATH=$PATH:$HADOOP_INSTALL/bin
I had same issue with pig-0.11. Seems this is cygwin specific issue.
Copying pig-0.11.1-withouthadoop to pig-withouthadoop.jar under PIG_HOME fixed the issue for me
I was trying to set up PIG on my gateway machine which has Windows 7 installed on it.
This issue is very specific to Cygwin.
After breaking my head for a couple of hours I found the solution :
Solution is very simple.
Just rename the jar file under ”pig-0.10.1-withouthadoop.jar” to “pig-withouthadoop.jar”.
Its documented here
Also, you can add path : (hadoop directory)\hadoop-v.v.v\bin to environment variables manually in Windows 7. This will solve this problem
which: no hadoop in (/usr/local/b.....)
You must visit this for installing pig 12 on hadoop 2.2.0 without any errors as it re compiles the pig library for hadoop version specified.
http://javatute.com/javatute/faces/post/hadoop/2014/installing-pig-11-for-hadoop-2-on-ubuntu-12-lts.xhtml
After following the steps, you will get the running pig without any errors on grunt.
Just enjoy doing.
% pig [return]
I had a similar problem with Pig 0.12.0 (and Hadoop 1.0.3) installed on Fedora 19.
When trying any Pig command, e.g.
pig -help
I was getting the error:
Cannot locate pig-withouthadoop.jar. do 'ant jar-withouthadoop.jar', and try again
Hadoop and Pig installation /bin folders were properly included in my PATH.
Simply copying pig-0.12.0-withouthadoop.jar to PIG_HOME folder fixed the issue for me.