I tried to compile Spark 1.3.1 using the following flags
mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 \
-Dscala-2.11 \
-Phive -Phive-0.13.1 -Phive-thriftserver \
-DskipTests clean package
The compilation failed with the following the error.
Any suggestions?
Thanks
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [01:08 min]
[INFO] Spark Project Core ................................. SUCCESS [02:38 min]
[INFO] Spark Project Bagel ................................ SUCCES [ 17.700 s]
[INFO] Spark Project GraphX ............................... SUCCESS [ 35.732 s]
[INFO] Spark Project ML Library ........................... SUCCESS [01:11 min]
[INFO] Spark Project Tools ................................ SUCCESS [ 6.718 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 6.837 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 3.534 s]
[INFO] Spark Project Streaming ............................ SUCCESS [ 43.771 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [ 48.411 s]
[INFO] Spark Project SQL .................................. SUCCESS [ 56.046 s]
[INFO] Spark Project Hive ................................. SUCCESS `enter code here`[01:01 min]
[INFO] Spark Project Assembly ............................. FAILURE [ 6.365 s]
[INFO] Spark Project External Twitter ..................... SKIPPED
[INFO] Spark Project External Flume ....................... SKIPPED
[INFO] Spark Project External Flume Sink .................. SKIPPED
[INFO] Spark Project External MQTT ........................ SKIPPED
[INFO] Spark Project External ZeroMQ ...................... SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project YARN ................................. SKIPPED
[INFO] Spark Project YARN Shuffle Service ................. SKIPPED
[INFO] Spark Project Hive Thrift Server ................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 09:45 min
[INFO] Finished at: 2015-05-15T11:25:53-06:00
[INFO] Final Memory: 77M/1176M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project spark-assembly_2.10: Could not resolve dependencies for project org.apache.spark:spark-assembly_2.10:pom:1.3.1: Could not find artifact org.apache.spark:spark-hive-thriftserver_2.11:jar:1.3.1 in central (https://repo1.maven.org/maven2) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project spark-assembly_2.10: Could not resolve dependencies for project org.apache.spark:spark-assembly_2.10:pom:1.3.1: Could not find artifact org.apache.spark:spark-hive-thriftserver_2.11:jar:1.3.1 in central (https://repo1.maven.org/maven2)
Seems no dependency of spark-hive-thriftserver_2.11 exists In maven repository, I also search it manually in maven repo and find nothing. I think thrift server is not ready for scala2.11 for now.
Related
I am trying to build Spark 2.2.0-rc2 release using mvn but unable to do so.
$ uname -a
Linux knoldus-Vostro-15-3568 4.4.0-46-generic #67-Ubuntu SMP Thu Oct 20 15:05:12 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
$ java -version
openjdk version "1.8.0_131"
Below is the error stack that I am getting:
$ ./build/mvn -Phadoop-2.7,yarn,mesos,hive,hive-thriftserver -DskipTests clean install
...
[INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) # spark-tags_2.11 ---
[INFO] Using zinc server for incremental compilation
java.lang.NoClassDefFoundError: Could not initialize class sun.util.calendar.ZoneInfoFile
at sun.util.calendar.ZoneInfo.getTimeZone(ZoneInfo.java:589)
at java.util.TimeZone.getTimeZone(TimeZone.java:560)
at java.util.TimeZone.setDefaultZone(TimeZone.java:666)
at java.util.TimeZone.getDefaultRef(TimeZone.java:636)
at java.util.Date.<init>(Date.java:254)
at java.util.zip.ZipUtils.dosToJavaTime(ZipUtils.java:71)
at java.util.zip.ZipUtils.extendedDosToJavaTime(ZipUtils.java:88)
at java.util.zip.ZipEntry.getTime(ZipEntry.java:194)
at sbt.IO$.next$1(IO.scala:278)
at sbt.IO$.sbt$IO$$extract(IO.scala:286)
at sbt.IO$$anonfun$unzipStream$1.apply(IO.scala:255)
at sbt.IO$$anonfun$unzipStream$1.apply(IO.scala:255)
at sbt.Using.apply(Using.scala:24)
at sbt.IO$.unzipStream(IO.scala:255)
at sbt.IO$$anonfun$unzip$1.apply(IO.scala:249)
at sbt.IO$$anonfun$unzip$1.apply(IO.scala:249)
at sbt.Using.apply(Using.scala:24)
at sbt.IO$.unzip(IO.scala:249)
at sbt.compiler.AnalyzingCompiler$$anonfun$compileSources$1$$anonfun$5.apply(AnalyzingCompiler.scala:140)
at sbt.compiler.AnalyzingCompiler$$anonfun$compileSources$1$$anonfun$5.apply(AnalyzingCompiler.scala:140)
at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:111)
at scala.collection.immutable.List.foldLeft(List.scala:84)
at scala.collection.TraversableOnce$class.$div$colon(TraversableOnce.scala:138)
at scala.collection.AbstractTraversable.$div$colon(Traversable.scala:105)
at sbt.compiler.AnalyzingCompiler$$anonfun$compileSources$1.apply(AnalyzingCompiler.scala:140)
at sbt.compiler.AnalyzingCompiler$$anonfun$compileSources$1.apply(AnalyzingCompiler.scala:139)
at sbt.IO$.withTemporaryDirectory(IO.scala:344)
at sbt.compiler.AnalyzingCompiler$.compileSources(AnalyzingCompiler.scala:139)
at sbt.compiler.IC$.compileInterfaceJar(IncrementalCompiler.scala:58)
at com.typesafe.zinc.Compiler$.compilerInterface(Compiler.scala:148)
at com.typesafe.zinc.Compiler$.create(Compiler.scala:53)
at com.typesafe.zinc.Compiler$$anonfun$apply$1.apply(Compiler.scala:40)
at com.typesafe.zinc.Compiler$$anonfun$apply$1.apply(Compiler.scala:40)
at com.typesafe.zinc.Cache.get(Cache.scala:41)
at com.typesafe.zinc.Compiler$.apply(Compiler.scala:40)
at com.typesafe.zinc.Main$.run(Main.scala:96)
at com.typesafe.zinc.Nailgun$.zinc(Nailgun.scala:93)
at com.typesafe.zinc.Nailgun$.nailMain(Nailgun.scala:82)
at com.typesafe.zinc.Nailgun.nailMain(Nailgun.scala)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.martiansoftware.nailgun.NGSession.run(NGSession.java:280)
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 5.657 s]
[INFO] Spark Project Tags ................................. FAILURE [ 0.371 s]
[INFO] Spark Project Sketch ............................... SKIPPED
[INFO] Spark Project Networking ........................... SKIPPED
[INFO] Spark Project Shuffle Streaming Service ............ SKIPPED
[INFO] Spark Project Unsafe ............................... SKIPPED
[INFO] Spark Project Launcher ............................. SKIPPED
[INFO] Spark Project Core ................................. SKIPPED
[INFO] Spark Project ML Local Library ..................... SKIPPED
[INFO] Spark Project GraphX ............................... SKIPPED
[INFO] Spark Project Streaming ............................ SKIPPED
[INFO] Spark Project Catalyst ............................. SKIPPED
[INFO] Spark Project SQL .................................. SKIPPED
[INFO] Spark Project ML Library ........................... SKIPPED
[INFO] Spark Project Tools ................................ SKIPPED
[INFO] Spark Project Hive ................................. SKIPPED
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project YARN Shuffle Service ................. SKIPPED
[INFO] Spark Project YARN ................................. SKIPPED
[INFO] Spark Project Mesos ................................ SKIPPED
[INFO] Spark Project Hive Thrift Server ................... SKIPPED
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Spark Project External Flume Sink .................. SKIPPED
[INFO] Spark Project External Flume ....................... SKIPPED
[INFO] Spark Project External Flume Assembly .............. SKIPPED
[INFO] Spark Integration for Kafka 0.8 .................... SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] Spark Project External Kafka Assembly .............. SKIPPED
[INFO] Spark Integration for Kafka 0.10 ................... SKIPPED
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED
[INFO] Kafka 0.10 Source for Structured Streaming ......... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6.855 s
[INFO] Finished at: 2017-05-30T13:47:02+05:30
[INFO] Final Memory: 50M/605M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first) on project spark-tags_2.11: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed.: CompileFailed -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :spark-tags_2.11
I am not able to understand this error.
scala> java.util.TimeZone.getDefault
res0: java.util.TimeZone = sun.util.calendar.ZoneInfo[id="Asia/Kolkata",offset=19800000,dstSavings=0,useDaylight=false,transitions=6,lastRule=null]
locale is LANG=en_IN LANGUAGE=en_IN:en LC_CTYPE="en_IN" LC_NUMERIC="en_IN" LC_TIME="en_IN" LC_COLLATE="en_IN" LC_MONETARY="en_IN" LC_MESSAGES="en_IN" LC_PAPER="en_IN" LC_NAME="en_IN" LC_ADDRESS="en_IN" LC_TELEPHONE="en_IN" LC_MEASUREMENT="en_IN" LC_IDENTIFICATION="en_IN" LC_ALL=
I guess the issue is with your time zone. Export LC_ALL=en_US.UTF-8 and start over. Make sure to have en_US.UTF-8 in locale for all entries.
$ locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL="en_US.UTF-8"
I had similar issue and the reason was missing timezone info file in java installation (after updating java 8 to newer version).
When I looked for tzdb.dat file - there was only link pointing to missing target
I was able to solve it on RedHat just by:
yum update tzdata-java
and this site was helpful here:
https://ekuric.wordpress.com/tag/tzdb-dat/
for other OS-es solution might be similar (there is even timezone updater tool from Oracle: https://www.oracle.com/technetwork/java/javase/documentation/tzupdater-readme-136440.html)
While building the hadoop using maven, I encountered one error that one command in ant script exited abnormally.
more details are in following error.
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 2.372 s]
[INFO] Apache Hadoop KMS .................................. FAILURE [ 5.222 s]
[INFO] Apache Hadoop Common Project ....................... SKIPPED
[INFO] Apache Hadoop HDFS ................................. SKIPPED
[INFO] Apache Hadoop HttpFS ............................... SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SKIPPED
[INFO] Apache Hadoop HDFS-NFS ............................. SKIPPED
[INFO] Apache Hadoop HDFS Project ......................... SKIPPED
[INFO] hadoop-yarn ........................................ SKIPPED
[INFO] hadoop-yarn-api .................................... SKIPPED
[INFO] hadoop-yarn-common ................................. SKIPPED
[INFO] hadoop-yarn-server ................................. SKIPPED
[INFO] hadoop-yarn-server-common .......................... SKIPPED
[INFO] hadoop-yarn-server-nodemanager ..................... SKIPPED
[INFO] hadoop-yarn-server-web-proxy ....................... SKIPPED
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SKIPPED
[INFO] hadoop-yarn-server-resourcemanager ................. SKIPPED
[INFO] hadoop-yarn-server-tests ........................... SKIPPED
[INFO] hadoop-yarn-client ................................. SKIPPED
[INFO] hadoop-yarn-applications ........................... SKIPPED
[INFO] hadoop-yarn-applications-distributedshell .......... SKIPPED
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 51.908 s
[INFO] Finished at: 2017-03-18T00:01:56+08:00
[INFO] Final Memory: 77M/771M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (dist) on project hadoop-kms: An Ant BuildException has occured: exec returned: 1
[ERROR] around Ant part ...<exec failonerror="true" dir="/Users/jinteng/work/hadoop-compile/hadoop-2.6.5-src/hadoop-common-project/hadoop-kms/target" executable="sh">... # 10:142 in /Users/jinteng/work/hadoop-compile/hadoop-2.6.5-src/hadoop-common-project/hadoop-kms/target/antrun/build-main.xml
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
I asked this question for sharing.
Background:
I killed the building process for the slow internet speed while downloading tomcat's tar packet.
Because of the interruption of downloading, the packet is not complete and tomcat-untar.sh would occured an exception while trying to untar the tomcat.
Later, I downloaded the packet separately and pushed it into downloads dir and found it works
[Run command] export MAVEN_OPTS="-Xmx1536m -XX:MaxPermSize=512m" &&
mvn clean install
[Error message] Running org.apache.atlas.web.adapters.TestEntityREST
Tests run: 11, Failures: 1, Errors: 0, Skipped: 8, Time elapsed:
13.995 sec <<< FAILURE! - in org.apache.atlas.web.adapters.TestEntityREST enter code here
cleanup(org.apache.atlas.web.adapters.TestEntityREST) Time elapsed:
0.217 sec <<< FAILURE!
java.lang.NullPointerException: null at org.apache.atlas.RequestContext.clear(RequestContext.java:97) at
org.apache.atlas.web.adapters.TestEntityREST.cleanup(TestEntityREST.java:80)
...
[INFO]
------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Apache Atlas Server Build
Tools .................... SUCCESS [ 1.489 s] [INFO] apache-atlas
....................................... SUCCESS [ 8.805 s] [INFO]
Apache Atlas Integration ........................... SUCCESS [
42.642 s] [INFO] Apache Atlas Common ................................ SUCCESS [ 14.276 s] [INFO] Apache
Atlas Typesystem ............................ SUCCESS [01:15 min]
[INFO] Apache Atlas Client ................................ SUCCESS
[ 17.168 s] [INFO] Apache Atlas Server API
............................ SUCCESS [ 8.440 s] [INFO] Apache Atlas
Notification .......................... SUCCESS [ 38.640 s] [INFO]
Apache Atlas Graph Database Projects ............... SUCCESS [
0.553 s] [INFO] Apache Atlas Graph Database API .................... SUCCESS [ 5.432 s] [INFO] Graph Database Common Code
......................... SUCCESS [ 5.636 s] [INFO] Shaded version
of Apache hbase client .............. SUCCESS [ 10.728 s] [INFO]
Apache Atlas Titan 0.5.4 Graph DB Impl ............. SUCCESS [01:58
min] [INFO] Apache Atlas Graph Database Implementation Dependencies
SUCCESS [ 0.846 s] [INFO] Shaded version of Apache hbase server
.............. SUCCESS [ 23.850 s] [INFO] Apache Atlas Repository
............................ SUCCESS [13:25 min] [INFO] Apache Atlas
Authorization ......................... SUCCESS [ 13.707 s] [INFO]
Apache Atlas Business Catalog ...................... SUCCESS [
31.334 s] [INFO] Apache Atlas UI .................................... SUCCESS [01:11 min] [INFO]
Apache Atlas Web Application ....................... FAILURE [04:29
min] [INFO] Apache Atlas Documentation .........................
SKIPPED [INFO] Apache Atlas FileSystem Model ......................
SKIPPED [INFO] Apache Atlas Plugin Classloader ....................
SKIPPED [INFO] Apache Atlas Hive Bridge Shim ......................
SKIPPED [INFO] Apache Atlas Hive Bridge ...........................
SKIPPED [INFO] Apache Atlas Falcon Bridge Shim ....................
SKIPPED [INFO] Apache Atlas Falcon Bridge .........................
SKIPPED [INFO] Apache Atlas Sqoop Bridge Shim .....................
SKIPPED [INFO] Apache Atlas Sqoop Bridge ..........................
SKIPPED [INFO] Apache Atlas Storm Bridge Shim .....................
SKIPPED [INFO] Apache Atlas Storm Bridge ..........................
SKIPPED [INFO] Apache Atlas Distribution ..........................
SKIPPED [INFO]
------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO]
------------------------------------------------------------------------ [INFO] Total time: 26:06 min [INFO] Finished at:
2017-02-09T01:20:53+09:00 [INFO] Final Memory: 165M/1437M [INFO]
------------------------------------------------------------------------ [ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:2.18.1:test
(default-test) on project atlas-webapp: There are test failures.
[ERROR] [ERROR] Please refer to
/Users/dongkillee/dev/bin/atlas/webapp/target/surefire-reports for
the individual test results. [ERROR] -> [Help 1] [ERROR] [ERROR] To
see the full stack trace of the errors, re-run Maven with the -e
switch. [ERROR] Re-run Maven using the -X switch to enable full
debug logging. [ERROR] [ERROR] For more information about the
errors and possible solutions, please read the following articles:
[ERROR] [Help 1]
```
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] [ERROR] After correcting the problems, you can resume the
build with the command [ERROR] mvn -rf :atlas-webapp
This is testcase error. Use following command. It will create your build successfully.
export MAVEN_OPTS="-Xmx1536m -XX:MaxPermSize=512m" && mvn clean install -DskipTests
I have explained the complete solution, here. briefly, cause it uses some test cases in maven plugins in pom.xml file, we need to add -DskipTests argument when building the Apache Atlas from scratch.
Do the following items to run the project without any error:
git clone https://github.com/apache/atlas
mvn clean install -DskipTests
mvn clean package -Pdist -DskipTests
I am try to run mvn clean eclipse:eclipse error under ambari source code as the document said ,
But with error :
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ........................................ SUCCESS [ 28.207 s]
[INFO] Apache Ambari Project POM .......................... SUCCESS [ 0.025 s]
[INFO] Ambari Web ......................................... SUCCESS [ 5.298 s]
[INFO] Ambari Views ....................................... SUCCESS [ 5.082 s]
[INFO] Ambari Admin View .................................. SUCCESS [03:43 min]
[INFO] ambari-metrics ..................................... SUCCESS [ 6.007 s]
[INFO] Ambari Metrics Common .............................. SUCCESS [ 2.274 s]
[INFO] Ambari Metrics Hadoop Sink ......................... SUCCESS [ 2.843 s]
[INFO] Ambari Metrics Flume Sink .......................... SUCCESS [ 1.166 s]
[INFO] Ambari Metrics Kafka Sink .......................... SUCCESS [ 0.229 s]
[INFO] Ambari Metrics Storm Sink .......................... SUCCESS [ 1.261 s]
[INFO] Ambari Metrics Collector ........................... FAILURE [ 11.478 s]
[INFO] Ambari Metrics Monitor ............................. SKIPPED
[INFO] Ambari Metrics Assembly ............................ SKIPPED
[INFO] Ambari Server ...................................... SKIPPED
[INFO] Ambari Agent ....................................... SKIPPED
[INFO] Ambari Client ...................................... SKIPPED
[INFO] Ambari Python Client ............................... SKIPPED
[INFO] Ambari Groovy Client ............................... SKIPPED
[INFO] Ambari Shell ....................................... SKIPPED
[INFO] Ambari Python Shell ................................ SKIPPED
[INFO] Ambari Groovy Shell ................................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 05:20 min
[INFO] Finished at: 2015-09-02T10:33:57+08:00
[INFO] Final Memory: 37M/141M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project ambari-metrics-timelineservice:Could not resolve dependencies for project org.apache.ambari:ambari-metrics-timelineservice:jar:2.0.0-SNAPSHOT: Could not find artifact org.apache.ambari:ambari-metrics-common:jar:2.0.0-SNAPSHOT in apache-hadoop (http://repo.hortonworks.com/content/groups/public/) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please rea d the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyReso lutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :ambari-metrics-timelineservice
I have no idea how to solve this error,
I tried googling with no answer
Please give me some advice
thanks
Your Ambari Metrics hasn't be built.
If you plan on installing the Ambari Metrics service, you will also need to build the Ambari Metrics project.
cd ambari-metrics
mvn clean package -Dbuild-rpm -DskipTests
For Ubuntu:
cd ambari-metrics
mvn clean package -Dbuild-deb -DskipTests
You can see more in
https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development
It is a dependency issue. A maven repository is a directory of packaged JAR files with pom.xml file. Maven searches for dependencies in the repositories. There are 3 types of maven repository:
1.Local Repository
2.Central Repository
3.Remote Repository
Maven searches for the dependencies in the following order:
Local repository > Central repository > Remote repository.
Maven searches the dependency and doesn't find the dependency into local,central and remote.
So first maven build the the ambari. After maven building the project,keep the dependency jar into local repository .
mvn clean install
It will package the jar and keep it on local repository.
Then you apply
mvn clean eclipse:eclipse
It will solve your problem
Hello I'm trying to build hadoop 2.6.0 on Windows 8.1. Unfortunately without luck so far.
I have installed:
jdk1.7.0_71 (added Variable JAVA_HOME with value C:\Program Files\Java\jdk1.7.0_71 to the User Variables)
cygwin64 (added its installation-directory as value D:\cygwin64\bin to the PATH Variable under System Variables)
Maven 3.2.5 (added its installation-directory as value D:\maven\bin to the PATH Variable under System Variables)
Protocol Buffer 2.5 (added its installation-directory as value D:\protobuf to the PATH Variable under System Variables)
Visual Studio 2010
In the Visual Studio 2010 Command Prompt (started as Administrator) I have changed the drive to d: and used the folder of the hadoop src as a starting point ("D:\hdp").
I do again set the JAVA_HOME in short notation and do set the Platform
set JAVA_HOME=C:\PROGRA~1\Java\jdk1.7.0_71
set Platform=x64
Afterwards I try to build hadoop using the following Maven command:
mvn -e -X package -Pdist -DskipTests -Dtar
After Building fails at "Apache Hadoop Common" stating the following:
[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.6.0:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: protoc failure -> [Help 1]
I have slightly changed the file "ProtocMojo.java" under D:\hdp\hadoop-maven-plugins\src\main\java\org\apache\hadoop\maven\plugin\protoc by inserting the following afer line 56 in the file:
protocCommand = "D:\\protobuf\\protoc.exe";
This helps to build further to "Apache Hadoop HDFS" where it fails again stating as follows:
[INFO] Executing tasks
Build sequence for target(s) `main' is [main]
Complete build sequence is [main, ]
main:
[mkdir] Skipping D:\hdp\hadoop-hdfs-project\hadoop-hdfs\target\native because it already exists.
[exec] Current OS is Windows 8.1
[exec] Executing 'cmake' with arguments:
[exec] 'D:\hdp\hadoop-hdfs-project\hadoop-hdfs/src/'
[exec] '-DGENERATED_JAVAH=D:\hdp\hadoop-hdfs-project\hadoop-hdfs\target/native/javah'
[exec] '-DJVM_ARCH_DATA_MODEL=64'
[exec] '-DREQUIRE_LIBWEBHDFS=false'
[exec] '-DREQUIRE_FUSE=false'
[exec] '-G'
[exec] 'Visual Studio 10 Win64'
[exec]
[exec] The ' characters around the executable and arguments are
[exec] not part of the command.
Execute:Java13CommandLauncher: Executing 'cmake' with arguments:
'D:\hdp\hadoop-hdfs-project\hadoop-hdfs/src/'
'-DGENERATED_JAVAH=D:\hdp\hadoop-hdfs-project\hadoop-hdfs\target/native/javah'
'-DJVM_ARCH_DATA_MODEL=64'
'-DREQUIRE_LIBWEBHDFS=false'
'-DREQUIRE_FUSE=false'
'-G'
'Visual Studio 10 Win64'
The ' characters around the executable and arguments are
not part of the command.
[exec] CMake Error: Could not create named generator Visual Studio 10 Win64
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 0.906 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 0.719 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 1.469 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.265 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 1.766 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 5.516 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 1.431 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 2.119 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 1.969 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:11 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 4.087 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 11.742 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.110 s]
[INFO] Apache Hadoop HDFS ................................. FAILURE [ 11.782 s]
[INFO] Apache Hadoop HttpFS ............................... SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SKIPPED
[INFO] Apache Hadoop HDFS-NFS ............................. SKIPPED
[INFO] Apache Hadoop HDFS Project ......................... SKIPPED
[INFO] hadoop-yarn ........................................ SKIPPED
[INFO] hadoop-yarn-api .................................... SKIPPED
[INFO] hadoop-yarn-common ................................. SKIPPED
[INFO] hadoop-yarn-server ................................. SKIPPED
[INFO] hadoop-yarn-server-common .......................... SKIPPED
[INFO] hadoop-yarn-server-nodemanager ..................... SKIPPED
[INFO] hadoop-yarn-server-web-proxy ....................... SKIPPED
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SKIPPED
[INFO] hadoop-yarn-server-resourcemanager ................. SKIPPED
[INFO] hadoop-yarn-server-tests ........................... SKIPPED
[INFO] hadoop-yarn-client ................................. SKIPPED
[INFO] hadoop-yarn-applications ........................... SKIPPED
[INFO] hadoop-yarn-applications-distributedshell .......... SKIPPED
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SKIPPED
[INFO] hadoop-yarn-site ................................... SKIPPED
[INFO] hadoop-yarn-registry ............................... SKIPPED
[INFO] hadoop-yarn-project ................................ SKIPPED
[INFO] hadoop-mapreduce-client ............................ SKIPPED
[INFO] hadoop-mapreduce-client-core ....................... SKIPPED
[INFO] hadoop-mapreduce-client-common ..................... SKIPPED
[INFO] hadoop-mapreduce-client-shuffle .................... SKIPPED
[INFO] hadoop-mapreduce-client-app ........................ SKIPPED
[INFO] hadoop-mapreduce-client-hs ......................... SKIPPED
[INFO] hadoop-mapreduce-client-jobclient .................. SKIPPED
[INFO] hadoop-mapreduce-client-hs-plugins ................. SKIPPED
[INFO] Apache Hadoop MapReduce Examples ................... SKIPPED
[INFO] hadoop-mapreduce ................................... SKIPPED
[INFO] Apache Hadoop MapReduce Streaming .................. SKIPPED
[INFO] Apache Hadoop Distributed Copy ..................... SKIPPED
[INFO] Apache Hadoop Archives ............................. SKIPPED
[INFO] Apache Hadoop Rumen ................................ SKIPPED
[INFO] Apache Hadoop Gridmix .............................. SKIPPED
[INFO] Apache Hadoop Data Join ............................ SKIPPED
[INFO] Apache Hadoop Ant Tasks ............................ SKIPPED
[INFO] Apache Hadoop Extras ............................... SKIPPED
[INFO] Apache Hadoop Pipes ................................ SKIPPED
[INFO] Apache Hadoop OpenStack support .................... SKIPPED
[INFO] Apache Hadoop Amazon Web Services support .......... SKIPPED
[INFO] Apache Hadoop Client ............................... SKIPPED
[INFO] Apache Hadoop Mini-Cluster ......................... SKIPPED
[INFO] Apache Hadoop Scheduler Load Simulator ............. SKIPPED
[INFO] Apache Hadoop Tools Dist ........................... SKIPPED
[INFO] Apache Hadoop Tools ................................ SKIPPED
[INFO] Apache Hadoop Distribution ......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:57 min
[INFO] Finished at: 2015-03-09T17:08:10+01:00
[INFO] Final Memory: 88M/1092M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-hdfs: An Ant BuildException has occured: exec returned: 1
[ERROR] around Ant part ...<exec dir="D:\hdp\hadoop-hdfs-project\hadoop-hdfs\target/native" executable="cmake" failonerror="true">... # 5:106 in D:\hdp\hadoop-hdfs-project\hadoop-h
dfs\target\antrun\build-main.xml
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-hdfs: An Ant BuildExcep
tion has occured: exec returned: 1
around Ant part ...<exec dir="D:\hdp\hadoop-hdfs-project\hadoop-hdfs\target/native" executable="cmake" failonerror="true">... # 5:106 in D:\hdp\hadoop-hdfs-project\hadoop-hdfs\targ
et\antrun\build-main.xml
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:355)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:216)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:160)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: An Ant BuildException has occured: exec returned: 1
around Ant part ...<exec dir="D:\hdp\hadoop-hdfs-project\hadoop-hdfs\target/native" executable="cmake" failonerror="true">... # 5:106 in D:\hdp\hadoop-hdfs-project\hadoop-hdfs\targ
et\antrun\build-main.xml
at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:355)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
... 19 more
Caused by: D:\hdp\hadoop-hdfs-project\hadoop-hdfs\target\antrun\build-main.xml:5: exec returned: 1
at org.apache.tools.ant.taskdefs.ExecTask.runExecute(ExecTask.java:646)
at org.apache.tools.ant.taskdefs.ExecTask.runExec(ExecTask.java:672)
at org.apache.tools.ant.taskdefs.ExecTask.execute(ExecTask.java:498)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:390)
at org.apache.tools.ant.Target.performTasks(Target.java:411)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1399)
at org.apache.tools.ant.Project.executeTarget(Project.java:1368)
at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:327)
... 21 more
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :hadoop-hdfs
This is where I'm stucked and do not know how to solve this problem. Help is appreciated as I am new to hadoop and building in general.
Thanks!
elro
I had the same error. Fixed it by making sure the cmake wasn't the one that comes with cygwin. For some reason, that cmake doesn't recognize Visual Studio.
Did you forget the native-win profile? Try mvn package -Pdist,native-win -DskipTests -Dtar
I've built hadoop on Windows before and didn't have to change any source code. Try changing the source back, run the above command, and post more of the protoc error you got with maven.