I'm trying to build Apache Spark to do this I execute mvn clean package (obviously in the spark root directory) halfway through the process I get a failure in the core file. The error log is as follows:
main:
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Core ................................. FAILURE [ 12.363 s]
[INFO] Spark Project ML Local Library ..................... SKIPPED
[INFO] Spark Project GraphX ............................... SKIPPED
[INFO] Spark Project Streaming ............................ SKIPPED
[INFO] Spark Project Catalyst ............................. SKIPPED
[INFO] Spark Project SQL .................................. SKIPPED
[INFO] Spark Project ML Library ........................... SKIPPED
[INFO] Spark Project Tools ................................ SKIPPED
[INFO] Spark Project Hive ................................. SKIPPED
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Spark Project External Flume Sink .................. SKIPPED
[INFO] Spark Project External Flume ....................... SKIPPED
[INFO] Spark Project External Flume Assembly .............. SKIPPED
[INFO] Spark Integration for Kafka 0.8 .................... SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] Spark Project External Kafka Assembly .............. SKIPPED
[INFO] Spark Integration for Kafka 0.10 ................... SKIPPED
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED
[INFO] Kafka 0.10 Source for Structured Streaming ......... SKIPPED
[INFO] Spark Project Java 8 Tests ......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 13.500 s
[INFO] Finished at: 2017-03-09T18:20:34+05:30
[INFO] Final Memory: 46M/498M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run (default) on project spark-core_2.11: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "bash" (in directory "C:\Users\Admin\Downloads\spark-2.1.0\spark-2.1.0\core"): CreateProcess error=2, The system cannot find the file specified
[ERROR] around Ant part ...<exec executable="bash">... # 4:27 in C:\Users\Admin\Downloads\spark-2.1.0\spark-2.1.0\core\target\antrun\build-main.xml
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run (default) on project spark-core_2.11: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "bash" (in directory "C:\Users\Admin\Downloads\spark-2.1.0\spark-2.1.0\core"): CreateProcess error=2, The system cannot find the file specified
around Ant part ...<exec executable="bash">... # 4:27 in C:\Users\Admin\Downloads\spark-2.1.0\spark-2.1.0\core\target\antrun\build-main.xml
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "bash" (in directory "C:\Users\Admin\Downloads\spark-2.1.0\spark-2.1.0\core"): CreateProcess error=2, The system cannot find the file specified
around Ant part ...<exec executable="bash">... # 4:27 in C:\Users\Admin\Downloads\spark-2.1.0\spark-2.1.0\core\target\antrun\build-main.xml
at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:342)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
... 20 more
Caused by: C:\Users\Admin\Downloads\spark-2.1.0\spark-2.1.0\core\target\antrun\build-main.xml:4: Execute failed: java.io.IOException: Cannot run program "bash" (in directory "C:\Users\Admin\Downloads\spark-2.1.0\spark-2.1.0\core"): CreateProcess error=2, The system cannot find the file specified
at org.apache.tools.ant.taskdefs.ExecTask.runExec(ExecTask.java:672)
at org.apache.tools.ant.taskdefs.ExecTask.execute(ExecTask.java:495)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:292)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:435)
at org.apache.tools.ant.Target.performTasks(Target.java:456)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1393)
at org.apache.tools.ant.Project.executeTarget(Project.java:1364)
at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:313)
... 22 more
Caused by: java.io.IOException: Cannot run program "bash" (in directory "C:\Users\Admin\Downloads\spark-2.1.0\spark-2.1.0\core"): CreateProcess error=2, The system cannot find the file specified
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at java.lang.Runtime.exec(Runtime.java:620)
at org.apache.tools.ant.taskdefs.launcher.Java13CommandLauncher.exec(Java13CommandLauncher.java:58)
at org.apache.tools.ant.taskdefs.Execute.launch(Execute.java:428)
at org.apache.tools.ant.taskdefs.Execute.execute(Execute.java:442)
at org.apache.tools.ant.taskdefs.ExecTask.runExecute(ExecTask.java:628)
at org.apache.tools.ant.taskdefs.ExecTask.runExec(ExecTask.java:669)
... 35 more
Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified
at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.<init>(ProcessImpl.java:386)
at java.lang.ProcessImpl.start(ProcessImpl.java:137)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 41 more
Looks like the build_main.xml has a problem with the bash executable. The xml file below:
<project name="maven-antrun-" default="main" >
<target name="main">
<exec executable="bash">
<arg value="C:\Users\Admin\Downloads\spark-2.1.0\spark-2.1.0\core/../build/spark-build-info"/>
<arg value="C:\Users\Admin\Downloads\spark-2.1.0\spark-2.1.0\core\target/extra-resources"/>
<arg value="2.1.0"/>
</exec>
</target>
</project>
I did my research and found out that changing the exec to bash.exe might solve the issue but when I edit my xml file and execute mvn -e clean package -rf :spark-core_2.11 same error occurs and the changes I made to build_main.xml get reverted.
Thanks in advance.
Update - I couldn't find a way around it. I had to download the Pre-built version of Spark.
I found one way around this problem, you can put %GIT_HOME%\bin into the "path" environment variable to use git bash.
Related
I am building spark using cygwin on a windows system. I can build fine when I do this and it works after
git clone https://github.com/apache/spark
dos2unix spark/build/mvn
build/mvn -DskipTests clean package
but when I try to use the -Psparkr profile
build/mvn -DskipTests -Psparkr clean package
I get an error, I tried with the -X and -e option but nothing seems to give me any indication why it fails.
[DEBUG] Executing command line: [cmd, /c, ..\R\install-dev.bat]
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 5.613 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 27.617 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 16.601 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 12.118 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 22.492 s]
[INFO] Spark Project Core ................................. FAILURE [03:19 min]
[INFO] Spark Project Bagel ................................ SKIPPED
[INFO] Spark Project GraphX ............................... SKIPPED
[INFO] Spark Project Streaming ............................ SKIPPED
[INFO] Spark Project Catalyst ............................. SKIPPED
[INFO] Spark Project SQL .................................. SKIPPED
[INFO] Spark Project ML Library ........................... SKIPPED
[INFO] Spark Project Tools ................................ SKIPPED
[INFO] Spark Project Hive ................................. SKIPPED
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Spark Project External Twitter ..................... SKIPPED
[INFO] Spark Project External Flume Sink .................. SKIPPED
[INFO] Spark Project External Flume ....................... SKIPPED
[INFO] Spark Project External Flume Assembly .............. SKIPPED
[INFO] Spark Project External MQTT ........................ SKIPPED
[INFO] Spark Project External MQTT Assembly ............... SKIPPED
[INFO] Spark Project External ZeroMQ ...................... SKIPPED
[INFO] Spark Project External Kafka ....................... SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] Spark Project External Kafka Assembly .............. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 04:44 min
[INFO] Finished at: 2015-09-24T09:37:59-04:00
[INFO] Final Memory: 46M/490M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:exec (sparkr-pkg) on project spark-core_2.10: Command execution failed. Process exited with an error: -1 (Exit value: -1) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:exec (sparkr-pkg) on project spark-core_2.10: Command execution failed.
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:862)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:286)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:197)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: Command execution failed.
at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:276)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
... 20 more
Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: -1 (Exit value: -1)
at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)
at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:166)
at org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:660)
at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:265)
... 22 more
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :spark-core_2.10
Any clues to why it would be failing?
Likely it is because of missing dependencies on R language which are required by -Psparkr compilation option.
On Debian GNU/Linux (in particular 8/jessie) you would need to run
apt-get install r-base r-base-dev
or
aptitude install r-base r-base-dev
Prefer aptitude version if you have it installed.
read the README.md of SparkR at https://github.com/apache/spark/tree/master/R.
cd the source code dir (spark-1.6.1/R) and run ./install-dev.sh
then, run build/mvn -DskipTests -Psparkr package will be fine
I'm following the tutorial to build and install hadoop.
http://www.srccodes.com/p/article/38/build-install-configure-run-apache-hadoop-2.2.0-microsoft-windows-os
However, when I give the below command from the VS2010 command prompt:
mvn package -Pdist,native-win -DskipTests -Dtar
I get the below error:
main:
[mkdir] Skipping C:\hdfs\hadoop-hdfs-project\hadoop-hdfs\target\native because it already exists.
[exec] Current OS is Windows 8.1
[exec] Executing 'cmake' with arguments:
[exec] 'C:\hdfs\hadoop-hdfs-project\hadoop-hdfs/src/'
[exec] '-DGENERATED_JAVAH=C:\hdfs\hadoop-hdfs-project\hadoop-hdfs\target/native/javah'
[exec] '-DJVM_ARCH_DATA_MODEL=64'
[exec] '-DREQUIRE_LIBWEBHDFS=false'
[exec] '-DREQUIRE_FUSE=false'
[exec] '-G'
[exec] 'Visual Studio 10 Win64'
[exec]
[exec] The ' characters around the executable and arguments are
[exec] not part of the command.
Execute:Java13CommandLauncher: Executing 'cmake' with arguments:
'C:\hdfs\hadoop-hdfs-project\hadoop-hdfs/src/'
'-DGENERATED_JAVAH=C:\hdfs\hadoop-hdfs-project\hadoophdfs\target/native/javah'
'-DJVM_ARCH_DATA_MODEL=64'
'-DREQUIRE_LIBWEBHDFS=false'
'-DREQUIRE_FUSE=false'
'-G'
'Visual Studio 10 Win64'
The ' characters around the executable and arguments are not part of the command.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 1.781s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 1.333s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 1.030s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.375s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 2.104s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 6.628s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 1.047s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 1.173s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 1.594s]
[INFO] Apache Hadoop Common ............................... SUCCESS [ 59.046s]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 1.905s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 6.491s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.150s]
[INFO] Apache Hadoop HDFS ................................. FAILURE [ 19.351s]
[INFO] Apache Hadoop HttpFS ............................... SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SKIPPED
[INFO] Apache Hadoop HDFS-NFS ............................. SKIPPED
[INFO] Apache Hadoop HDFS Project ......................... SKIPPED
[INFO] hadoop-yarn ........................................ SKIPPED
[INFO] hadoop-yarn-api .................................... SKIPPED
[INFO] hadoop-yarn-common ................................. SKIPPED
[INFO] hadoop-yarn-server ................................. SKIPPED
[INFO] hadoop-yarn-server-common .......................... SKIPPED
[INFO] hadoop-yarn-server-nodemanager ..................... SKIPPED
[INFO] hadoop-yarn-server-web-proxy ....................... SKIPPED
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SKIPPED
[INFO] hadoop-yarn-server-resourcemanager ................. SKIPPED
[INFO] hadoop-yarn-server-tests ........................... SKIPPED
[INFO] hadoop-yarn-client ................................. SKIPPED
[INFO] hadoop-yarn-applications ........................... SKIPPED
[INFO] hadoop-yarn-applications-distributedshell .......... SKIPPED
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SKIPPED
[INFO] hadoop-yarn-site ................................... SKIPPED
[INFO] hadoop-yarn-registry ............................... SKIPPED
[INFO] hadoop-yarn-project ................................ SKIPPED
[INFO] hadoop-mapreduce-client ............................ SKIPPED
[INFO] hadoop-mapreduce-client-core ....................... SKIPPED
[INFO] hadoop-mapreduce-client-common ..................... SKIPPED
[INFO] hadoop-mapreduce-client-shuffle .................... SKIPPED
[INFO] hadoop-mapreduce-client-app ........................ SKIPPED
[INFO] hadoop-mapreduce-client-hs ......................... SKIPPED
[INFO] hadoop-mapreduce-client-jobclient .................. SKIPPED
[INFO] hadoop-mapreduce-client-hs-plugins ................. SKIPPED
[INFO] Apache Hadoop MapReduce Examples ................... SKIPPED
[INFO] hadoop-mapreduce ................................... SKIPPED
[INFO] Apache Hadoop MapReduce Streaming .................. SKIPPED
[INFO] Apache Hadoop Distributed Copy ..................... SKIPPED
[INFO] Apache Hadoop Archives ............................. SKIPPED
[INFO] Apache Hadoop Rumen ................................ SKIPPED
[INFO] Apache Hadoop Gridmix .............................. SKIPPED
[INFO] Apache Hadoop Data Join ............................ SKIPPED
[INFO] Apache Hadoop Ant Tasks ............................ SKIPPED
[INFO] Apache Hadoop Extras ............................... SKIPPED
[INFO] Apache Hadoop Pipes ................................ SKIPPED
[INFO] Apache Hadoop OpenStack support .................... SKIPPED
[INFO] Apache Hadoop Amazon Web Services support .......... SKIPPED
[INFO] Apache Hadoop Client ............................... SKIPPED
[INFO] Apache Hadoop Mini-Cluster ......................... SKIPPED
[INFO] Apache Hadoop Scheduler Load Simulator ............. SKIPPED
[INFO] Apache Hadoop Tools Dist ........................... SKIPPED
[INFO] Apache Hadoop Tools ................................ SKIPPED
[INFO] Apache Hadoop Distribution ......................... SKIPPED
[INFO] -----------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] -----------------------------------------------------------------------
[INFO] Total time: 01:47 min
[INFO] Finished at: 2015-03-28T21:18:11+00:00
[INFO] Final Memory: 78M/363M
[INFO] -----------------------------------------------------------------------
[WARNING] The requested profile "native-bin" could not be activated because it does not exist.
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-hdfs: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "cmake" (in directory "C:\hdfs\hadoop-hdfs-project\hadoop-hdfs\target\native"): CreateProcess error=2, The system cannot find the file specified [ERROR] around Ant part ...<exec failonerror="true" dir="C:\hdfs\hadoop-hdfs-project\hadoop-hdfs\target/native" executable="cmake">... # 5:107 in C:\hdfs\hadoop-hdfs-project\hadoop-hdfs\target\antrun\build-main.xml
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-hdfs: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "cmake" (in directory "C:\hdfs\hadoop-hdfs-project\hadoop-hdfs\target\native"): CreateProcess error=2, The system cannot find the file specifiedaround Ant part ...<exec failonerror="true" dir="C:\hdfs\hadoop-hdfs-project\hadoop-hdfs\target/native" executable="cmake">... # 5:107 in C:\hdfs\hadoop-hdfs-project\hadoop-hdfs\target\antrun\build-main.xml
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:862)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:286)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:197)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "cmake" (in directory "C:\hdfs\hadoop-hdfs-project\hadoop-hdfs\target\native"): CreateProcess error=2, The system cannot find the file specifiedaround Ant part ...<exec failonerror="true" dir="C:\hdfs\hadoop-hdfs-project\hadoop-hdfs\target/native" executable="cmake">... # 5:107 in C:\hdfs\hadoop-hdfs-project\hadoop-hdfs\target\antrun\build-main.xml
at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:355)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
... 20 more
Caused by: C:\hdfs\hadoop-hdfs-project\hadoop-hdfs\target\antrun\build-main.xml:5: Execute failed: java.io.IOException: Cannot run program "cmake" (in directory "C:\hdfs\hadoop-hdfs-project\hadoop-hdfs\target\native"): CreateProcess error=2, The system cannot find the file specified
at org.apache.tools.ant.taskdefs.ExecTask.runExec(ExecTask.java:675)
at org.apache.tools.ant.taskdefs.ExecTask.execute(ExecTask.java:498)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:390)
at org.apache.tools.ant.Target.performTasks(Target.java:411)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1399)
at org.apache.tools.ant.Project.executeTarget(Project.java:1368)
at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:327)
... 22 more
Caused by: java.io.IOException: Cannot run program "cmake" (in directory "C:\hdfs\hadoop-hdfs-project\hadoop-hdfs\target\native"): CreateProcess error=2, The system cannot find the file specified
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at java.lang.Runtime.exec(Runtime.java:620)
at org.apache.tools.ant.taskdefs.Execute$Java13CommandLauncher.exec(Execute.java:862)
at org.apache.tools.ant.taskdefs.Execute.launch(Execute.java:481)
at org.apache.tools.ant.taskdefs.Execute.execute(Execute.java:495)
at org.apache.tools.ant.taskdefs.ExecTask.runExecute(ExecTask.java:631)
at org.apache.tools.ant.taskdefs.ExecTask.runExec(ExecTask.java:672)
... 34 more
Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified
at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.<init>(ProcessImpl.java:386)
at java.lang.ProcessImpl.start(ProcessImpl.java:137)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 40 more
I don't understand the error at all. I have cmake installed and C:\cmake\bin added to the path as well.
Hmm...this error makes me think you had a typo in your Maven command:
[WARNING] The requested profile "native-bin" could not be activated because it does not exist.
Check your command; make sure it's native-win, not native-bin
I also wrote a guide recently for building Hadoop on Server 2012 R2 (should be identical for 8.1). Let me know.
I'm trying to compile Spark 1.2.0 using maven 3.2.2, scala 2.10.4, java 1.8.0_05 and what I'm getting is:
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 3.513 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 8.909 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 6.625 s]
[INFO] Spark Project Core ................................. FAILURE [01:06 min]
[INFO] Spark Project Bagel ................................ SKIPPED
[INFO] Spark Project GraphX ............................... SKIPPED
[INFO] Spark Project Streaming ............................ SKIPPED
[INFO] Spark Project Catalyst ............................. SKIPPED
[INFO] Spark Project SQL .................................. SKIPPED
[INFO] Spark Project ML Library ........................... SKIPPED
[INFO] Spark Project Tools ................................ SKIPPED
[INFO] Spark Project Hive ................................. SKIPPED
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Spark Project External Twitter ..................... SKIPPED
[INFO] Spark Project External Flume Sink .................. SKIPPED
[INFO] Spark Project External Flume ....................... SKIPPED
[INFO] Spark Project External MQTT ........................ SKIPPED
[INFO] Spark Project External ZeroMQ ...................... SKIPPED
[INFO] Spark Project External Kafka ....................... SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:26 min
[INFO] Finished at: 2015-01-17T22:10:43+01:00
[INFO] Final Memory: 41M/554M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.0:compile (scala-compile-first) on project spark-core_2.10: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.0:compile failed. CompileFailed -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.0:compile (scala-compile-first) on project spark-core_2.10: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.0:compile failed.
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:224)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:347)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:154)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:213)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:157)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.0:compile failed.
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:143)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
... 19 more
Caused by: Compilation failed
at sbt.compiler.AnalyzingCompiler.call(AnalyzingCompiler.scala:105)
at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:48)
at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:41)
at sbt.compiler.AggressiveCompile$$anonfun$3$$anonfun$compileScala$1$1.apply$mcV$sp(AggressiveCompile.scala:99)
at sbt.compiler.AggressiveCompile$$anonfun$3$$anonfun$compileScala$1$1.apply(AggressiveCompile.scala:99)
at sbt.compiler.AggressiveCompile$$anonfun$3$$anonfun$compileScala$1$1.apply(AggressiveCompile.scala:99)
at sbt.compiler.AggressiveCompile.sbt$compiler$AggressiveCompile$$timed(AggressiveCompile.scala:166)
at sbt.compiler.AggressiveCompile$$anonfun$3.compileScala$1(AggressiveCompile.scala:98)
at sbt.compiler.AggressiveCompile$$anonfun$3.apply(AggressiveCompile.scala:143)
at sbt.compiler.AggressiveCompile$$anonfun$3.apply(AggressiveCompile.scala:87)
at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:39)
at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:37)
at sbt.inc.IncrementalCommon.cycle(Incremental.scala:99)
at sbt.inc.Incremental$$anonfun$1.apply(Incremental.scala:38)
at sbt.inc.Incremental$$anonfun$1.apply(Incremental.scala:37)
at sbt.inc.Incremental$.manageClassfiles(Incremental.scala:65)
at sbt.inc.Incremental$.compile(Incremental.scala:37)
at sbt.inc.IncrementalCompile$.apply(Compile.scala:27)
at sbt.compiler.AggressiveCompile.compile2(AggressiveCompile.scala:157)
at sbt.compiler.AggressiveCompile.compile1(AggressiveCompile.scala:71)
at com.typesafe.zinc.Compiler.compile(Compiler.scala:184)
at com.typesafe.zinc.Compiler.compile(Compiler.scala:164)
at sbt_inc.SbtIncrementalCompiler.compile(SbtIncrementalCompiler.java:92)
at scala_maven.ScalaCompilerSupport.incrementalCompile(ScalaCompilerSupport.java:303)
at scala_maven.ScalaCompilerSupport.compile(ScalaCompilerSupport.java:119)
at scala_maven.ScalaCompilerSupport.doExecute(ScalaCompilerSupport.java:99)
at scala_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:482)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
... 20 more
Could you please guide me where might be the problem? I tried to find solution elsewhere. There was a problem reported in SPARK-3794 but I think that I have patched version (although version description in jira ticket is weired).
I'm running packaging with following mvn command:
mvn -Dhadoop.version=2.4.1 -DskipTests clean package -X
Concise
The issue could be caused by or a combination of the following:
openjdk instead of oracle jdk
a zinc server is still running
The JAVA_HOME is incorrect
Verbose
The issue could be caused because openjdk was used:
user#host $ java -version
openjdk version "1.8.0_111"
OpenJDK Runtime Environment (build 1.8.0_111-b15)
OpenJDK 64-Bit Server VM (build 25.111-b15, mixed mode)
instead of the oracle one. Once the path was updated, i.e.:
user#host $ java -version
java version "1.8.0_92"
Java(TM) SE Runtime Environment (build 1.8.0_92-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode)
the issue did not occur anymore when the following commands:
git clone https://github.com/apache/spark.git && \
cd spark && \
build/mvn -DskipTests clean package
were run and the build succeeded:
[INFO] Kafka 0.10 Source for Structured Streaming ......... SUCCESS [ 9.715 s]
[INFO] Spark Project Java 8 Tests ......................... SUCCESS [ 5.586 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 06:12 min (Wall Clock)
[INFO] Finished at: 2016-12-19T14:20:39+01:00
[INFO] Final Memory: 71M/859M
[INFO] ------------------------------------------------------------------------
The issue could also caused because a Zinc server was still running, e.g.:
user#host $ ps -ef | grep -i zinc
once this process was killed using kill <enter pid zinc server process> the issue was solved.
Ensure that the JAVA_HOME is correct by issuing JAVA_HOME=/opt/jdk1.8.0_92 and verify it by issuing: echo $JAVA_HOME
This might work for you.
Before build run:
./dev/change-scala-version.sh 2.11
to change the Scala version.
To compile Spark with maven, you should do the following steps
Change scala version to the scala version in your machine: ./dev/change-scala-version.sh <version>
Shutdown zinc: ./build/zinc-<version>/bin/zinc -shutdown
Compile Spark: ./build/mvn -Pyarn -Phadoop-<version> -Dscala-<version> -DskipTests clean package
You might try disabling the use of Zinc Server from ./spark/pom.xml
Comment out
<!--<useZincServer>true</useZincServer>-->
I did this, then saw JVM Out of Memory Exception. That also might be the issue for you.
I need help as i am trying to figure this out from last 2-3 days..
I am setting up Hadoop on Windows-7 (64-bit) machine. This is to try out the integration of R with Hadoop.
I followed instructions for Hadoop installation as given in the URL - http://www.srccodes.com/p/article/38/build-install-configure-run-apache-hadoop-2.2.0-microsoft-windows-os
Environment Variables are as below
JAVE_HOME : C:\Program Files\Java\jdk1.6.0_45
M2_HOME : C:\Hadoop\apache-maven-3.1.1
PATH : C:\cygwin64\bin;C:\Hadoop\apache-maven-3.1.1\bin;C:\Hadoop\protoc-2.5.0-win32
mvn package -Pdist,native-win -DskipTests -Dtar
i ran above command from Windows SDK 7.1 command prompt and getting error below... Early help is hugely appreciated..
****************************************************************************************
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................ SUCCESS [8.893s]
[INFO] Apache Hadoop Project POM ......................... SUCCESS [4.782s]
[INFO] Apache Hadoop Annotations ......................... SUCCESS [9.500s]
[INFO] Apache Hadoop Assemblies .......................... SUCCESS [2.145s]
[INFO] Apache Hadoop Project Dist POM .................... FAILURE [4.141s]
[INFO] Apache Hadoop Maven Plugins ....................... SKIPPED
[INFO] Apache Hadoop Auth ................................ SKIPPED
[INFO] Apache Hadoop Auth Examples ....................... SKIPPED
[INFO] Apache Hadoop Common .............................. SKIPPED
[INFO] Apache Hadoop NFS ................................. SKIPPED
[INFO] Apache Hadoop Common Project ...................... SKIPPED
[INFO] Apache Hadoop HDFS ................................ SKIPPED
[INFO] Apache Hadoop HttpFS .............................. SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SKIPPED
[INFO] Apache Hadoop HDFS-NFS ............................ SKIPPED
[INFO] Apache Hadoop HDFS Project ........................ SKIPPED
[INFO] hadoop-yarn ....................................... SKIPPED
[INFO] hadoop-yarn-api ................................... SKIPPED
[INFO] hadoop-yarn-common ................................ SKIPPED
[INFO] hadoop-yarn-server ................................ SKIPPED
[INFO] hadoop-yarn-server-common ......................... SKIPPED
[INFO] hadoop-yarn-server-nodemanager .................... SKIPPED
[INFO] hadoop-yarn-server-web-proxy ...................... SKIPPED
[INFO] hadoop-yarn-server-resourcemanager ................ SKIPPED
[INFO] hadoop-yarn-server-tests .......................... SKIPPED
[INFO] hadoop-yarn-client ................................ SKIPPED
[INFO] hadoop-yarn-applications .......................... SKIPPED
[INFO] hadoop-yarn-applications-distributedshell ......... SKIPPED
[INFO] hadoop-mapreduce-client ........................... SKIPPED
[INFO] hadoop-mapreduce-client-core ...................... SKIPPED
[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SKIPPED
[INFO] hadoop-yarn-site .................................. SKIPPED
[INFO] hadoop-yarn-project ............................... SKIPPED
[INFO] hadoop-mapreduce-client-common .................... SKIPPED
[INFO] hadoop-mapreduce-client-shuffle ................... SKIPPED
[INFO] hadoop-mapreduce-client-app ....................... SKIPPED
[INFO] hadoop-mapreduce-client-hs ........................ SKIPPED
[INFO] hadoop-mapreduce-client-jobclient ................. SKIPPED
[INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
[INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
[INFO] hadoop-mapreduce .................................. SKIPPED
[INFO] Apache Hadoop MapReduce Streaming ................. SKIPPED
[INFO] Apache Hadoop Distributed Copy .................... SKIPPED
[INFO] Apache Hadoop Archives ............................ SKIPPED
[INFO] Apache Hadoop Rumen ............................... SKIPPED
[INFO] Apache Hadoop Gridmix ............................. SKIPPED
[INFO] Apache Hadoop Data Join ........................... SKIPPED
[INFO] Apache Hadoop Extras .............................. SKIPPED
[INFO] Apache Hadoop Pipes ............................... SKIPPED
[INFO] Apache Hadoop Tools Dist .......................... SKIPPED
[INFO] Apache Hadoop Tools ............................... SKIPPED
[INFO] Apache Hadoop Distribution ........................ SKIPPED
[INFO] Apache Hadoop Client .............................. SKIPPED
[INFO] Apache Hadoop Mini-Cluster ........................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 32.981s
[INFO] Finished at: Thu Feb 13 14:06:51 IST 2014
[INFO] Final Memory: 32M/190M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (pre-dist) on project hadoop-project-dist: An Ant BuildException has occured : exec returned: 2 -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (pre-dist) on project hadoop-project-dist: An Ant BuildException has occured: exec returned: 2
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:317)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:152)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:555)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:214)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:158)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: An Ant BuildException
has occured: exec returned: 2
at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:283)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:106)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
... 19 more
Caused by: C:\Hadoop\hdfs\hadoop-project-dist\target\antrun\build-main.xml:31: exec returned: 2
at org.apache.tools.ant.taskdefs.ExecTask.runExecute(ExecTask.java:650)
at org.apache.tools.ant.taskdefs.ExecTask.runExec(ExecTask.java:676)
at org.apache.tools.ant.taskdefs.ExecTask.execute(ExecTask.java:502)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:390)
at org.apache.tools.ant.Target.performTasks(Target.java:411)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1397)
at org.apache.tools.ant.Project.executeTarget(Project.java:1366)
at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:270)
... 21 more
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :hadoop-project-dist
Thanks in advance
Gopal
Have you solved the problem? I had the similar problem when I was installing Hadoop. I found that all the problems in building the source file phase were related to Path variable setting.
1. Make sure you have JDK installed not only JRE available
2. One additional path setting needed, if you are following instruction on this link: http://www.srccodes.com/p/article/38/build-install-configure-run-apache-hadoop-2.2.0-microsoft-windows-os :
Add .NET MsBuild.exe home directory to Path, it should be similar to this C:\Windows\Microsoft.NET\Framework64\v4.030319
3. Check all the path values are corrected. During my installation, I had a problem which was related to Cygwin path, cygwin was installed in the folder named cygwin64, but at the beginning I just add cygwin as path value, which took me some time to figure it out.
I use the following command, it succeeds.
mvn clean package -DskipTests -Dtar
"mvn package -Pdist,native-win -DskipTests -Dtar" According to this command you are trying to compile and package "native libraries"(native-win keyword) and to do this whatever packages given in your given link is not sufficient.
To solve this you have two alternatives:
1) No need of native libraries, hadoop will use "builtin-java classes" instead of native libraries. use this command : "mvn package -Pdist -DskipTests -Dtar" instead of yours
2)if you want to go for native libraries- Follow Native Libraries Guide. Here all needed packages and other important information for building native libraries is given.
My suggestion is use 1st option because i read somewhere that building native libraries on windows have lot of issues and not preferred on windows.
Read "Building.txt" in Hadoop src folder it may help you for using variety of commands for packaging.
Above may be the solution for your problem.Try it
I had the same error. When I ran mvn package -Pdist,native-win -DskipTests -Dtar with -X, maven showed me that there are some troubles with executing dist-copynativelibs.sh script.
The problem was that Windows command prompt can't run sh commands, but only bash commands. Don't know why - Cygwin was installed correctly and added to path. So I have changed executable="sh" to executable="bash" in pom file D:\hdfs\hadoop-project-dist\pom.xml:
<exec executable="bash" dir="${project.build.directory}" failonerror="true">
<arg line="./dist-copynativelibs.sh"/>
</exec>
and build was success.
I'm trying to build the Hadoop trunk using Maven. I got the project with svn. But then, I tried mvn clean install. But I'm getting the following error:
[INFO] Apache Hadoop Main ................................ SUCCESS [1.616s]
[INFO] Apache Hadoop Project POM ......................... FAILURE [0.322s]
[INFO] Apache Hadoop Annotations ......................... SKIPPED
[INFO] Apache Hadoop Project Dist POM .................... SKIPPED
.... etc ....
.... etc ....
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.361s
[INFO] Finished at: Sat Jun 15 20:52:13 IST 2013
[INFO] Final Memory: 18M/43M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (create-testdirs) on project hadoop-project: Error executing ant tasks: /home/sreejith/svn/hadoop-trunk/hadoop-project/target/antrun/build-main.xml (No such file or directory) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (create-testdirs) on project hadoop-project: Error executing ant tasks: /home/sreejith/svn/hadoop-trunk/hadoop-project/target/antrun/build-main.xml (No such file or directory)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:217)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
Caused by: org.apache.maven.plugin.MojoExecutionException: Error executing ant tasks: /home/sreejith/svn/hadoop-trunk/hadoop-project/target/antrun/build-main.xml (No such file or directory)
at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:287)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
... 19 more
Caused by: java.io.FileNotFoundException: /home/sreejith/svn/hadoop-trunk/hadoop-project/target/antrun/build-main.xml (No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:212)
at java.io.FileOutputStream.<init>(FileOutputStream.java:104)
at org.codehaus.plexus.util.FileUtils.fileWrite(FileUtils.java:470)
at org.apache.maven.plugin.antrun.AntRunMojo.writeTargetToProjectFile(AntRunMojo.java:499)
at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:226)
... 21 more
I googled all I could. Could someone give me a clue why this is happening?
Found a solution. Do a sudo chown -R username parent-directory if you haven't already. Now try, mvn install -DskipTests. Everything should go fine and you should get BUILD SUCCESS.
P.S: Thanks ThomasJungblut
Try running 'protoc' and verify that it has the right GCC libraries installed. That was a problem I had on a build from source. It happened when I ran the command in a window where I had sourced a bunch of library configs for a software program, which seemed to screw up the proper library loading.
I had built and installed the google protoc from source, and downloaded maven, and installed, and worked off the raw git of hadoop-common. No other prep work changed from standard Centos6.