I have download Apache Spark 1.4.1 from the official site. As follows:
I don't have hadoop installed in my machine.
Apache provides build command. So, I tried to start building the project using following command
build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package
But build was failed with the following error:
[INFO] Spark Project External Kafka Assembly ............. SKIPPED
[INFO] Spark Project YARN Shuffle Service ................ SKIPPED
[INFO] --------------------------------------------------------------------- ---
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 7.840s
[INFO] Finished at: Wed Jul 29 10:43:04 IST 2015
[INFO] Final Memory: 15M/43M
[INFO] ------------------------------------------------------------------------
[ERROR] Plugin org.apache.maven.plugins:maven-enforcer-plugin:1.4 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-enforcer-plugin:jar:1.4: Could not transfer artifact org.apache.maven.plugins:maven-enforcer-plugin:pom:1.4 from/to central (https://repo1.maven.org/maven2): peer not authenticated -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException
I am new with Apache Spark please give suggestions.
since it is precompiled Binary distribution you have downloaded. you do not need to compile it again using maven2. just put it on path and use it directly.
Related
Im trying to add GIRAPH 1.1.0 to HADOOP 2.6.0
I have to edit the pom.xml somehow in order to package GIRAPH correctly. I run the command mvn -Phadoop_yarn -Dhadoop.version=2.6.0 package
I edited the default pom.xml file in the line (1292) :
<id>hadoop_2</id>
<modules>
<module>giraph-accumulo</module>
<module>giraph-hbase</module>
<module>giraph-hcatalog</module>
<module>giraph-hive</module>
<module>giraph-gora</module>
<module>giraph-rexster</module>
<module>giraph-dist</module>
</modules>
<properties>
<hadoop.version>2.6.0</hadoop.version>
but when I run the command it gives
[INFO] Apache Giraph Parent .............................. SUCCESS [4.183s]
[INFO] Apache Giraph Core ................................ FAILURE [5.364s]
[INFO] Apache Giraph Examples ............................ SKIPPED
[INFO] Apache Giraph Distribution ........................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 9.911s
[INFO] Finished at: Mon Mar 16 19:05:38 EET 2015
[INFO] Final Memory: 55M/1020M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on project giraph-core: Compilation failure: Compilation failure:
[ERROR] /usr/local/giraph/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,68] cannot find symbol
[ERROR] symbol: variable SASL_PROPS
[ERROR] location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] /usr/local/giraph/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62] cannot find symbol
[ERROR] symbol: variable SASL_PROPS
[ERROR] location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :giraph-core
my question is how should the pom.xml be modified for it to work with hadoop 2.6.0 ?
You are changing the hadoop version of the wrong profile. With the "-P" parameter, you are choosing it wrong.
For building giraph with hadoop_2 profile, you should use -Phadoop_2 instead of -Phadoop_yarn, like this:
mvn -Phadoop_2 -Dhadoop.version=2.6.0 package
But, if you use Phadoop_2 profile, you won't be able to resolve the error that is related to SASL_PROPS variable, so, for avoiding it, you should use -Phadoop_yarn profile (but editing the hadoop version in the hadoop_yarn profile, instead).
I'm trying to install my grails plugin to local or remote repo, but when I run
mvn install -DskipTests I'd always get this message:
|Loading Grails 2.3.9
|Configuring classpath
|Running pre-compiled script
.
|Environment set to development
................................
|Packaging Grails application
................................
|Plugin packaged grails-mcplugin-license-0.4.3.zip
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 20.532s
[INFO] Finished at: Thu Jun 05 14:12:03 MSK 2014
[INFO] Final Memory: 13M/153M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.grails:grails-maven-plugin:2.4.0:package-plugin (default->package-plugin) on project McpluginLicense: Unable to copy the plugin ZIP to the target >directory -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
I'm using grails-maven plugin right from github because it solves this problem -https://jira.grails.org/browse/MAVEN-237, can it be related to my issue? How do I track it down? I tried to look at maven debug output but I didn't notice anything that can help
I suggest installing the release plugin and then running
grails maven-install
The documentation is here.
I'm trying to following along with the Cloudera CDK-examples Demo (https://github.com/cloudera/cdk-examples/tree/master/demo); this is part of a conference (Big Data Techon 2013) session by Cloudera on Hadoop App development
However, I am unable to get mvn install to build successfully. I keep getting warnings like this
[WARNING] The POM for com.cloudera.cdk:cdk-maven-plugin:jar:0.8.1-SNAPSHOT is missing, no dependency information available
[WARNING] Failed to retrieve plugin descriptor for com.cloudera.cdk:cdk-maven-plugin:0.8.1-SNAPSHOT: Plugin com.cloudera.cdk:cdk-maven-plugin:0.8.1-SNAPSHOT or one of its dependencies could not be resolved: Failed to read artifact descriptor for com.cloudera.cdk:cdk-maven-plugin:jar:0.8.1-SNAPSHOT
and then it fails with these errors... I tried to troubleshoot on my own using Google and the link mentioned in the error, but it was to no avail
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4.263s
[INFO] Finished at: Tue Oct 15 15:21:23 PDT 2013
[INFO] Final Memory: 6M/57M
[INFO] ------------------------------------------------------------------------
[ERROR] No plugin found for prefix 'cdk' in the current project and in the plugin groups [org.apache.maven.plugins, org.codehaus.mojo] available from the repositories [local (/home/cloudera/.m2/repository), cdh.repo (https://repository.cloudera.com/artifactory/cloudera-repos), central (http://repo.maven.apache.org/maven2)] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/NoPluginFoundForPrefixException
Looking at the pom.xml here, I see that the snapshots is disabled for both repository and pluginRepository.
You could try to enable it and try building.
When I try to build hadoop using:
mvn install -e -DskipTests
It always throw the following error:
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /home/xiu/myGit/hadoop2/hadoop-mr1-project/hadoop-mr1/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java:[153,4] getDelegationTokenSecretManager() is not public in org.apache.hadoop.hdfs.server.namenode.FSNamesystem; cannot be accessed from outside package
[INFO] 1 error
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop MR1 ................................. FAILURE [3.144s]
[INFO] Apache Hadoop MR1 Examples ........................ SKIPPED
[INFO] Apache Hadoop MR1 Project ......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.353s
[INFO] Finished at: Thu Jul 18 11:01:30 PDT 2013
[INFO] Final Memory: 32M/100M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile (default-testCompile) on project hadoop-mr1: Compilation failure
[ERROR] /home/xiu/myGit/hadoop2/hadoop-mr1-project/hadoop-mr1/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java:[153,4] getDelegationTokenSecretManager() is not public in org.apache.hadoop.hdfs.server.namenode.FSNamesystem; cannot be accessed from outside package
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
I have tried clean and reinstall different versions of maven with no luck. Any expert knows what is going on here?
I came across same problem during hadoop 2.2.0 source code building. During "mvn install -DskipTests" this error came in "Hadoop Auth" Folder.
From somewhere(i dont remember from where) i came to know that there is one missing dependency in pom.xml of this Hadoop Auth folder
dependency is
<dependency>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty-util</artifactId>
<scope>test</scope>
</dependency>
I added this dependency and again tried "mvn install -DskipTests". My error resolved.
You just try to add this dependency in pom.xml of "APACHE Hadoop MR1". May ur error will resolve
Seems like Maven compiler plugin complaining that getDelegationTokenSecretManager() method in Hadoop FSNameSystem is not accessible due to method visibility i.e. it's not public:
[ERROR] /home/xiu/myGit/hadoop2/hadoop-mr1-project/hadoop-mr1/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java:[153,4] getDelegationTokenSecretManager() is not public in org.apache.hadoop.hdfs.server.namenode.FSNamesystem; cannot be accessed from outside package
[INFO] 1 error
I would suggest you to check on your Hadoop version defined in your pom.xml, and double check that the method is really accessible in the Hadoop version JavaDoc.
So I'm following the tutorials online to build a cloud connector but I can't even get the skeleton to be generated!
tutorial 1
tutorial 2
I get the following error, using the create-cloud-connector.bat script given by the Mule Cloud Connect Development Kit
> [INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.170s
[INFO] Finished at: Mon Jul 30 13:00:57 EDT 2012
[INFO] Final Memory: 7M/122M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-archetype-plugin:2.2:generate (default-cli) on project standalone-pom: The desired archetype does not exist (org.mule.tools:mule-cloud-connector-archetype:2.0.16) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
what are some things I could be doing wrong?
I updated the Maven settings.xml file
the first question is " Name of the cloud service the new connector uses
[default: ] " can I enter anything there? (or does it have to be like an existing service? what if I'm building one that doesn't exist?)
It looks like the tutorial is pointing at rather old version of the archetype (2.0.16) that maven can't even find.
I recommend to replace the generation step with the following command
mvn archetype:generate -DarchetypeGroupId=org.mule.tools.devkit -DarchetypeArtifactId=mule-devkit-archetype-cloud-connector -DarchetypeVersion=3.3.0
You can find additional informations here