Problems compiling Hadoop - compilation

That's the problem: I have done a simply Hadoop program to "clean" a graph saved in a text file that I will use later (with Hadoop), but I can't compile it!
The compiler can't find Hadoop classes (IntWritable, Text ecc...), and each time I get a "cannot find symbol" error.
I've tried with:
javac -classpath path/to/hadoop/root/hadoop-core-{version}.jar filename.java
I'm running with ubuntu 11.04, and the Hadoop version is 1.0.3.

the problem is that hadoop-core-{version}.jar depends on some other jars. You can find all the dependencies on the Maven repository web site :
http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-core/1.0.3
You should use Maven or add all the dependencies to your project to be able to build it.

Related

Error: Unable to access jarfile build/libs/gs-spring-boot-0.1.0.jar?

I follow the instructions in https://spring.io/guides/gs/spring-boot/#scratch, but when it says to run:
./gradlew build && java -jar build/libs/gs-spring-boot-0.1.0.jar
the build fails with the above error.
There is message before the failure that says:
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
See https://docs.gradle.org/4.8.1/userguide/command_line_interface.html#sec:command_line_warnings
but everyone online says that's just a warning.
The build doesn't appear to create or download build/libs/gs-spring-boot-0.1.0.jar.
Currently completely blocked on first attempt to use Gradle.
I just had this problem.
The tutorial is in error in what you need to run. It should be
$ gradlew build && java -jar build/libs/gs-rest-service-0.1.0.jar
I think that they updated the code, but forgot to update the tutorial.
I had the same issue when build a simple project with Maven on Intellij IDEA. (Ubuntu 18.04.2).
Just typed terminal (in project directory):
$ sudo mvn package
$ java -jar ./target/(your-project-name)-(<version> at pom.xml).jar
For example my project name is hello-world-spring and version name in pom.xml is <version>0.0.1-SNAPSHOT</version>, I have to type:
$ sudo mvn package
$ java -jar ./target/hello-world-spring-0.0.1-SNAPSHOT.jar
Maybe this method can work for gradle as well.
Please check the path of the jar file build/libs/gs-spring-boot-0.1.0.jar. For your case, the jar might be in a different folder. If your code is in a module in the main project, then the jar will be in the build folder of the module.
If you git clone the repo, then the tutorial works. If you "To start from scratch, move on to Build with Gradle.", then the tutorial doesn't work. There are missing setup steps.
I got the same issue and I changed the command to java -jar target/rest-service-0.0.1-SNAPSHOT.jar (I checked the .jar file in target folder and found that the file name was incorrect).
Parent folder of my project was having spaces in it's name, i changed it to the underscore and it worked.
Looked at the command line as it was in the official guide:
./gradlew clean build && java -jar build/libs/gs-actuator-service-0.1.0.jar
First, the above command line has two parts:
(1) ./gradlew clean build //Use gradle wrapper to build
(2) java -jar build/libs/gs-actuator-service-0.1.0.jar //To run an application packaged as a JAR file
Now, one might run into issues with one part or both parts. Separating them and running just on thing at a time helped troubleshoot.
(1) didn't work for my Windows, I did the following instead and that built the application successfully.
.\gradlew.bat clean build
Now moving to (2) java -jar build/libs/gs-actuator-service-0.1.0.jar
It literally means that "Run a jar file that is called gs-actuator-service-0.1.0.jar under this directory/path: build/libs/" Again, for Windows, this translates to build\libs\ , and there's one more thing that may catch you: The jar file name can be slightly different depending on how it was actually named by the configuration in initial/setting.gradle:
rootProject.name = 'actuator-service'
Note that the official guide changed it from 'gs-actuator-service' to 'actuator-service' in their sample code but hasn't updated the tutorial accordingly. But now you know where the jar file name comes from, that doesn't matter anymore, and you have the choice to rename it however you want.
Having all the factors adjusted, below is what eventually worked in my case:
java -jar build\libs\actuator-service-0.0.1-SNAPSHOT.jar
or
java -jar C:\MyWorkspace\Spring\gs-actuator-service\initial\build\libs\actuator-service-0.0.1-SNAPSHOT.jar //with fully qualified path
If you are curious where does "-0.0.1-SNAPSHOT" come from, here it is:
in build.gradle
version = '0.0.1-SNAPSHOT'
Again, you have the choice to modify it however you want. For example, if I changed it to 0.0.2-SNAPSHOT, the command line should be adjusted accordingly
java -jar build\libs\actuator-service-0.0.2-SNAPSHOT.jar
Reference: https://docs.oracle.com/javase/tutorial/deployment/jar/basicsindex.html
Because you are trying to execute .jar file that doesn't exist. After building the project go to ./build/libs and check the name of freshly built .jar file and then in your project directory run:
./gradlew build && java -jar build/libs/name-of-your-jar-file.jar
or you can set version property to empty string in your build.gradle file
version = ''
after that:
./gradlew build && java -jar build/libs/your-project-name.jar
For Windows, these commands solved the problem: "Error: Unable to access jarfile springboot.jar":
cd target
java -jar springboot-0.0.1-SNAPSHOT.jar
run ./mvnw package
Now a folder named target is created and you can see a jar file inside it.
then execute java -jar target/<jarfilename>

How to run Apache Tez Locally?

One of the ways that one can execute tez is in local mode beside integrated with hadoop. In order to run it localy
I read this page and understood the changes I have to make and I updated tez-site.xml configuration. But I don't know how to start it.
I tried running one of the tez-examples (e.g. wordCount) that has a main method. But it stalls and don't print anything to stdout. Is there anything that I have to start first?
How can I run tez in local mode?
I managed to run it with including needed libraries. I could changing pom and build the final jar file with the dependencies, but I preferred not to change the project.
After building it with mvn clean install -DskipTests=true -Dmaven.javadoc.skip=true
I ran it with setting java classpath:
java -cp tez-dist/target/tez-0.7.0/lib/*:tez-dist/target/tez-0.7.0/* org.apache.tez.examples.OrderedWordCount in.txt out

Example Jar in Hadoop release

I am learning Hadoop with book 'Hadoop in Action' by Chuck Lam. In first chapter the books says that Hadoop installation will have example jar and by running 'hadoop jar hadoop-*-examples.jar' will show all the examples. But when I run the command then it throw error 'Could not find or load main class org.apache.hadoop.util.RunJar'. My guess is that installed Hadoop doesn't have example jar. I have installed 'hadoop-2.1.0-beta.tar.gz' on cygwin on Win 7 laptop. Please suggest how to get example jar.
run following command
hadoop jar PathToYourJarFile wordcount inputPath OutputPath
you can get examples jar file at your hadoop installation directory
What I can suggest here is you should manually go to the Hadoop installation directory and look for a jar name similar to hadoop-examples.jar yourself. Different distribution can have different names for the jar.
If you are in Cygwin, while in the Hadoop Installation directory you can also do a ls *examples*.jar to find the same, narrowing down the file listing to any jar file containing examples as a string.
You can then directly use the jar file name like --
hadoop jar <exampleJarYourFound.jar>
Hope this takes you to a solution.

How to run GIS codes through hadoop's prompt?

I am running a GIS code through hadoop's prompt in following manner:
Wrote the GIS code in Eclipse including all the GIS jars (relevant).
Went into the dir. where my eclipse workspace is.
Compiled the code by adding all the relevant jars in the classpath. *(The compilation was successful).
Built the jar.
Now running the same jar using hadoop: bin/hadoop jar my_jar_file_name.jar my_pkg_structure.Main_class_file
Now, inspite of the code being error free, when i try to execute through hadoop's propmpt, it gives me multiple issues.
Is there a workable alternative way to do the same without any hassles?
Also note, the gid code runs beautifully in eclipse. Since, I have to do Geo processing over hadoop, I need to run it through hadoop's prompt.

How run mahout in action example ReutersToSparseVectors?

I want run "ReutersToSparseVectors.java". I can compile and created JAR file without problem.
I compiled this file by below command:
javac -classpath hadoop-core-0.20.205.0.jar:lucene-core-3.6.0.jar:mahout-core-0.7.jar:mahout-math-0.7.jar ReutersToSparseVectors.java
created JAR file with below command:
jar cvf ReutersToSparseVectors.jar ReutersToSparseVectors.class
When I write java -jar ReutersToSparseVectors.jar to run, give me below error:
Failed to load Main-Class manifest attribute from
ReutersToSparseVectors.jar
Do you can help me to solve this problem?
IF this example can run with hadoop, please me that how i can run this with hadoop.
instead of using -jar option, then it's better to to run:
java -cp mahout-core.jar:... mia.clustering.ch09.ReutersToSparseVectors
or you can use mvn exec:java command, as described in README for examples...
mvn exec:java -Dexec.mainClass="mia.clustering.ch09.ReutersToSparseVectors"
Or you can run this file directly from your IDE (assuming, that you correctly imported Maven project).
P.S. your command isn't working, because to run with -jar switch, the .jar file should have special entry in manifest that describes that class should be started by default...
P.P.S. It's better to use book's examples with Mahout 0.7, as they were tested for it. You can use it with version 0.7 if you need, by then you need to take source code from mahout-0.7 branch of repository with examples (link is above)

Resources