At the moment I'm trying to run my first flink application. I already tested the java file (KMeans.java) in the IDE and it works perfectly but I can't handle to get this java file run as an jar in command line.
The build was successfully created with mvn clean package.
But if I run my jar file in command line flink run -c KMeans name.jar
this error message appears:
The program finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: The
program's entry point class 'KMeans' was not found in the jar file.
at
org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:617)
at
org.apache.flink.client.program.PackagedProgram.(PackagedProgram.java:199)
at
org.apache.flink.client.cli.CliFrontend.buildProgram(CliFrontend.java:856)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:206)
at
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1044)
at
org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1120)
at java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
at
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at
org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1120)
Caused by: java.lang.ClassNotFoundException: KMeans at
java.net.URLClassLoader.findClass(URLClassLoader.java:381) at
java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
java.lang.ClassLoader.loadClass(ClassLoader.java:357) at
java.lang.Class.forName0(Native Method) at
java.lang.Class.forName(Class.java:348) at
org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:614)
... 10 more
So I looked up my generated target folder and there is a KMeans.class file in the classes folder. So I'm doing this wrong?
You need to specify the full class name, e.g., org.apache.flink.examples.java.clustering.KMeans.
Note that you only need to use the -c flag if the JAR file doesn't specify the class to run in its manifest.
Related
The strange thing is that I don't use the missing class DispatchApplication and because I have no error when launching it using idea, so i do not know how to debug it.
I tried to clean idea cache, clean local maven repository, restart my PC
but my colleague can run with a jar, and each of them don't know why this happen and how solve this
D:\projectFolder\midel-data\data-integration\data-integration-service\target>java -jar data-integration-service-1.0.jar
Exception in thread "main" java.lang.ClassNotFoundException: com.lty.dispatch.DispatchApplication
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:151)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:46)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)
This looks to be an issue with dependencies.
You might want to check your IDE settings if you have explicitly enabled any additional jars in the class path.
Also, check out your pom.xml/.gradle file for any difference in dependency versions.
If this class is part of some custom jar, make sure you include it while packaging your jar.
I'm trying to build this project from github:
https://github.com/tomakehurst/wiremock
The thing is, when I download it and run gradle build, I don't think all the required jar files are being loaded onto the class path. I get this error when I actually try to run the thing:
Exception in thread "main" java.lang.NoClassDefFoundError: joptsimple/OptionParser
at com.github.tomakehurst.wiremock.standalone.CommandLineOptions.<init>(CommandLineOptions.java:75)
at com.github.tomakehurst.wiremock.standalone.WireMockServerRunner.run(WireMockServerRunner.java:49)
at com.github.tomakehurst.wiremock.standalone.WireMockServerRunner.main(WireMockServerRunner.java:110)
Caused by: java.lang.ClassNotFoundException: joptsimple.OptionParser
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 3 more
What gives? Isn't grade supposed to have downloaded this dependency by itself?
Try to execute gradle clean build jarAll
See this github issue for reference
From what I see in the build code, the task jarAll handles the joptsimple classes (and various others)
Working in the Apache Pig interactive shell in HDP 2.3 for Windows, I've got an existing ORC file in /path/to/file. If I load and then save that using:
a = LOAD '/path/to/file' USING OrcStorage('');
STORE a INTO '/path/to/second_file' USING OrcStorage('');
Then everything works. However, if I try:
a = LOAD '/path/to/file' USING OrcStorage('');
b = LIMIT a 10;
STORE b INTO '/path/to/third_file' USING OrcStorage('');
Then I get the following error traceback in the logs for the second job (out of two that it schedules):
2015-08-25 16:03:42,161 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.lang.NoClassDefFoundError: org/apache/hadoop/hive/ql/io/orc/OrcNewOutputFormat
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:657)
at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:726)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getStoreFunc(POStore.java:251)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.getCommitters(PigOutputCommitter.java:88)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.<init>(PigOutputCommitter.java:71)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getOutputCommitter(PigOutputFormat.java:289)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:476)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:458)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1560)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:458)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:377)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1518)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1515)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1448)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.io.orc.OrcNewOutputFormat
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
I suspect that the classpath for the two jobs is different, causing a ClassNotFound. Is that likely to be the case? If so, how can I fix it? (Bonus question: Why has this happened?)
Check the dependent library for OrcStorage is placed in all nodes.
The first option only spawn single job
The second option will spawn multiple jobs which maybe run in different machine
which doesnt have the dependent library in its classpath.
I am new to Hadoop and I have just setup Hadoop 1.2.1 on my Mac laptop (Mavericks). I then created a simple WordCount project in IntelliJ IDEA and was able to run the code on a dummy text file. I am having trouble with successfully creating a jar file which will replicate my execution through the IDE. I get the following error:
java -jar ./out/artifacts/WordCount_jar/WordCount.jar test.txt out [19:35:21]
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory
at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:146)
at neu.cs.parallelprogramming.WordCount.main(WordCount.java:48)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 2 more
FAIL: 1
Could anyone let me know what I am missing?
I guess you have to specify your class (which implements the Map/Reduce function).
E.g., $ java -jar ./WordCount.jar classWordCount input.txt output
or $ hadoop jar yourprogram.jar **yourclass** inputpath outputpath
Can somebody tell me what does this error means? and how can I get the output?
Exception in thread "main" java.lang.ClassNotFoundException: org.myorg.LineIndexer
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
This is the code I want to execute :
http://code.google.com/p/hadoop-excercise/source/browse/trunk/lineindexer/LineIndexer.java?spec=svn15&r=15
Is the class included in your jar? Seems not to be the case.
So you have to include the class in the jar, you are passing while starting the job.
I followed the following steps and it worked.
PS: Please make sure you have a sample.txt file in the HDFS and LineIndexer.java in the current directory.
javac -classpath $HADOOP_HOME/hadoop-core.jar *.java
jar cvf li.jar *.class
hadoop jar ii.jar LineIndexer sample.txt li1
hadoop fs -cat li1/part-00000