When I tried to start-all.sh namenode is not running with the log below: Please help
ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.lang.ClassFormatError: Unknown constant tag 59 in class file org/mortbay/util/DateCache
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
at org.mortbay.log.StdErrLog.<clinit>(StdErrLog.java:38)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:188)
at org.mortbay.log.Log.class$(Log.java:67)
at org.mortbay.log.Log.<clinit>(Log.java:72)
at org.mortbay.component.Container.add(Container.java:200)
at org.mortbay.component.Container.update(Container.java:164)
at org.mortbay.component.Container.update(Container.java:106)
at org.mortbay.jetty.Server.setConnectors(Server.java:160)
at org.mortbay.jetty.Server.addConnector(Server.java:134)
at org.apache.hadoop.http.HttpServer.<init>(HttpServer.java:158)
at org.apache.hadoop.http.HttpServer.<init>(HttpServer.java:137)
at org.apache.hadoop.hdfs.server.namenode.NameNode$1.run(NameNode.java:363)
at org.apache.hadoop.hdfs.server.namenode.NameNode$1.run(NameNode.java:358)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:358)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:306)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:461)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1208)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1217)
2014-02-21 15:13:07,336 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at DWNPCPU229/127.0.1.1
************************************************************/
UPDATE
hadoop version is Hadoop 0.20.2-cdh3u0
and,
java version is java version "1.7.0_15"
Could it be a incompatible problem.
I was right. The problem was with version incompatibility. Now I installed Hadoop 0.20.2-cdh3u4 and it start well.
Related
I launched a Dataproc cluster with Hadoop 2.9.2, downloaded Flink 1.7.2 and tried launching it using the following command:
./bin/yarn-session.sh -n 2
This failed with the following error message:
Setting HADOOP_CONF_DIR=/etc/hadoop/conf because no HADOOP_CONF_DIR was set.
2019-02-15 12:56:05,679 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, localhost
2019-02-15 12:56:05,680 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123
2019-02-15 12:56:05,681 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 1024m
2019-02-15 12:56:05,681 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 1024m
2019-02-15 12:56:05,681 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2019-02-15 12:56:05,681 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1
2019-02-15 12:56:05,682 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.port, 8081
2019-02-15 12:56:06,241 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-02-15 12:56:06,323 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to robert (auth:SIMPLE)
2019-02-15 12:56:06,534 ERROR org.apache.flink.yarn.cli.FlinkYarnSessionCli - Error while running the Flink Yarn session.
java.lang.NoClassDefFoundError: javax/ws/rs/ext/MessageBodyReader
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.hadoop.yarn.util.timeline.TimelineUtils.<clinit>(TimelineUtils.java:50)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:179)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.getClusterDescriptor(FlinkYarnSessionCli.java:984)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.createDescriptor(FlinkYarnSessionCli.java:273)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.createClusterDescriptor(FlinkYarnSessionCli.java:451)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:588)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$2(FlinkYarnSessionCli.java:810)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:810)
Caused by: java.lang.ClassNotFoundException: javax.ws.rs.ext.MessageBodyReader
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 49 more
Normally, the best way to install Flink would be to use the initialization action which installs via apt-get from Dataproc's distro to get the version of Flink that has been cross-compiled and tested against the given Dataproc version. In that case, you shouldn't need to modify any environment variables for it to work.
If you do want to use a specific version of Flink with a Dataproc version which doesn't provide that version of Flink in the distro, indeed you may need to install from a download of Flink like you have done. You can also look at one of the earlier versions of the initialization action when it installed Flink from a tarball to see the kinds of configuration that may have been necessary. For example, based on that earlier init action you may also want to export HADOOP_CONF_DIR=/etc/hadoop/conf.
To solve this problem, I needed to export the following variable:
export HADOOP_CLASSPATH=`hadoop classpath`
Getting this error while I executed start-all.cmd command. Also I am unable to access http://localhost:8088 but I am able to acceess http://localhost:9870
The error code below is from the Resource Manager command prompt
FATAL resourcemanager.ResourceManager: Error starting ResourceManager
java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.getDeclaredMethods(Class.java:1975)
at com.google.inject.spi.InjectionPoint.getInjectionPoints(InjectionPoint.java:688)
at com.google.inject.spi.InjectionPoint.forInstanceMethodsAndFields(InjectionPoint.java:380)
at com.google.inject.spi.InjectionPoint.forInstanceMethodsAndFields(InjectionPoint.java:399)
at com.google.inject.internal.BindingBuilder.toInstance(BindingBuilder.java:84)
at org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebApp.setup(RMWebApp.java:56)
at org.apache.hadoop.yarn.webapp.WebApp.configureServlets(WebApp.java:160)
at com.google.inject.servlet.ServletModule.configure(ServletModule.java:55)
at com.google.inject.AbstractModule.configure(AbstractModule.java:62)
at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340)
at com.google.inject.spi.Elements.getElements(Elements.java:110)
at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:138)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104)
at com.google.inject.Guice.createInjector(Guice.java:96)
at com.google.inject.Guice.createInjector(Guice.java:73)
at com.google.inject.Guice.createInjector(Guice.java:62)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.build(WebApps.java:356)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:401)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:1137)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1245)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1446)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollectorManager
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 36 more
2018-06-30 15:32:39,168 INFO resourcemanager.ResourceManager: Transitioning to standby state
2018-06-30 15:32:39,169 INFO resourcemanager.ResourceManager: Transitioned to standby state
2018-06-30 15:32:39,169 INFO resourcemanager.ResourceManager: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down ResourceManager at HP/someipaddress
Use java version 1.8.x_xxx
Verify java version in command prompt >java -version
copy "hadoop-yarn-server-timelineservice-3.x.x" from ~\hadoop-3.x.x\share\hadoop\yarn\timelineservice to ~\hadoop-3.x.x\share\hadoop\yarn folder.
the error disappears and the resource manager works fine.
I installed giraph-1.0.0 on my 2-node cluster, built with hadoop version 2.0.4-alpha where as i have hadoop 2.7.1 installed on my cluster. When I tried to run the giraph example for Simple Shortest Path Vertex, I got the following NoClassDefFoundError.
hadoop jar $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.0.0-for-hadoop-2.0.4-alpha-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsVertex -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/hduser/input/tinygraph.txt -of org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/hduser/output/shortestpaths -w 1
The error is
16/08/05 14:26:09 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one.
16/08/05 14:26:09 WARN job.GiraphConfigurationValidator: Output format vertex index type is not known
16/08/05 14:26:09 WARN job.GiraphConfigurationValidator: Output format vertex value type is not known
16/08/05 14:26:09 WARN job.GiraphConfigurationValidator: Output format edge value type is not known
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/client/YarnClientImpl
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:803)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:83)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:126)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.client.YarnClientImpl
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 21 more
The command I used for building giraph is this.
mvn -Phadoop_yarn -DskipTests -Dhadoop.version=2.0.4-alpha clean package
I am not sure if I need to rebuild apache giraph or it is something else which is the problem. Thanks in advance.
I am doing sqoop import in MAC OSX 10.9.4 and getting error as:
14/10/24 11:51:41 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5
14/10/24 11:51:41 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
14/10/24 11:51:41 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
14/10/24 11:51:41 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
14/10/24 11:51:41 INFO tool.CodeGenTool: Beginning code generation
14/10/24 11:51:42 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `test_log` AS t LIMIT 1
14/10/24 11:51:42 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `test_log` AS t LIMIT 1
14/10/24 11:51:42 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /Users/hadoop/hadoop-2.5.1/share/hadoop/mapreduce
Note: /tmp/sqoop-hadoop/compile/2398ed5be0489f1c76d6fb556f3e0e9d/test_log.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
14/10/24 11:51:45 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/2398ed5be0489f1c76d6fb556f3e0e9d/test_log.jar
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/InputFormat
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:455)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.sqoop.manager.ImportJobContext.<init>(ImportJobContext.java:51)
at com.cloudera.sqoop.manager.ImportJobContext.<init>(ImportJobContext.java:33)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:483)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.InputFormat
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 58 more
Hadoop version : 2.5.1
Sqoop version :1.4.5
os : OSX10.9.4
env varibles are set:
export HADOOP_COMMON_HOME=/Users/hadoop/hadoop-2.5.1/
export HADOOP_MAPRED_HOME=/Users/hadoop/hadoop-2.5.1/share/hadoop/mapreduce/
Sqoop jar (sqoop-1.4.5.jar) is placed in /Users/hadoop/hadoop-2.5.1/lib/
and mysql connector in /Users/hadoop/sqoop/lib
also the core.jar exist in
/Users/hadoop/hadoop-2.5.1/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.1.jar
for f in ${HADOOP_MAPRED_HOME}/*.jar; do
HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:$f;
done
add this to bin/configure-sqoop
Somehow the Jars were not accessible. After puting the Sqoop jars in HDFS, Sqoop import went fine.
This link helped
Sqoop jar files not found
cd $SQOOP_HOME/bin
vim configure-sqoop
add below line
add_to_classpath ${HADOOP_MAPRED_HOME}
after "add_to_classpath ${SQOOP_JAR_DIR}"
Try to check your if in your $HADOOP_COMMON_LIB_NATIVE_DIR you can find the library.
If not you can put it in the same dir of your mapreduce implementation:
/Users/hadoop/hadoop-2.5.1/share/hadoop/mapreduce/
I had the same problem ! Just copy hadoop-mapreduce-client-core-3.0.0-SNAPSHOT.jar into the lib folder from sqooq. Dirty but it works. You probably just want to make some tests
I am new to pig and i have downloaded from
http://apache.techartifact.com/mirror/pig/pig-0.10.1/
Now when i write pig in my linux terminal it displays the following message
2013-04-26 17:14:53,641 [main] INFO org.apache.pig.Main - Logging error messages to: /home/vishal/Downloads/pig_1366976693634.log
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/JobConf
at org.apache.pig.Main.run(Main.java:587)
at org.apache.pig.Main.main(Main.java:111)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.JobConf
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 7 mor
Is it that i have to include jar or what else can be the issue
Thanks
You need to include the mapred java archive, depending on which MapReduce version you use, MRv1 or MRv2 (=YARN).
FYI: java.lang.NoClassDefFoundError is always about a forgotten/mistyped JAR-file