I'm running a mahout job on Amazon EMR and getting the following exception:
ArrayUtil.oversize(II)I
attempt_201311181700_0002_m_000000_0: SLF4J: Class path contains multiple SLF4J bindings.
attempt_201311181700_0002_m_000000_0: SLF4J: Found binding in [jar:file:/home/hadoop/lib/slf4j-log4j12-1.7.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201311181700_0002_m_000000_0: SLF4J: Found binding in [jar:file:/mnt/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job_201311181700_0002/jars/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201311181700_0002_m_000000_0: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
attempt_201311181700_0002_m_000000_0: SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Error: org.apache.lucene.util.ArrayUtil.oversize(II)I
attempt_201311181700_0002_m_000000_1: SLF4J: Class path contains multiple SLF4J bindings.
attempt_201311181700_0002_m_000000_1: SLF4J: Found binding in [jar:file:/home/hadoop/lib/slf4j-log4j12-1.7.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201311181700_0002_m_000000_1: SLF4J: Found binding in [jar:file:/mnt/var/lib/hadoop/mapred/taskTracker/hadoop/jobcache/job_201311181700_0002/jars/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201311181700_0002_m_000000_1: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
attempt_201311181700_0002_m_000000_1: SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Error: org.apache.lucene.util.ArrayUtil.oversize(II)I
Exception in thread "main" java.lang.IllegalStateException: Job failed!
at org.apache.mahout.vectorizer.collocations.llr.CollocDriver.generateCollocations(CollocDriver.java:238)
at org.apache.mahout.vectorizer.collocations.llr.CollocDriver.generateAllGrams(CollocDriver.java:187)
at org.apache.mahout.vectorizer.DictionaryVectorizer.createTermFrequencyVectors(DictionaryVectorizer.java:184)
at clustering.AmazonClusteringDriver.main(AmazonClusteringDriver.java:122)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:187)
I excluded the slf4j dependency in mahout dependency; however, it doesn't solve the problem. So, where is the problem?
This is a very wrong place to ask this.
You should be asking this on the Mahout developer mailing list.
https://cwiki.apache.org/confluence/display/MAHOUT/Mailing+Lists,+IRC+and+Archives#MailingLists%2CIRCandArchives-MahoutUserList
Related
I am getting this error while starting hive database.
G:\hive\bin>hive
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/G:/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/G:/hadoop-3.3.4/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Connecting to jdbc:hive2://
Error applying authorization policy on hive configuration: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
Beeline version 2.1.0 by Apache Hive
Error applying authorization policy on hive configuration: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
Connection is already closed.
I have configured Hadoop on a single node cluster, Hadoop cluster is configured properly even I executed MapReduce in a cluster. Now I have newly installed hive tool and did configuration, but when I start Hive shell, I am getting below error.
[dsawale#localhost apache-hive-2.1.0-bin]$ bin/hive
which: no hbase in (/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/usr/lib/jvm/jre-1.8.0-openjdk/bin:/home/dsawale/hadoop-3.2.1/bin:/home/dsawale/kafka_2.11-2.4.1/bin:/home/dsawale/sqoop-1.4.7.bin__hadoop-2.6.0/bin:/home/dsawale/apache-hive-2.1.0-bin/bin:/home/dsawale/.local/bin:/home/dsawale/bin:/home/dsawale/spark-2.4.5-bin-hadoop2.7/bin:/usr/lib/jvm/jre-1.8.0-openjdk/bin:/home/dsawale/hadoop-3.2.1/bin:/home/dsawale/kafka_2.11-2.4.1/bin:/home/dsawale/sqoop-1.4.7.bin__hadoop-2.6.0/bin:/home/dsawale/apache-hive-2.1.0-bin/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/dsawale/apache-hive-2.1.0-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/dsawale/hadoop-3.2.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
at org.apache.hadoop.mapred.JobConf.setJar(JobConf.java:536)
at org.apache.hadoop.mapred.JobConf.setJarByClass(JobConf.java:554)
at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:448)
at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:3612)
at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:3570)
at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:76)
at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:60)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:657)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
How can I resolve this?
I am trying to setup http://www.geomesa.org/documentation/tutorials/geomesa-raster.html
I have this jai_imageio-1.1.jar at geomesa libs which contains TIFFImageWriterSpi.
when i try to ingest raster using this command:
geomesa ingestraster -u root -p qwerty -t natearth -f "/home/gaurav/Downloads/CPSC-771/geoserver-2.8.3/data_dir/coverages/retile/1/NE2_HR_LC_SR_W_DR_01_01.tif" -F geotiff
Overall log:
Using GEOMESA_HOME = /home/gaurav/Downloads/CPSC-771/geomesa-1.2.1/dist/tools/geomesa-tools-1.2.1/
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/gaurav/Downloads/CPSC-771/Installs/accumulo-1.6.5/lib/slf4j-log4j12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/gaurav/Downloads/CPSC-771/Installs/zookeeper-3.4.8/contrib/fatjar/zookeeper-3.4.8-fatjar.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Exception in thread "main" java.lang.NoClassDefFoundError: it/geosolutions/imageioimpl/plugins/tiff/TIFFImageWriterSpi
at org.geotools.gce.geotiff.GeoTiffFormat.<clinit>(GeoTiffFormat.java:106)
at org.geotools.gce.geotiff.GeoTiffFormatFactorySpi.createFormat(GeoTiffFormatFactorySpi.java:88)
at org.geotools.coverage.grid.io.GridFormatFinder.findFormats(GridFormatFinder.java:185)
at org.geotools.coverage.grid.io.GridFormatFinder.findFormat(GridFormatFinder.java:236)
at org.geotools.coverage.grid.io.GridFormatFinder.findFormat(GridFormatFinder.java:216)
at org.locationtech.geomesa.tools.ingest.RasterIngest$class.getReader(RasterIngest.scala:57)
at org.locationtech.geomesa.tools.ingest.LocalRasterIngest.getReader(LocalRasterIngest.scala:26)
at org.locationtech.geomesa.tools.ingest.LocalRasterIngest.ingestRasterFromFile(LocalRasterIngest.scala:52)
at org.locationtech.geomesa.tools.ingest.LocalRasterIngest$$anonfun$runIngestTask$1$$anonfun$apply$mcV$sp$1.apply(LocalRasterIngest.scala:48)
at org.locationtech.geomesa.tools.ingest.LocalRasterIngest$$anonfun$runIngestTask$1$$anonfun$apply$mcV$sp$1.apply(LocalRasterIngest.scala:48)
at scala.collection.parallel.mutable.ParArray$ParArrayIterator.foreach_quick(ParArray.scala:143)
at scala.collection.parallel.mutable.ParArray$ParArrayIterator.foreach(ParArray.scala:136)
at scala.collection.parallel.ParIterableLike$Foreach.leaf(ParIterableLike.scala:972)
at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply$mcV$sp(Tasks.scala:49)
at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:48)
at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:48)
at scala.collection.parallel.Task$class.tryLeaf(Tasks.scala:51)
at scala.collection.parallel.ParIterableLike$Foreach.tryLeaf(ParIterableLike.scala:969)
at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask$class.compute(Tasks.scala:152)
at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:443)
at scala.concurrent.forkjoin.RecursiveAction.exec(RecursiveAction.java:160)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.ClassNotFoundException: it.geosolutions.imageioimpl.plugins.tiff.TIFFImageWriterSpi
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 25 more
Read your error message.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/gaurav/Downloads/CPSC-771/Installs/accumulo-1.6.5/lib/slf4j-log4j12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/gaurav/Downloads/CPSC-771/Installs/zookeeper-3.4.8/contrib/fatjar/zookeeper-3.4.8-fatjar.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Here is the link
While i start hive by using hive; and this error appear.
How can i fix this error?
Logging initialized using configuration in jar:file:/usr/local/Cellar/hive/0.14.0/libexec/lib/hive-common-0.14.0.jar!/hive-log4j.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/Cellar/hadoop/2.6.0/libexec/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/Cellar/hive/0.14.0/libexec/lib/hive-jdbc-0.14.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:444)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:672)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
ps.sorry for bad english
I am trying to run mahout on my local system and when I run "./bin/mahout" I get the below mentioned error. All I am trying to do is run mahout without hadoop and try out the 20Newsgroup example.
I did "mvn compile" and "mvn install -Dmaven.test.skip=true" in the core, distribution and example directory. Not sure what else am I missing. I know that you could run mahout without Hadoop running on your system.
Appreciate if someone could help.
hadoop binary is not in PATH,HADOOP_HOME/bin,HADOOP_PREFIX/bin, running locally
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/Aanchal/mahout-distribution-0.7/mahout-examples-0.7-job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/Aanchal/mahout-distribution-0.7/lib/slf4j-jcl-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/Aanchal/mahout-distribution-0.7/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/ProgramDriver
at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:96)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.ProgramDriver
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Add this line
CLASSPATH=${CLASSPATH}:$MAHOUT_HOME/lib/hadoop/hadoop-core-0.20.204.0.jar;
to the end of the section in mahout.sh/bat file
# add release dependencies to CLASSPATH
for f in $MAHOUT_HOME/lib/*.jar; do
CLASSPATH=${CLASSPATH}:$f;
done
This exception indicates that the class was not found on the classpath i.e we are trying to load the class definition and class/jar containing the class does not exist in the classpath.
Please check your PATH and HADOOP_HOME configuration and update these variables accordingly.