I am trying to load an XML file using XMLLoader(Piggybank) in Pig, but I get an error saying "unable to open iterator for alias B".
I have written the following code:
REGISTER /home/hdfs/spig/trunk/contrib/piggybank/java/piggybank.jar
A = LOAD '/core-site.xml'using org.apache.pig.piggybank.storage.XMLLoader('property') as (x:chararray);
B = foreach A GENERATE FLATTEN(REGEX_EXTRACT_ALL(x,'<property>\\s*<name>(.*) </name>\\s*<value>(.*)</value>\\s*<description>(.*)</description>\\s*</property>'));
dump B;
The following is the log file:
Pig Stack Trace
ERROR 1066: Unable to open iterator for alias A
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable
to open iterator for alias A at
org.apache.pig.PigServer.openIterator(PigServer.java:935) at
org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:754)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:376)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) at
org.apache.pig.Main.run(Main.java:631) at
org.apache.pig.Main.main(Main.java:177) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by:
java.io.IOException: Job terminated with anomalous status FAILED at
org.apache.pig.PigServer.openIterator(PigServer.java:927) ... 13 more
Looks like one of your jobs might have failed.
Caused by: java.io.IOException: Job terminated with anomalous status FAILED
Related
I have a use case where i am executing hive query and storing output to a file.
hive -S -e "SELECT * from test.employee where empid=1" > /mapr/Piyush/test/output.txt
The query is executing fine but i am getting logs also along with the data in the file. I am guessing it's because of the log4j properties. The problem here is i have no access to the log4j config file so i can't do any changes in it.
I tried setting couple of configuration like.
set hive.root.logger=ERROR, console
and
set hive.root.logger=INFO,console
and
set hive.server2.logging.operation.enabled=false
But nothing helps. I need just the data from the table in the file. Please let me know if i am missing anything or any solution to this issue.
Sample output file
2017-06-09 11:49:18,265 main ERROR Cannot access RandomAccessFile {}) java.io.FileNotFoundException: /mapr/pankaj-hive.log (Permission denied)
2017-06-09 11:49:18,272 main ERROR Unable to invoke factory method in class class org.apache.logging.log4j.core.appender.RollingRandomAccessFileAppender for element RollingRandomAccessFile. java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:136)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:813)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:753)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:745)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:389)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:169)
at org.apache.logging.log4j.core.config.builder.impl.DefaultConfigurationBuilder.build(DefaultConfigurationBuilder.java:158)
at org.apache.logging.log4j.core.config.builder.impl.DefaultConfigurationBuilder.build(DefaultConfigurationBuilder.java:43)
at org.apache.logging.log4j.core.config.properties.PropertiesConfigurationFactory.getConfiguration(PropertiesConfigurationFactory.java:149)
at org.apache.logging.log4j.core.config.properties.PropertiesConfigurationFactory.getConfiguration(PropertiesConfigurationFactory.java:46)
at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:413)
at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:257)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:519)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:536)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:214)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:146)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:41)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:185)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:103)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:42)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:284)
at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:655)
at org.apache.hadoop.util.ShutdownHookManager.<clinit>(ShutdownHookManager.java:44)
at org.apache.hadoop.util.RunJar.run(RunJar.java:200)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingRandomAccessFileManager$RollingRandomAccessFileManagerFactory#48aca48b] unable to create manager for [/mapr/pankaj-hive.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingRandomAccessFileManager$FactoryData#13fd2ccd]
at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:73)
at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:61)
at org.apache.logging.log4j.core.appender.rolling.RollingRandomAccessFileManager.getRollingRandomAccessFileManager(RollingRandomAccessFileManager.java:84)
at org.apache.logging.log4j.core.appender.RollingRandomAccessFileAppender.createAppender(RollingRandomAccessFileAppender.java:206)
... 33 more
2017-06-09 11:49:18,277 main ERROR Null object returned for RollingRandomAccessFile in Appenders.
2017-06-09 11:49:18,281 main ERROR Unable to locate appender "DRFA" for logger config "root"
2017-06-09 11:49:19,640 main ERROR Cannot access RandomAccessFile {}) java.io.FileNotFoundException: /mapr/pankaj-hive.log (Permission denied)
2017-06-09 11:49:19,644 main ERROR Unable to invoke factory method in class class org.apache.logging.log4j.core.appender.RollingRandomAccessFileAppender for element RollingRandomAccessFile. java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:136)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:813)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:753)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:745)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:389)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:169)
at org.apache.logging.log4j.core.config.builder.impl.DefaultConfigurationBuilder.build(DefaultConfigurationBuilder.java:158)
at org.apache.logging.log4j.core.config.builder.impl.DefaultConfigurationBuilder.build(DefaultConfigurationBuilder.java:43)
at org.apache.logging.log4j.core.config.properties.PropertiesConfigurationFactory.getConfiguration(PropertiesConfigurationFactory.java:149)
at org.apache.logging.log4j.core.config.properties.PropertiesConfigurationFactory.getConfiguration(PropertiesConfigurationFactory.java:46)
at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:236)
at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:445)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:228)
at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:140)
at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:113)
at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:98)
at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:156)
at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jDefault(LogUtils.java:155)
at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:91)
at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:83)
at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:66)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:661)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:646)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingRandomAccessFileManager$RollingRandomAccessFileManagerFactory#48aca48b] unable to create manager for [/mapr/pankaj-hive.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingRandomAccessFileManager$FactoryData#bff34c6]
at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:73)
at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:61)
at org.apache.logging.log4j.core.appender.rolling.RollingRandomAccessFileManager.getRollingRandomAccessFileManager(RollingRandomAccessFileManager.java:84)
at org.apache.logging.log4j.core.appender.RollingRandomAccessFileAppender.createAppender(RollingRandomAccessFileAppender.java:206)
... 34 more
2017-06-09 11:49:19,649 main ERROR Null object returned for RollingRandomAccessFile in Appenders.
2017-06-09 11:49:19,652 main ERROR Unable to locate appender "DRFA" for logger config "root"
1 piyush bangalore 20
I want output as just
1 piyush bangalore 20
Hi as per hive documentation you can't change logging property by setting hive.root.logger via 'set' command because hive reads the logger properties at time of initialization i.e prior to opening cli.
you can do it like this:
hive --hiveconf hive.root.logger=OFF -S -e "SELECT * from test.employee where empid=1" > /mapr/Piyush/test/output.txt
P.S i have assumed that you don't want any logging. You can change logging level as per your requirement.
Below i am attaching screenshot of official documentation with link
Click for official Documentation
I have an error when I try to load or save my data from Apache Pig into anything but CSV. Here is my pig code:
REGISTER /usr/local/Cellar/pig/0.15.0/libexec/*.jar
REGISTER /usr/local/Cellar/pig/0.15.0/libexec/lib/*.jar
REGISTER /usr/local/Cellar/hbase/1.1.2/libexec/lib/*.jar
REGISTER /usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/*.jar
REGISTER /usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/*.jar
REGISTER /usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/*.jar
REGISTER /usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/*.jar
REGISTER /usr/local/Cellar/pig/0.15.0/libexec/lib/piggybank.jar
--generated_data = LOAD 'tableHive' USING org.apache.hive.hcatalog.pig.HCatLoader(',') AS (level:chararray, score:INT, attraction:chararray);
generated_data = LOAD 'CSVResults/ireland.csv' USING PigStorage(',') AS (level:chararray, score:INT, attraction:chararray);
DUMP generated_data;
fiveRating = FILTER generated_data BY (float)score>4;
level6 = FILTER fiveRating BY (float)level>5;
groupedbylevel = group level6 by attraction;
countAttractions = FOREACH groupedbylevel {
level6Attractions = CROSS level6.level;
generate group, COUNT(level6Attractions) AS listBylevel6;
};
orderlist = ORDER countAttractions BY listBylevel6 DESC;
limitorder = LIMIT orderlist 20;
STORE limitorder into 'Level6AttractionsIreland-limited2' using PigStorage(',');
STORE countAttractions into 'hbase://Level6AttractionsIreland' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('Ireland:level Ireland:score Ireland:attraction');
STORE countAttractions INTO 'Level6AttractionsIreland' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage(',');
and here is the error from the pig log file:
Pig Stack Trace
---------------
ERROR 2999: Unexpected internal error. java.io.IOException: java.lang.reflect.InvocationTargetException
java.lang.RuntimeException: java.io.IOException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:211)
at org.apache.pig.backend.hadoop.hbase.HBaseStorage.getOutputFormat(HBaseStorage.java:928)
at org.apache.pig.newplan.logical.visitor.InputOutputFileValidatorVisitor.visit(InputOutputFileValidatorVisitor.java:69)
at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66)
at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
at org.apache.pig.newplan.logical.relational.LogicalPlan.validate(LogicalPlan.java:212)
at org.apache.pig.PigServer$Graph.compile(PigServer.java:1767)
at org.apache.pig.PigServer$Graph.access$300(PigServer.java:1443)
at org.apache.pig.PigServer.execute(PigServer.java:1356)
at org.apache.pig.PigServer.executeBatch(PigServer.java:415)
at org.apache.pig.PigServer.executeBatch(PigServer.java:398)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:171)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
at org.apache.pig.Main.run(Main.java:631)
at org.apache.pig.Main.main(Main.java:177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:459)
at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:436)
at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:317)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:198)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:160)
at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:206)
... 25 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:457)
... 30 more
Caused by: java.lang.NoClassDefFoundError: org/cloudera/htrace/Trace
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:218)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:83)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.retrieveClusterId(HConnectionManager.java:907)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:701)
... 35 more
Caused by: java.lang.ClassNotFoundException: org.cloudera.htrace.Trace
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 41 more
================================================================================
As you can see I've tried to add every jar file that might be relevant, and I've also removed everything but the load and store commands to see if it is the code that is causing it, but I get the same results. I am very new to Pig so apologies if this is a silly mistake, I have searched for answers elsewhere, but nothing is working for me at this stage. Also I am on a mac with Hadoop, HBase and Hive installed locally and I am running the command 'pig -x local test.pig' in terminal. Any advice would be great, thanks!
I'm not sure where you grabbed your dependencies from, but your code is looking for some Cloudera package that doesn't exist, thus the error.
Caused by: java.lang.ClassNotFoundException: org.cloudera.htrace.Trace
I just downloaded Hbase with brew install hbase, and the correct class file is org.apache.htrace.Trace located in /usr/local/Cellar/hbase/.../htrace-core-*.jar
So, I would recommend downloading the latest version of the HBase libraries.
I'm trying to read a csv file on pig shell on mac. All I'm doing is load a file into a variable and dump the variable. Here is how I'm doing it:
movies = LOAD '/user/myhome/movies_data.csv' USING PigStorage(',') as (id,name,year,rating,duration);
DUMP movies;
The data I'm using is downloaded from github provided here
This file is available in locally installed hdfs on my mac. When I do dump I get an error:
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias movies
at org.apache.pig.PigServer.openIterator(PigServer.java:935) at
org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:754)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:376)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66) at
org.apache.pig.Main.run(Main.java:565) at
org.apache.pig.Main.main(Main.java:177) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497) at
org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by:
java.io.IOException: Job terminated with anomalous status FAILED at
org.apache.pig.PigServer.openIterator(PigServer.java:927) ... 13 more
When I hit the app cluster link when this job is run, I get the following exception:
Diagnostics: Exception from container-launch.
Container id: container_1443887668938_0007_02_000001 Exit code: 127
Stack trace: ExitCodeException exitCode=127: at
org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at
org.apache.hadoop.util.Shell.run(Shell.java:455) at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) Container exited with a
non-zero exit code 127 Failing this attempt. Failing the application.
Pig version is 0.15.0 and hadoop is 2.6.1. Am I missing something here?
You could use CSVLoader from piggybank. Get the piggybank jar if not available and register it and use the CSVLoader. Something like this.
register '/your/path/to/piggybank/jar' ;
define CSVLoader org.apache.pig.piggybank.storage.CSVLoader();
movies = LOAD '/user/myhome/movies_data.csv' USING CSVLoader as (id,name,year,rating,duration);
I am new to hive technology.
I have been trying to setup hive in my machine . By whenever i try to start hive by executing hive shell script in bin folder it give a Null pointer execution and finally ExceptionInInitializerError.
Following is the stack trace
Exception in thread "main" java.lang.ExceptionInInitializerError
at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:58)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:641)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
Caused by: java.lang.RuntimeException: Could not load shims in class null
at org.apache.hadoop.hive.shims.ShimLoader.createShim(ShimLoader.java:123)
at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:115)
at org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:80)
at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:248)
... 8 more
Caused by: java.lang.NullPointerException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:186)
at org.apache.hadoop.hive.shims.ShimLoader.createShim(ShimLoader.java:120)
... 11 more
Does any one know how to resolve it ?
I am using apache-hive-0.13.1
and hadoop-0.22.0 .
After installed Hive by the instruction on Hive apache wiki step by step, I invoked hive shell and typed "CREATE TABLE pokes (foo INT, bar STRING);", then it comes following error, log is also included.
I'm new to Hive, any suggestion or comments? Thanks a lot! I found the questions asked over google the web, but did not find solution.
I use single machine mode for Hadoop on Mac.
hive> CREATE TABLE pokes (foo INT, bar STRING);
FAILED: Error in metadata: javax.jdo.JDOFatalInternalException: Unexpected exception caught.
NestedThrowables:
java.lang.reflect.InvocationTargetException
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
log file:
Last login: Tue Jun 14 00:27:51 on ttys001
Zhiyong-Xies-MacBook-Pro:~ hadoop$ cat /tmp/*/hive.log
2011-06-14 00:31:54,834 ERROR metastore.HiveMetaStore (HiveMetaStore.java:executeWithRetry(334)) - JDO datastore error. Retrying metastore command after 1000 ms (attempt 1 of 1)
2011-06-14 00:31:56,012 ERROR exec.DDLTask (SessionState.java:printError(374)) - FAILED: Error in metadata: javax.jdo.JDOFatalInternalException: Unexpected exception caught.
NestedThrowables:
java.lang.reflect.InvocationTargetException
org.apache.hadoop.hive.ql.metadata.HiveException: javax.jdo.JDOFatalInternalException: Unexpected exception caught.
NestedThrowables:
java.lang.reflect.InvocationTargetException
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:491)
at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3233)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:221)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:132)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1238)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1050)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:885)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:224)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:358)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:593)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: javax.jdo.JDOFatalInternalException: Unexpected exception caught.
NestedThrowables:
java.lang.reflect.InvocationTargetException
at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1186)
at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:803)
at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:698)
at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:237)
at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:266)
at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:199)
at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:174)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:369)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:321)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:466)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:240)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.(HiveMetaStore.java:203)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:107)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2010)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2020)
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:485)
... 15 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at javax.jdo.JDOHelper$16.run(JDOHelper.java:1958)
at java.security.AccessController.doPrivileged(Native Method)
at javax.jdo.JDOHelper.invoke(JDOHelper.java:1953)
at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1159)
... 32 more
Caused by: java.lang.NullPointerException
at org.datanucleus.plugin.NonManagedPluginRegistry.registerBundle(NonManagedPluginRegistry.java:443)
at org.datanucleus.plugin.NonManagedPluginRegistry.registerBundle(NonManagedPluginRegistry.java:355)
at org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensions(NonManagedPluginRegistry.java:215)
at org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensionPoints(NonManagedPluginRegistry.java:156)
at org.datanucleus.plugin.PluginManager.registerExtensionPoints(PluginManager.java:82)
at org.datanucleus.OMFContext.(OMFContext.java:156)
at org.datanucleus.OMFContext.(OMFContext.java:137)
at org.datanucleus.ObjectManagerFactoryImpl.initialiseOMFContext(ObjectManagerFactoryImpl.java:132)
at org.datanucleus.jdo.JDOPersistenceManagerFactory.initialiseProperties(JDOPersistenceManagerFactory.java:363)
at org.datanucleus.jdo.JDOPersistenceManagerFactory.(JDOPersistenceManagerFactory.java:307)
at org.datanucleus.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:255)
at org.datanucleus.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:182)
... 40 more
2011-06-14 00:31:56,014 ERROR ql.Driver (SessionState.java:printError(374)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
http://getsatisfaction.com/cloudera/topics/hive_error_error_in_metadata_javax_jdo_jdofatalinternalexception
Removing the $HADOOP_HOME/build directory worked like a charm.
Removing the directory $HADOOP_HOME/build (actually just renaming it) also works for hive-0.9.0 and hadoop-1.0.4.