All Hive functions fail - hadoop

I honestly don't know what is going on. Everything fails except basically SHOW DATABASES.
Nothing on this page (below) even works. Everything gives me a NoViableAltException. This is both in Hive and Beeline CLI.
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-DateFunctions
My opinion is I shouldn't have to be connected to a DB to execute these commands which didn't initially work. But, I decided to try connecting to the default DB and still all the same failures. Created a new DB and same failures. Sources I've used for doc info...
http://www.folkstalk.com/2011/11/date-functions-in-hive.html
http://www.folkstalk.com/2011/11/difference-between-normal-tables-and.html
http://hortonworks.com/wp-content/uploads/2016/05/Hortonworks.CheatSheet.SQLtoHive.pdf
Add minutes to datetime in Hive
Nothing works. All NoViableAltException exception. I started the cloudera manger and all things check healthy except Impala. I'm using the CDH5 quickstart docker image. Tried both hive and beeline cli. Any help would be appreciated.
EDIT: Here is an example from another StackOverflow question that fails though different exception.
0: jdbc:hive2://localhost:10000/default> from_unixtime(unix_timestamp());
Error: Error while compiling statement: FAILED: ParseException line 1:0 cannot recognize input near 'from_unixtime' '(' 'unix_timestamp' (state=42000,code=40000)
0: jdbc:hive2://localhost:10000/default>
This one is a different exception but I can't get anything other than show databases/tables/etc to work.
Here is another...
hive> DATEDIFF('2000-03-01', '2000-01-10');
NoViableAltException(26#[])
at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1024)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:201)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:423)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:311)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1194)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1120)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1108)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:218)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:170)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:381)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:773)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:691)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
FAILED: ParseException line 1:0 cannot recognize input near 'DATEDIFF' '(' ''2000-03-01''
Literally nothing I see on the net that should work works.

Just add SELECT.
SELECT from_unixtime(unix_timestamp());
SELECT DATEDIFF('2000-03-01', '2000-01-10');

Related

Corrupt theme file unrecognized magic number: 0 [CodenameOne]

Why am getting below error in codename one while running or opening theme file. please provide solution on same.
java.io.IOException: Corrupt theme file unrecognized magic number: 0
com.svastikagames.gkquiz.MyApplication.start() Error java.lang.NullPointerException
at com.codename1.ui.util.Resources.openFileImpl(Resources.java:332)
at com.codename1.ui.util.Resources.openFile(Resources.java:261)
at com.codename1.ui.util.Resources.<init>(Resources.java:181)
at com.codename1.ui.util.Resources.open(Resources.java:729)
at com.codename1.ui.util.Resources.openLayered(Resources.java:694)
at com.codename1.ui.util.Resources.openLayered(Resources.java:658)
at generated.StateMachineBase.startApp(StateMachineBase.java:59)
at generated.StateMachineBase.<init>(StateMachineBase.java:31)
at generated.StateMachineBase.<init>(StateMachineBase.java:124)
at userclasses.StateMachine.<init>(StateMachine.java:236)
at com.svastikagames.gkquiz.MyApplication.start(MyApplication.java:25)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.codename1.impl.javase.Executor$1$1.run(Executor.java:100)
at com.codename1.ui.Display.processSerialCalls(Display.java:1149)
at com.codename1.ui.Display.mainEDTLoop(Display.java:966)
at com.codename1.ui.RunnableWrapper.run(RunnableWrapper.java:120)
at com.codename1.impl.CodenameOneThread.run(CodenameOneThread.java:176)
Usually this error means the file is 0 length and got corrupted at some point due to an unclear reason.
If the file is of 0 length there is nothing we can do to help recover its content unless you are using the XML team mode.
If you are using that then just launch the designer tool from command line and edit the theme it should actually work since it loads the resources from the XML's under the res directory and not the res file itself. If it prints out errors during save these might help you narrow down the problem and you can also refer these errors to us.
In order to launch the designer from command line use this on Unix/Linux/Mac:
java -jar ~/.codenameone/designer_1.jar
And on Windows:
java -jar MY_USER_DIRECTORY\.codenameone\designer_1.jar

WebHCAT Error in getting Hive Table Metadata. Command was terminated due to timeout(10000ms). See templeton.exec.timeout property","exitcode":143

If I issue this webhcat REST call in my cloudera 5.4.1 environment
curl -s 'http://mywebhcat:50111/templeton/v1/ddl/database/default/table/person?
user.name=admin&format=extended'; echo; echo;
everything works fine and I see the metadata for the table Person.
But there is another table called foo_bar if I change the REST call above to
curl -s 'http://mywebhcat:50111/templeton/v1/ddl/database/default/table/foo_bar?
user.name=admin&format=extended'; echo; echo;
Then I get an error
{"statement":"use default; show table extended like
foo_bar;","error":"unable to show table:
foo_bar","exec":{"stdout":"","stderr":"which: no /opt/cloudera/parcels/
CDH-5.4.1-1.cdh5.4.1.p0.6/lib/hadoop/bin/hadoop in ((null))\ndirname: missing operand\nTry
`dirname --help' for more information.\nlog4j:ERROR setFile(null,true) call failed.\njava.io
.FileNotFoundException: /opt/cloudera/parcels/CDH-5.4.1-1.cdh5.4.1.p0.6/lib/hive/logs/hcat.
log (No such file or directory)\n\tat java.io.FileOutputStream.open0(Native Method)\n\tat
java.io.FileOutputStream.open(FileOutputStream.java:270)\n\tat java.io.FileOutputStream.<
init>(FileOutputStream.java:213)\n\tat java.io.FileOutputStream.<init>(FileOutputStream.
java:133)\n\tat org.apache.log4j.FileAppender.setFile(FileAppender.java:294)\n\tat org.
apache.log4j.FileAppender.activateOptions(FileAppender.java:165)\n\tat org.apache.log4j.
DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)\n\tat org.apache
.log4j.config.PropertySetter.activate(PropertySetter.java:307)\n\tat org.apache.log4j.config
.PropertySetter.setProperties(PropertySetter.java:172)\n\tat org.apache.log4j.config.
PropertySetter.setProperties(PropertySetter.java:104)\n\tat org.apache.log4j.
PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)\n\tat org.apache.log4j.
PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)\n\tat org.apache.log4j.
PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)\n\tat org.apache.
log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)\n\tat org.apache.log4j
.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)\n\tat org.apache.log4j.
PropertyConfigurator.configure(PropertyConfigurator.java:415)\n\tat org.apache.hadoop.hive.
common.LogUtils.initHiveLog4jDefault(LogUtils.java:127)\n\tat org.apache.hadoop.hive.common.
LogUtils.initHiveLog4jCommon(LogUtils.java:77)\n\tat org.apache.hadoop.hive.common.LogUtils.
initHiveLog4j(LogUtils.java:58)\n\tat org.apache.hive.hcatalog.cli.HCatCli.main(HCatCli.
java:65)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect
.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.
DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.
reflect.Method.invoke(Method.java:497)\n\tat org.apache.hadoop.util.RunJar.run(RunJar.
java:221)\n\tat org.apache.hadoop.util.RunJar.main(RunJar.java:136)\nlog4j:ERROR Either
File or DatePattern options are not set for appender [DRFA].\nOK\nTime taken: 1.035
seconds\n Command was terminated due to timeout(10000ms). See templeton.exec.timeout
property","exitcode":143}}
I don't know why it complains about a missing log directory only for foo_bar table but successfully returns the metadata regarding Person.
BTW, I can go into the hive console and do a select count(*) query on both Person and foo_bar.
EDIT::
Upon reading the error message again, it seems the core problem is
Command was terminated due to timeout(10000ms). See templeton.exec.timeout property","exitcode":143
But cloudera manager is not aware of this property "templeton.exec.timeout" ... what do I do... I don't want to edit files manually as there are many many nodes in the cluster.
Edit2::
I went inside each hadoop node and did
sudo vi /opt/cloudera/parcels/CDH-5.4.1-1.cdh5.4.1.p0.6/etc/hive-webhcat/conf.dist/webhcat-default.xml
I found the timeout value and increased it to 1000000. I did this on each one and then I restarted Hive and WebHCat server using cloudera manager. but I got exactly the same error message.

Hive: select * from table fails for tables having more than 5000 rows via HiveServer2

I have a table sdh in Hive which has 100000 rows.
When I execute the command
select * from sdh on CLI, all rows are displayed
But the same command just hangs when I run it via HiveServer2 on beeline
All other tables which have 1000 odd rows work fine via CLI or beeline.
Anyone else face similar issue?
I got error from logs
org.apache.thrift.TApplicationException: Internal error processing FetchResults
at org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71)
at org.apache.hive.service.cli.thrift.TCLIService$Client.recv_FetchResults(TCLIService.java:489)
at org.apache.hive.service.cli.thrift.TCLIService$Client.FetchResults(TCLIService.java:476)
at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:285)
at org.apache.hive.beeline.BufferedRows.<init>(BufferedRows.java:42)
at org.apache.hive.beeline.BeeLine.print(BeeLine.java:1541)
at org.apache.hive.beeline.Commands.execute(Commands.java:741)
at org.apache.hive.beeline.Commands.sql(Commands.java:657)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:763)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:630)
at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:363)
at org.apache.hive.beeline.BeeLine.main(BeeLine.java:346)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
Error: Error retrieving next row (state=,code=0)
Use the option "--incremental=false" with beeline. Otherwise, it tries to read the entire results into memory at same time, and runs out of memory if the results are too large for available memory.
More discussion on making this option the default is here - https://issues.apache.org/jira/browse/HIVE-7224
Please use the option --incremental=true and place it before the -e option in beeline. For me it worked that way.

Oracle Data Integrator (ODI - v11.1.1.3) "unable to load language: beanshell" Error

Following an install of Eclipse 3.7.2 on my Ubuntu 12.04 development machine, I have been unable to execute any ODI packages/interfaces/procedures. On execution (for both simulated and actual runs), an error is thrown (java trace below). I am not sure if it's anything to do with the Eclipse install, but it seems likely. Does anyone have an idea how to fix this?
Also, when launching ODI from the terminal using 'bash odi', the following error is displayed in the terminal:
2013-08-15 14:43:46.162 ERROR Error during RuntimeClassLoader initialization. ODI will start without RuntimeClassLoader
Error output:
oracle.odi.core.exception.OdiRuntimeException: Error during Code Interpretor creation
at com.sunopsis.dwg.codeinterpretor.SnpCodeInterpretor.getInstance(SnpCodeInterpretor.java:209)
at com.sunopsis.dwg.codeinterpretor.SnpGeneratorSQLCIT.<init>(SnpGeneratorSQLCIT.java:300)
at com.sunopsis.graphical.dialog.SnpsDialogExecution.doPackageExecuter(SnpsDialogExecution.java:907)
at oracle.odi.ui.action.SnpsPopupActionExecuteHandler.actionPerformed(SnpsPopupActionExecuteHandler.java:68)
at oracle.odi.ui.SnpsActionControler.handleEvent(SnpsActionControler.java:75)
at oracle.ide.controller.IdeAction.performAction(IdeAction.java:529)
at oracle.ide.controller.IdeAction.actionPerformedImpl(IdeAction.java:884)
at oracle.ide.controller.IdeAction.actionPerformed(IdeAction.java:501)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1995)
at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2318)
at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:387)
at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:242)
at javax.swing.AbstractButton.doClick(AbstractButton.java:357)
at javax.swing.plaf.basic.BasicMenuItemUI.doClick(BasicMenuItemUI.java:809)
at javax.swing.plaf.basic.BasicMenuItemUI$Handler.mouseReleased(BasicMenuItemUI.java:850)
at java.awt.Component.processMouseEvent(Component.java:6297)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3275)
at java.awt.Component.processEvent(Component.java:6062)
at java.awt.Container.processEvent(Container.java:2039)
at java.awt.Component.dispatchEventImpl(Component.java:4660)
at java.awt.Container.dispatchEventImpl(Container.java:2097)
at java.awt.Component.dispatchEvent(Component.java:4488)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4575)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4236)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4166)
at java.awt.Container.dispatchEventImpl(Container.java:2083)
at java.awt.Window.dispatchEventImpl(Window.java:2489)
at java.awt.Component.dispatchEvent(Component.java:4488)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:674)
at java.awt.EventQueue.access$400(EventQueue.java:81)
at java.awt.EventQueue$2.run(EventQueue.java:633)
at java.awt.EventQueue$2.run(EventQueue.java:631)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)
at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:98)
at java.awt.EventQueue$3.run(EventQueue.java:647)
at java.awt.EventQueue$3.run(EventQueue.java:645)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:644)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:269)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:184)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:174)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:169)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:161)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)
Caused by: org.apache.bsf.BSFException: unable to load language: beanshell
at org.apache.bsf.BSFManager.loadScriptingEngine(BSFManager.java:718)
at com.sunopsis.dwg.codeinterpretor.SnpCodeInterpretor.loadEngine(SnpCodeInterpretor.java:85)
at com.sunopsis.dwg.codeinterpretor.SnpCodeInterpretor.<init>(SnpCodeInterpretor.java:75)
at com.sunopsis.dwg.codeinterpretor.SnpCodeInterpretor.getInstance(SnpCodeInterpretor.java:184)
... 45 more
After digging around for about a day on this issue, I brazenly tried running ODI as the root user on the off chance that this was a permissions issue. I started ODI from the command line (using 'bash odi') for greater verbosity, and it loaded without the error mentioned above. Something gave me the impression that this wasn't a permissions issue, but one related to the user settings.
To rectify the issue, I removed my user's odi settings folder (renaming it, for safety):
mv ~/.odi ~/.backup_odi
Then I started ODI from the terminal under my own user (i.e. not root) - there were no errors! None of my connections were available in the new settings folder though. This I fixed by closing ODI and entering the following:
cp ~/.backup_odi/oracledi/snps_login_work.xml ~/.odi/oracledi/
If anybody else encounters this issue, I hope you find this post quicker than it took me to fix it!
org.apache.bsf.BSFException: unable to load language: beanshell
The exception was thrown because bsh-2.Ob4.jar was not in the classpath and it is a dependent jar of bsf.jar

hive failed execution error return code 2 from org.apache.hadoop.hive.ql.exec.mapredtask

I have one query. It is executing fine on Hive CLI and returning the result. But when I am executing it with the help of Hive JDBC, I am getting an error below:
java.sql.SQLException: Query returned non-zero code: 9, cause: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
at org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:192)
What is the problem? Also I am starting the Hive Thrift Server through Shell Script. (I have written a shell script which has command to start Hive Thrift Server) Later I decided to start Hive thrift Server manually by typing command as:
hadoop#ubuntu:~/hive-0.7.1$ bin/hive --service hiveserver
Starting Hive Thrift Server
org.apache.thrift.transport.TTransportException: Could not create ServerSocket on address 0.0.0.0/0.0.0.0:10000.
at org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:99)
at org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:80)
at org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:73)
at org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:384)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
hadoop#ubuntu:~/hive-0.7.1$
Please help me out from this.
Thanks
For this error :
java.sql.SQLException: Query returned non-zero code: 9, cause: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
at org.apache.hadoop.hive.jdbc.HiveStatement.executeQuer
Go to this link :
http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/UsingEMR_Hive.html
and add
**hadoop-0.20-core.jar
hive/lib/hive-exec-0.7.1.jar
hive/lib/hive-jdbc-0.7.1.jar
hive/lib/hive-metastore-0.7.1.jar
hive/lib/hive-service-0.7.1.jar
hive/lib/libfb303.jar
lib/commons-logging-1.0.4.jar
slf4j-api-1.6.1.jar
slf4j-log4j12-1.6.1.jar**
to the class path of your project , add this jars from the lib of hadoop and hive, and try the code. and also add the path of hadoop, hive, and hbase(if your are using) lib folder path to the project class path, like you have added the jars.
and for the second error you got
type
**netstat -nl | grep 10000**
if it shows something means hive server is already running. the second error comes only when the port you are specifying is already acquired by some other process, by default server port is 10000 so very with the above netstat command which i said.
Note : suppose you have connected using code exit from ... bin/hive of if you are connected through bin/hive > then code will not connect because i think (not sure) only one client can connect to the hive server.
do above steps hopefully will solve your problem.
NOTE : exit from cli when you are going to execute the code, and dont start cli while code is being executing.
Might be some issue with permission, just try some query like "SELECT * FROM " which won't start MR jobs.
Try to paste these property before the codes.
SET hive.exec.dynamic.partition = true;
SET hive.exec.dynamic.partition.mode = nonstrict;
set hive.auto.convert.join = false;
set hive.exec.max.dynamic.partitions=100000;
set hive.exec.max.dynamic.partitions.pernode=10000;

Resources