Maybe when you see the "Error in semantic" in the title, you consider it as syntax error?
Of course not, I will show you what happened.
hive> use android;
OK
Time taken: 0.223 seconds
hive> desc tb_user_basics;
OK
col_datetime string
col_is_day_new string
col_is_hour_new string
col_ch string
...
p_date string
p_hourmin string
Time taken: 0.189 seconds
hive> select count(distinct col_udid) from android.tb_user_basics where p_date>='20121001' and p_date<='20121231';
FAILED: Error in semantic analysis: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
hive>
>
> select count(distinct col_udid) from android.tb_user_basics where p_date>='20121001' and p_date<='20121231';
FAILED: Error in semantic analysis: Unable to fetch table tb_user_basics
I'm very sure the table does exist in the database android. After the first statement failed, it appears that the table is missing.(Even I add the db prefix in the table name)
I'm wonderring whether it's because of the volumn of data is very big, maybe you have noticed that the time range is [20121001, 20121231].
I run the command before many times, always raise this error. But if I change the contition to "p_date='20121001'", the statement can run normally. (since the volumn is smaller? )
I'm expecting your answers, Thanks.
Probably you are in strict mode. One of strict mode feature is that partitions has to be specified, so this is why queries with "p_date='20121001'" in where cause are working.
Please try the non-strict mode:
set hive.mapred.mode=nonstrict;
Related
I am very new in iseries/DB2.
We use V7R3. We have table that is generated every day by RPG program as physical file. For accessing the table data from java we use jt400.jar jdbc driver.
Most of the table queries work fine but some complex queries that are using "DENSE_RANK() OVER(ORDER BY" and "ROW_NUMBER() OVER(PARTITION BY" time to time hanging and causing CPU 100%. only killing the job on AS400 side is resolving the issue.
in the AS400 log I see:
Job 969954/QUSER/QZDASOINIT started on 02/21/19 at 09:36:46 in subsystem
QUSRWRK in QSYS. Job entered system on 02/21/19 at 09:36:46.
User USERXX from client X.X.X.X connected to server.
Use of function TIMESTAMP_FORMAT in QSYS2 not valid.
Use of function TIMESTAMP_FORMAT in QSYS2 not valid.
Data mapping error on member TABLE_NAME.
Data mapping error on member TABLE_NAME.
Data mapping error on member TABLE_NAME.
Data mapping error on member TABLE_NAME.
Value in date, time, or timestamp string not valid.
It looks similar as problem described in the Why am I getting a "[SQL0802] Data conversion of data mapping error" exception?
and probably the problem is related to invalid data stored to DATE type columns.
Looking to the DATE columns I see that some records displayed as <null> in SQuirrel SQL Client. Interesting that here are 2 different <null>'s returned by distinct query.
If I run
select distinct VARCHAR_FORMAT(DATE_COLUMN, 'YYYY/MM/DD') from TABLE_NAME
I get
0001/01/01
and
9999/12/31
for these <null>'s rows.
if I run Select * from TABLE where DATE_COLUMN is null I don't get any results. So I am not sure what kind of <null>'s is that.
Not sure if these records can cause an issue.
UPD: when I run
Select * from TABLE
I see errors in JDBC client log:
Warning: [SQL0181] Value in date, time, or timestamp string not valid.
SQLState: 01534
ErrorCode: 181
Warning: [SQL0181] Value in date, time, or timestamp string not valid.
SQLState: 01534
ErrorCode: 181Warning: [SQL0181] Value in date, time, or timestamp string not valid.
SQLState: 01534
ErrorCode: 181
Warning: [SQL0181] Value in date, time, or timestamp string not valid.
SQLState: 01534
ErrorCode: 181
Query 1 of 1, Rows read: 100, Elapsed time (seconds) - Total: 0.252, SQL query: 0.005, Reading results: 0.247
based on https://www.consolut.com/en/s/sap-ides-access/d/s/p/40/doc/XH-SQL0181/ it should be incorrect date somewhere in the table
The question is there any way to find and filter records that have "invalid" data (causing above exception in the log) from SQL side?
The following query is giving me the error:
Execute error: Error while processing statement: FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.mr.MapRedTask
Does anyone know why or how to resolve this issue?
proc sql;
connect to hadoop(server='xxx' port=10000 schema=xxx SUBPROTOCOL=hive2 sql_functions=all);
execute(
create table a as
select
*,
lag(claim_flg,1) over (order by ptnt_id,month) as lag1
from b
) by hadoop;
disconnect from hadoop;
quit;
It appears to be a limitation issue in HIVE database:
Hive Limit of 127 Expressions per Table
Due to a limitation in the Hive database, tables can contain a maximum of 127 expressions. When the 128th expression is read, the directive fails and the SAS log receives a message similar to the following:
ERROR: java.sql.SQLException: Error while processing statement: FAILED:
Execution Error, return
code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
ERROR: Unable to execute Hadoop query.
ERROR: Execute error.
SQL_IP_TRACE: None of the SQL was directly passed to the DBMS.
The Hive limitation applies anytime a table is read as part of a directive. For SAS Data Loader, the error can occur in aggregations, profiles, when viewing results, and when viewing sample data.
Source: http://support.sas.com/documentation/cdl/en/dmddug/67908/HTML/default/viewer.htm#p1fl149uastoudn1v7r2u5ff8aft.htm
Let's import a simple table in Hive:
hive> CREATE EXTERNAL TABLE tweets (id BIGINT, id_str STRING, user STRUCT<id:BIGINT, screen_name:STRING>)
ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.JsonSerde'
LOCATION '/projets/tweets';
OK
Time taken: 2.253 seconds
hive> describe tweets.user;
OK
id bigint from deserializer
screen_name string from deserializer
Time taken: 1.151 seconds, Fetched: 2 row(s)
I cannot figure out where is the syntax error here:
hive> select user.id from tweets limit 5;
OK
Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating user.id
Time taken: 0.699 seconds
I am using the version 1.2.1 of Hive.
I finally found the answer. It seems it is a problem with the JAR used to serialize/deserialize the JSON. The default one (Apache) is not able to perform a good job on the data I have.
I tried all these typical JAR (in parenthesis, the class for 'ROW FORMAT SERDE'):
hive-json-serde-0.2.jar (org.apache.hadoop.hive.contrib.serde2.JsonSerde)
hive-serdes-1.0-SNAPSHOT.jar (com.cloudera.hive.serde.JSONSerDe)
hive-serde-1.2.1.jar (org.apache.hadoop.hive.serde2.DelimitedJSONSerDe)
hive-serde-1.2.1.jar (org.apache.hadoop.hive.serde2.avro.AvroSerDe)
All of them gave me different kinds of errors. I list them there so the next guy can Google them:
Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating user.id
java.lang.ClassCastException: org.json.JSONObject cannot be cast to [Ljava.lang.Object;
Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.Long
Failed with exception
java.io.IOException:org.apache.hadoop.hive.serde2.SerDeException: DelimitedJSONSerDe cannot deserialize.
Failed with exception java.io.IOException:org.apache.hadoop.hive.serde2.avro.AvroSerdeException: Expecting a AvroGenericRecordWritable
Finally, the working JAR is json-serde-1.3-jar-with-dependencies.jar which can be found here. This one is working with 'STRUCT' and can even ignore some malformed JSON. I have also to use for the creation of the table this class:
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
WITH SERDEPROPERTIES ("ignore.malformed.json" = "true")
LOCATION ...
If needed, it is possible to recompile it from here or here. I tried the first repository and it is compiling fine for me, after adding the necessary libs. The repository has also been updated recently.
When I'm trying to drop table then I'm getting error
SQL Error: ORA-00604: error occurred at recursive SQL level 2
ORA-01422: exact fetch returns more than requested number of rows
00604. 00000 - "error occurred at recursive SQL level %s"
*Cause: An error occurred while processing a recursive SQL statement
(a statement applying to internal dictionary tables).
*Action: If the situation described in the next error on the stack
can be corrected, do so; otherwise contact Oracle Support.
One possible explanation is a database trigger that fires for each DROP TABLE statement. To find the trigger, query the _TRIGGERS dictionary views:
select * from all_triggers
where trigger_type in ('AFTER EVENT', 'BEFORE EVENT')
disable any suspicious trigger with
alter trigger <trigger_name> disable;
and try re-running your DROP TABLE statement
I noticed following line from error.
exact fetch returns more than requested number of rows
That means Oracle was expecting one row but It was getting multiple rows. And, only dual table has that characteristic, which returns only one row.
Later I recall, I have done few changes in dual table and when I executed dual table. Then found multiple rows.
So, I truncated dual table and inserted only row which X value. And, everything working fine.
I know the post is old and solved, but maybe someone is facing or will face my situation, so I want to leave the aquired knowledge here, after deal with the error for a week. I was facing the error: "ORA-00604: error occurred at recursive SQL level 1" , but with the internal error: " ORA-06502: error: character string buffer too smal numeric or value", this happened only when I try to logon the database, and using an specific driver, trying to connect from an Visual Studio C# application, the weirdest thing on that moment was that I connect from SQLDeveloper or TOAD and everything worked fine.
Later I discovered that my machine name had this format "guillermo-aX474b5", then I proceed to rename it like this "guillermo" without the "-" and the other stuff, and it worked!! Looks like in some drivers and situations, ORACLE Database does'nt like the "-" in the LogOn connection.
Hope it helps!
When I enter show tables; all tables were showed on the screen as return, but when I query any of the table, it saidTable not found... Would anyone please advise why would this happen suddenly? Suddenly means I was browsing a table in the last min and it turned out like this afterwards...
hive> show tables;
OK
mahoutpoc
prod
rc_agg2
rc_uum
uum
Time taken: 1.541 seconds
hive> select * from rc_uum limit 10;
FAILED: SemanticException [Error 10001]: Line 1:14 Table not found 'rc_uum'
This is because 'rc_umm' != 'rc_uum'.