I created a Hive table with table description metadata using this command:
create table sbx_ppppp.comments (
s string comment 'uma string',
i int comment 'um inteiro'
) comment 'uma tabela com comentários';
But it isn't correctly displayed when I double click the table:
The table description also isn't displayed in the table tooltip or in the table list when I double click the database name.
When I run the describe formatted table sbx_ppppp.comments command with the comment is correctly displayed as table property:
col_name |data_type |comment |
----------------------------+------------------------------------------------+---------------------------------------------------------------------------+
# col_name |data_type |comment |
s |string |uma string |
i |int |um inteiro |
| | |
# Detailed Table Information| | |
Database: |sbx_ppppp | |
OwnerType: |USER | |
Owner: |ppppp | |
CreateTime: |Fri Apr 29 18:31:31 BRT 2022 | |
LastAccessTime: |UNKNOWN | |
Retention: |0 | |
Location: |hdfs://BNDOOP03/corporativo/sbx_ppppp/comments | |
Table Type: |MANAGED_TABLE | |
Table Parameters: | | |
|COLUMN_STATS_ACCURATE |{\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"i\":\"true\",\"s\":\"true\"}}|
|bucketing_version |2 |
|comment |uma tabela com comentários |
|numFiles |0 |
|numRows |0 |
|rawDataSize |0 |
|totalSize |0 |
|transactional |true |
|transactional_properties |default |
|transient_lastDdlTime |1651267891 |
| | |
# Storage Information | | |
SerDe Library: |org.apache.hadoop.hive.ql.io.orc.OrcSerde | |
InputFormat: |org.apache.hadoop.hive.ql.io.orc.OrcInputFormat | |
OutputFormat: |org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat| |
Compressed: |No | |
Num Buckets: |-1 | |
Bucket Columns: |[] | |
Sort Columns: |[] | |
Storage Desc Params: | | |
|serialization.format |1 |
In "Table Parameters" you can see the value "uma tabela com comentários" for the "comment" parameter.
I'm using Cloudera ODBC driver version 2.6.11.1011 to connect to Hive. DBeaver is version 22.0.3.202204170718. I don't know if this is a bug in DBeaver or in Cloudera ODBC driver. Maybe I'm not correctly setting the table description.
Related
I have been trying to run this piece of code to drop current day's partition from hive a table and for some reason it does not drop the partition from the hive table. Not sure what's worng.
Table Name : prod_db.products
desc:
+----------------------------+-----------------------+-----------------------+--+
| col_name | data_type | comment |
+----------------------------+-----------------------+-----------------------+--+
| name | string | |
| cost | double | |
| load_date | string | |
| | NULL | NULL |
| # Partition Information | NULL | NULL |
| # col_name | data_type | comment |
| | NULL | NULL |
| load_date | string | |
+----------------------------+-----------------------+-----------------------+--+
## I am using the following code
SET hivevar:current_date=current_date();
ALTER TABLE prod_db.products DROP PARTITION(load_date='${current_date}');
Before and After picture of partitions:
+-----------------------+--+
| partition |
+-----------------------+--+
| load_date=2022-04-07 |
| load_date=2022-04-11 |
| load_date=2022-04-18 |
| load_date=2022-04-25 |
+-----------------------+--+
It runs without any error but doesn't work but won't drop the partition. Table is internal/managed.
I tried different ways mentioned on stack but it is just not working for me.
Help.
You dont need to set a variable. You can directly drop using direct sql.
Alter table prod_db.products
drop partition (load_date= current_date());
I'm moving data from one hive table to another using Spark after applying some transformations. While doing so, I'm getting an exception saying avsc file can't be read (as per my understanding) but I can see the avsc file in HDFS.
Please advise what could be the reason.
I've source hive table defined as:
+----------------------------------------------------+--+
| createtab_stmt |
+----------------------------------------------------+--+
| CREATE TABLE `exchr`( |
| `exchr_sk` bigint, |
| `rec_ctry_cd` string, |
| `cob_dt` date, |
| `ccy_cd` string, |
| `exchr_val` decimal(10,5)) |
| ROW FORMAT SERDE |
| 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' |
| STORED AS INPUTFORMAT |
| 'org.apache.hadoop.mapred.TextInputFormat' |
| OUTPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' |
| LOCATION |
| 'hdfs://xxxx/EXCHR' |
| TBLPROPERTIES ( |
| 'COLUMN_STATS_ACCURATE'='true', |
| 'last_modified_by'='hive', |
| 'last_modified_time'='1500408192', |
| 'numFiles'='0', |
| 'numRows'='3', |
| 'rawDataSize'='73', |
| 'totalSize'='0', |
| 'transient_lastDdlTime'='1501780655') |
+----------------------------------------------------+--+
I've a target hive table defined as:
+----------------------------------------------------+--+
| createtab_stmt |
+----------------------------------------------------+--+
| CREATE EXTERNAL TABLE `fx_rate`( |
| `stamp` string COMMENT '', |
| `curr_code` string COMMENT '', |
| `fx_rate` double COMMENT '') |
| PARTITIONED BY ( |
| `cb_dt` string) |
| ROW FORMAT SERDE |
| 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' |
| STORED AS INPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' |
| OUTPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' |
| LOCATION |
| 'hdfs://xxxx/fx_rate' |
| TBLPROPERTIES ( |
| 'avro.schema.url'='hdfs://xxxx/fx_rate.avsc', |
| 'transient_lastDdlTime'='1502313078') |
+----------------------------------------------------+--+
when i'm trying to insert data into this table using spark with below code, I encounter Encountered exception determining schema. Returning signal schema to indicate problem
Spark code:
val TABLE_2_0 = sqlContext.sql("select * from xxxx.exchr");
val SELECT_1_0 = TABLE_2_0.select(TABLE_2_0.col("*"));
val SELECT_0_0 = SELECT_1_0.select(date_format(SELECT_1_0.col("cob_dt"), "yyyymmdd").as("stamp"), lit(null).as("curr_code"), round(lit(10)/SELECT_1_0.col("exchr_val"),10).as("fx_rate"), date_format(SELECT_1_0.col("cb_dt"), "yyyymmdd").as("cb_dt")).limit(1000);
SELECT_0_0.toDF().write.mode("append").insertInto("xxxx.fx_rate")
Exception trace:
java.lang.NullPointerException
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:381)
at org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFromFS(AvroSerdeUtils.java:154)
at org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:135)
at org.apache.hadoop.hive.serde2.avro.AvroSerDe.determineSchemaOrReturnErrorSchema(AvroSerDe.java:172)
at org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:103)
at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.newSerializer(InsertIntoHiveTable.scala:59)
at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.outputClass$lzycompute(InsertIntoHiveTable.scala:53)
17/08/10 23:46:44 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 1.0 (TID 3, myhost.net, executor 2): org.apache.hadoop.hive.serde2.SerDeException: Encountered exception determining schema. Returning signal schema to indicate problem: null
at org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:523)
at org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializer(TableDesc.java:97)
at org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializer(TableDesc.java:88)
at org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializer(TableDesc.java:81)
at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.org$apache$spark$sql$hive$execution$InsertIntoHiveTable$$writeToFile$1(InsertIntoHiveTable.scala:92)
at org.apache.spark.sql.hive.execution.InsertIntoHiveTable$$anonfun$saveAsHiveFile$3.apply(InsertIntoHiveTable.scala:84)
at org.apache.spark.sql.hive.execution.InsertIntoHiveTable$$anonfun$saveAsHiveFile$3.apply(InsertIntoHiveTable.scala:84)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:229)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
I am working on Oracle 11G.
One of my Materialized view has become UNKNOWN (MY_MAT_VW1). You can check the output of the ALL_MVIEWS below.
OWNER | MVIEW_NAME | CONTAINER_NAME | QUERY | QUERY_LEN | UPDATABLE | UPDATE_LOG | MASTER_ROLLBACK_SEG | MASTER_LINK | REWRITE_ENABLED | REWRITE_CAPABILITY | REFRESH_MODE | REFRESH_METHOD | BUILD_MODE | FAST_REFRESHABLE | LAST_REFRESH_TYPE | LAST_REFRESH_DATE | STALENESS | AFTER_FAST_REFRESH | UNKNOWN_PREBUILT | UNKNOWN_PLSQL_FUNC | UNKNOWN_EXTERNAL_TABLE | UNKNOWN_CONSIDER_FRESH | UNKNOWN_IMPORT | UNKNOWN_TRUSTED_FD | COMPILE_STATE | USE_NO_INDEX | STALE_SINCE | NUM_PCT_TABLES | NUM_FRESH_PCT_REGIONS | NUM_STALE_PCT_REGIONS
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
MY_DB | MY_MAT_VW1 | MY_MAT_VW1 | select.. | 6728 | N | | | | N | GENERAL | DEMAND | COMPLETE | IMMEDIATE | NO | COMPLETE | 14-Nov-16 | UNKNOWN | NA | N | Y | N | N | N | N | VALID | N | 0 | | |
MY_DB | MY_MAT_VW2 | MY_MAT_VW2 | select.. | 7074 | N | | | | N | TEXTMATCH | DEMAND | COMPLETE | IMMEDIATE | NO | COMPLETE | 13-Nov-16 | FRESH | NA | N | N | N | N | N | N | FRESH | N | 0 | 0 | |
The queries for the materialized view contain complex joins between multiple tables, inline views and unions.
As per my understanding (UNKNOWN_PLSQL_FUNC column) I guess there is a PLSQL Function which is causing the staleness to become UNKNOWN. However I am not sure which one.
I tried re-compiling and refreshing it but no luck.
Can anyone provide me some information on how to detect the root cause and make sure it does not become UNKNOWN again.
Also is there any implication of it on the data stored within it?
Below is just a sample I've created to replicate the scenario.
SELECT * FROM ENTITY_T;
ID | ENTITY_TYPE | FIRST_NAME | LAST_NAME | LEGAL_NAME
--------------------------------------------------
1 | INDIVIDUAL | JOHN | LESSEN |
2 | INDIVIDUAL | ROSAN | MEL |
3 | CORP | SIGMA | | SIGMA CORPORATION
--Function to get name base upon type
CREATE OR REPLACE FUNCTION GET_NAME (P_ID IN NUMBER)
RETURN VARCHAR2
DETERMINISTIC
AS
LV_NAME VARCHAR2(200);
BEGIN
SELECT CASE ENTITY_TYPE WHEN 'INDIVIDUAL' THEN FIRST_NAME ||' '|| LAST_NAME
WHEN 'CORP' THEN LEGAL_NAME
ELSE 'NONE'
END INTO LV_NAME
FROM ENTITY_T
WHERE ID=P_ID;
RETURN LV_NAME;
EXCEPTION
WHEN NO_DATA_FOUND THEN
RETURN 'NO ID FOUND';
WHEN OTHERS THEN
RETURN 'OTHER ERROR';
END;
--Materialized view creation
CREATE MATERIALIZED VIEW TEST_MV
AS
SELECT ID,ENTITY_TYPE,GET_NAME(ID) NAME
FROM ENTITY_T;
SELECT MVIEW_NAME,STALENESS,AFTER_FAST_REFRESH,UNKNOWN_PLSQL_FUNC,COMPILE_STATE,STALE_SINCE
FROM ALL_MVIEWS WHERE MVIEW_NAME='TEST_MV';
MVIEW_NAME | STALENESS | AFTER_FAST_REFRESH | UNKNOWN_PLSQL_FUNC | COMPILE_STATE | STALE_SINCE
----------------------------------------------------------------------------------------------
TEST_MV | UNKNOWN | NA | Y | VALID |
The Oracle Issue/Doc ID 757537.1 mentioned by JSapkota states clearly, that this is not a bug, but correct/expected behaviour:
STALENESS of the mview, refering to PL/SQL function is set to UNKOWN
as one cannot determine PL/SQL function changes. Current behaviour is
correct as per the design & code.
I guess using DETERMINISTIC functions instead of the default scope could prevent it.
As per the My Oracle Support this could be a bug(7582462).
As there is no solution to this bug, you have to deal with fact that staleness will show unknown, or not use functions on Materialized View definition.
Reference:DBA_MVIEWS Shows STALENESS Value of UNKNOWN After Refresh (Doc ID 757537.1)
Database: Oracle 12c
I want to take single partition, or a set of partitions, disconnect it from a Table, or set of tables on DB1 and move it to another table on another database. I would like to avoid doing DML to do this for performance reasons (It needs to be fast).
Each Partition will contain between three and four hundred million records.
Each Partition will be broken up into approximately 300 Sub-Partitions.
The task will need to be automated.
Some thoughts I had:
Somehow put each partition in it's own datafile upon creation, then detaching from the source and attaching it to the destination?
Extract the whole partition (not record-by-record)
Any other non-DML Solutions are also welcom
Example (Move Part#33 from both to DB#2, preferably with a single, operation):
__________________ __________________
| DB#1 | | DB#2 |
|------------------| |------------------|
|Table1 | |Table1 |
| Part#1 | | Part#1 |
| ... | | ... |
| Part#33 | ----> | Part#32 |
| Subpart#1 | | |
| ... | | |
| Subpart#300 | | |
|------------------| |------------------|
|Table2 | |Table2 |
| Part#1 | | Part#1 |
| ... | | ... |
| Part#33 | ----> | Part#32 |
| Subpart#1 | | |
| ... | | |
| Subpart#300 | | |
|__________________| |__________________|
Please read the document below with all the examples of exchanging partitions of table.
https://oracle-base.com/articles/misc/partitioning-an-existing-table-using-exchange-partition
Currently am working with magento project..
in which i have stucked on the point ..
that is when admin add any subcategory/category
by
Manage category -> custome design
here its gives two option like
Active from and Active to..
can anyone, who know about magento database, tell me where or in which table this two value store
Thanks for any suggestion or help!
I've listed the attributes for catalog categories below. Since categories are an EAV type, you'll need to look in a particular subtable to get your values. In this case, custom_design_from and custom_design_to are datetime values, and the name of your entity is catalog_category_entity, so the table you want is catalog_category_entity_datetime.
Next problem you'll find is getting the right attribute ID. Since they're liable to change, here's the SQL query to run in order to grab them:
select attribute_id, attribute_code from eav_attribute where entity_type_id = 3 and attribute_code in ('custom_design_from', 'custom_design_to');
I get 52 and 53, but YMWV. Hope that helps!
Thanks,
Joe
+----------------------+--------------+
| attribute_code | backend_type |
+----------------------+--------------+
| name | varchar |
| is_active | int |
| url_key | varchar |
| description | text |
| image | varchar |
| meta_title | varchar |
| meta_keywords | text |
| meta_description | text |
| display_mode | varchar |
| landing_page | int |
| is_anchor | int |
| path | static |
| position | static |
| all_children | text |
| path_in_store | text |
| children | text |
| url_path | varchar |
| custom_design | varchar |
| custom_design_apply | int |
| custom_design_from | datetime |
| custom_design_to | datetime |
| page_layout | varchar |
| custom_layout_update | text |
| level | static |
| children_count | static |
| available_sort_by | text |
| default_sort_by | varchar |
| include_in_menu | int |
+----------------------+--------------+
Active from is an attribute whose attribute_code is custom_design_from(attribute_id 57) and Active To is an attribute whose attribute_code(attribute_id 58) is custom_design_to.
This both attributes value are stored in database table `catalog_category_entity_datetime`.
Check above table with row like value of entity_id is your category id, attribute_id is 57 and active from value is store in value field of table same active to value is stored in value field with entity_id is your category id, attribute_id is 58.