i have a problem with creating an external hadoop table in db2.
I use this create statement:
CREATE EXTERNAL HADOOP TABLE DATA_LAKE.TABLE_TEST (
Id int,
blabla varchar(10)
)
but when i run this i got an error saying:
The name of the object to be created is identical to the existing name
"TABLE_TEST" of type "Table".. SQLCODE=-601,SQLSTATE=42710.
On my file share i don't have any table name with this name and also in db2 under DATA_LAKE schema i don't have any TABLE_TEST table. I also tried to find where is this table, maybe in a catalog , but i didn't find anything.
SELECT * FROM SYSCAT.TABLES
WHERE TABNAME LIKE '%TABLE_TEST%'
Please for any help
Related
I am trying to create iceberg table using trino but ddl is not executing stating that sort_order not find .below sample ddl
Create table <tableschema>.test ( C1 datetime,
C2 double ,
C3 double
)
With ( format='parquet',
Location='S3a:// location of file',
Partitioning=ARRAY['day(C1)'],
Sort_order=ARRAY['C2','C3']
)
DDL not executing when I am defining the sort_order for table . Without sort_order table getting created
Can you please help with this is anything I need to modify in DDL . I am using trino-jdbc380.jar for driver to connect
I have issue with uploading files to my database through an external table:
I can create, read, drop directories, tables by anton_gridushko user, but have no idea why I have troubles when I try to load data to created table.
I used this tutorial but did everything in an SQL Developer worksheet.
Could you give me advice, please, what to do?
My commands sequence is
1.create directory lang_external as 'C:\loader';
2.SQL> grant read,write on directory lang_external to anton_gridushko;
3.CREATE TABLE languages(
language_id INT,
language_name VARCHAR2(30)
)
ORGANIZATION EXTERNAL(
TYPE oracle_loader
DEFAULT DIRECTORY lang_external
ACCESS PARAMETERS
(FIELDS TERMINATED BY ',')
LOCATION ('languages.csv')
);
4. run into issue here:
SELECT
language_id,
language_name
FROM
languages
ORDER BY
language_name;
I'm use oracle 12c with a user u1. With user u1 I create table T1 like this,
create SEQUENCE T1_SEQ START WITH 1;
CREATE table T1(
id number(11) DEFAULT T1_SEQ.nextval PRIMARY KEY ,
name varchar2(255)
);
Now I want to export the schema to others, and use below command to export
expdp u1/password dumpfile=u1.dmp schemas=u1
, but when others use u1.dmp to import the schema with a new user named u2,
impdp u2/password remap_schema=u1:u2 DUMPFILE=u1.DMP TABLE_EXISTS_ACTION=REPLACE
error happend, beacause default value of table T1 column id add U1 as prefix.
U1.T1_SEQ.nextval
and here is the new create table statement
CREATE table T1(
id number(11) DEFAULT U1.T1_SEQ.nextval PRIMARY KEY ,
name varchar2(255)
);
How can I remove the U1's influence while I want to move one schema to another schema.
Could you give me some suggestions? Thanks in advance!.
this is a known and ongoing issue according to Ask Tom.
So You can,
excluding the affected objects from the import
using the sqlfile option to import the affected objects
amending the sqlfile script output to point to the correct objects before running it
Please See relevant topic
It's a really simple test actually. I create a couple external schemas and create an external table in one of the schemas and then querying svv_external_tables shows the table exists in ALL schemas!! What am I missing?
create external schema mytestschema from data catalog
database 'mytestdb'
iam_role 'arn:aws:iam::123456789:role/spectrumrole'
;
create external table mytestdb.mytestschema.newtable (
col1 varchar(200),
col2 varchar(200),
col3 varchar(200)
)
partitioned by (cycle_date varchar(20) )
stored as parquet
location 's3://s3loc';
select * from svv_external_tables;
External schema doesn't hold the table descriptions, it just hold the connection parameters to a database in data catalog. Or put the other way - whatever is in the data catalog database is shown in every external schema that points to it.
I have successfully created and added Dynamic partitions in an Internal table in hive. i.e. by using following steps:
1-created a source table
2-loaded data from local into source table
3- created another table with partitions - partition_table
4- inserted the data to this table from source table resulting in creation of all the partitions dynamically
My question is, how to perform this in external table? I read so many articles on this, but i am confused , that do I have to specify path to the already existing partitions for creating partitions for external table??
example:
Step 1:
create external table1 ( name string, age int, height int)
location 'path/to/dataFile/in/HDFS';
Step 2:
alter table table1 add partition(age)
location 'path/to/already/existing/partition'
I am not sure how to proceed with partitioning in external tables. Can somebody please help by giving step by step description of the same?.
Thanks in advance!
Yes, you have to tell Hive explicitly what is your partition field.
Consider you have a following HDFS directory on which you want to create a external table.
/path/to/dataFile/
Let's say this directory already have data stored(partitioned) department wise as follows:
/path/to/dataFile/dept1
/path/to/dataFile/dept2
/path/to/dataFile/dept3
Each of these directories have bunch of files where each file
contains actual comma separated data for fields say name,age,height.
e.g.
/path/to/dataFile/dept1/file1.txt
/path/to/dataFile/dept1/file2.txt
Now let's create external table on this:
Step 1. Create external table:
CREATE EXTERNAL TABLE testdb.table1(name string, age int, height int)
PARTITIONED BY (dept string)
ROW FORMAT DELIMITED
STORED AS TEXTFILE
LOCATION '/path/to/dataFile/';
Step 2. Add partitions:
ALTER TABLE testdb.table1 ADD PARTITION (dept='dept1') LOCATION '/path/to/dataFile/dept1';
ALTER TABLE testdb.table1 ADD PARTITION (dept='dept2') LOCATION '/path/to/dataFile/dept2';
ALTER TABLE testdb.table1 ADD PARTITION (dept='dept3') LOCATION '/path/to/dataFile/dept3';
Done, run select query once to verify if data loaded successfully.
1. Set below property
set hive.exec.dynamic.partition=true
set hive.exec.dynamic.partition.mode=nonstrict
2. Create External partitioned table
create external table1 ( name string, age int, height int)
location 'path/to/dataFile/in/HDFS';
3. Insert data to partitioned table from source table.
Basically , the process is same. its just that you create external partitioned table and provide HDFS path to table under which it will create and store partition.
Hope this helps.
The proper way to do it.
Create the table and mention it is partitioned.
create external table1 ( name string, age int, height int)
partitioned by (age int)
stored as ****(your format)
location 'path/to/dataFile/in/HDFS';
Now you have to refresh the partitions in the hive metastore.
msck repair table table1
This will take care of loading all your partitions into the hive metastore.
You can use msck repair table at any point during your process to have the metastore updated.
Follow the below steps:
Create a temporary table/Source table
create table source_table(name string,age int,height int) row format delimited by ',';
Use your delimiter as in the file instead of ',';
Load data into the source table
load data local inpath 'path/to/dataFile/in/HDFS';
Create external table with partition
create external table external_dynamic_partitions(name string,height int)
partitioned by (age int)
location 'path/to/dataFile/in/HDFS';
Enable dynamic partition mode to nonstrict
set hive.exec.dynamic.partition.mode=nonstrict
Load data to external table with partitions from source file
insert into table external_dynamic partition(age)
select * from source_table;
That's it.
You can check the partitions information using
show partitions external_dynamic;
You can even check if it is an external table or not using
describe formatted external_dynamic;
External table is a type of table in Hive where the data is not moved to the hive warehouse. That means even if U delete the table, the data still persists and you will always get the latest data, which is not the case with Managed table.