I was migrating the data from Oracle DB to AWS S3 using DMS, In the Oracle DB in one of the table the datatype for the column is Long, but the dms while reading and transfering to S3 getting the below message in the Logs, replacing the actual column and table definition names.
Column 'sample_column' was removed from table definition 'sample_table: the column data type is LOB and the table has no primary key or unique index.
But i verified that the source datatype for the sample_column was Long.
How do i resolve this issue ?
From the documentation: https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html
Oracle data type: LONG
AWS DMS data type: CLOB
The LONG data type isn't supported in batch-optimized apply mode
(TurboStream CDC mode). To use this data type with AWS DMS, you must
enable the use of LOBs for a specific task. During CDC, AWS DMS
supports LOB data types only in tables that have a primary key.
So the type conversion appears to be expected, and the real issue here is that you have no primary key on the table.
Related
we are in the process of migrating Oracle tables to redshift tables. We found that few tables are having CLOB data type. In redshift we converted CLOB to Varchar(65535) type. While doing copy command , we are getting
The length of the data column investigation_process is longer than the length defined in the table. Table: 65000, Data: 90123.
Which data type we need to use? Please share your suggestion.
Redshift isn't designed to store CLOB (or BLOB) data. Most databases that do store the CLOB separately from the table contents to not burden all queries with the excess data. A CLOB reference is stored in the table contents and a replacement of CLOB for reference is performed at result generation.
CLOBs should be stored in S3 and references to the appropriate CLOB (S3 key) stored in the Redshift table. The issue is that there isn't a prepackaged tool for doing the CLOB for reference replacement with Redshift AFAIK. Your solution will need some retooling to perform this replacement actions for all data users. It's doable, it's just going to take a data layer that performs the needed replacement.
I am using Oracle SQL Developer 18.3 but when I want to edit(or insert) a column with RAW datatype it shows the field as read only and does not allow to edit.
As you may know Oracle SQL Developer shows RAW datatype as hex string despite BLOB datatype that it does not show the value but you can download and upload the BLOB data.
I know that I can update(or insert) the RAW data as hex string like this :
CREATE TABLE t1(the_id NUMBER PRIMARY KEY, raw_col RAW(2000));
INSERT INTO t1(the_id, raw_col) VALUES(1, '1a234c');
But I want do it by Oracle SQL Developer GUI.
Sorry, we do not have a 'raw' editor like we have for BLOBs, so it's up to using SQL.
If you want a reason for that omission, it's partly due to the fact that RAW is not a commonly used data type in Oracle Database.
Related: if you're talking about LONG RAW
We (Oracle) recommend you stop using it, and instead convert them to BLOBs.
The LONG RAW datatype is provided for backward compatibility with
existing applications. For new applications, use the BLOB and BFILE
datatypes for large amounts of binary data. Oracle also recommends
that you convert existing LONG RAW columns to LOB columns. LOB columns
are subject to far fewer restrictions than LONG columns. Further, LOB
functionality is enhanced in every release, whereas LONG RAW
functionality has been static for several releases.
The following HQL works to create a Hive table in HDInsight which I can successfully query. But, I have several questions about WHY it works:
My data rows are, in fact, terminated by carriage return line feed, so why does 'COLLECTION ITEMS TERMINATED BY \002' work? And what is \002 anyway? And no location for the blob is specified so, again, why does this work?
All attempts at creating the same table and specifying "CREATE EXTERNAL TABLE...LOCATION '/user/hive/warehouse/salesorderdetail'" have failed. The table is created but no data is returned. Leave off "external" and don't specify any location and suddenly it works. Wtf?
CREATE TABLE IF NOT EXISTS default.salesorderdetail(
SalesOrderID int,
ProductID int,
OrderQty int,
LineTotal decimal
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
COLLECTION ITEMS TERMINATED BY '\002'
MAP KEYS TERMINATED BY '\003'
STORED AS TEXTFILE
Any insights are greatly appreciated.
UPDATE:Thanks for the help so far. Here's the exact syntax I'm using to attempt external table creation. (I've only changed the storage account name.) I don't see what I'm doing wrong.
drop table default.salesorderdetailx;
CREATE EXTERNAL TABLE default.salesorderdetailx(SalesOrderID int,
ProductID int,
OrderQty int,
LineTotal decimal)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
COLLECTION ITEMS TERMINATED BY '\002'
MAP KEYS TERMINATED BY '\003'
STORED AS TEXTFILE
LOCATION 'wasb://mycn-1#my.blob.core.windows.net/mycn-1/hive/warehouse/salesorderdetailx'
When you create your cluster in HDInsight, you have to specify underlying blob storage. It assumes that you are referencing that blob storage. You don't need to specific a location because your query is creating an internal table (see answer #2 below) which is created at a default location. External tables need to specify a location in Azure blob storage (outside of the cluster) so that the data in the table is not deleted when the cluster is dropped. See the Hive DDL for more information.
By default, tables are created as internal, and you have to specify the "external" to make them external tables.
Use EXTERNAL tables when:
Data is used outside Hive
You need data to be updateable in real time
Data is needed when you drop the cluster or the table
Hive should not own data and control settings, directories, etc.
Use INTERNAL tables when:
You want Hive to manage the data and storage
Short term usage (like a temp table)
Creating table based on existing table (AS SELECT)
Does the container "user/hive/warehouse/salesorderdetail" exist in your blob storage? That might explain why it is failing for your external table query.
I m accessing remote database through DBLINK, but when trying to fire select query to fetch image stored in BLOB datatype it gives me following error:
ORA-22992: cannot use LOB locators selected from remote tables
Thanks in Advance.
You didn't provide the statement you are trying to execute, but I guess you are trying to do something which simply isn't possible in Oracle 10g.
The docs at http://docs.oracle.com/cd/B19306_01/server.102/b14200/sql_elements001.htm#sthref161 state:
Oracle Database has limited support for remote LOBs. Remote LOBs are
supported in three ways.
Create table as select or insert as select.
Functions on remote LOBs returning scalars. SQL and PL/SQL functions having a LOB parameter and returning a scalar datatype are
supported. Other SQL functions and DBMS_LOB APIs are not supported for
use with remote LOB columns.
Data Interface for remote LOBs. You can insert a character or binary buffer into a remote CLOB or BLOB, and select a remote CLOB or
BLOB into a character or binary buffer.
These are the only supported syntax involving LOBs in remote tables.
No other usage is supported.
See the link for extended examples.
To access data of type BLOB use the PIC datatype. Build your query like this:
*select(
**select PIC from AGENT_SIGNATURES_TB#DBLINK**
) as PIC
from dual*
At least that worked for me on an Oracle database.
I have a Data Flow from an Oracle table to an MSSQL table with one field of data type BIT. The Oracle table is using the characters Y and N at the moment (I'm unsure of the data type and have no way of checking), but the MSSQL table needs to be data type BIT. What type of cast can I use on the Oracle query so that the data is pulled smoothly over?
Use char(1) and then use a derived column transformation like this:
(DT_BOOL)(OracleField == "Y"?1:0)
Give this column a name like OracleFieldAsBool
and then use it instead of the original column in the rest of your data flow.