I have a simple question, how exactly do you use the oracle import tool to import a database with the option of automatically resize the column length, so it automatically fits the data before importing the data.
Give you an example: if I have a table TABLE1 that has a column called "comment", comment field length is 250. Since I'm importing TABLE1 from source (which is in western character set) into target database (which is AL-UTF32 character set). Some of the records data will grow, i.e. 1 record's comment field data will grow from 250 into 260 because of the character set conversion.
My question is: how do I import TABLE1, so that target database will automatically change the field "comment" from 250 into the max data field length of this field (after character set conversion grows the data). So I can import TABLE1 with no errors.
What's the import option or command line? Is there a way to know which columns cause data issue?
Thank you
Ideally, you would build your target table beforehand, with the column widths you need defined at that point. You would then tailor a sqlldr (SQL Loader) control file to your input format.
Related
I have flat file data source and Oracle Destination. When I use lookup transformation to validate some data exists in certain tables, it always gives no match. I have converted the data type in derived column, it exactly matches with the table data type. Data type of the column is Unicode string. Exact matches leads to no match output. I am using native SSIS OLEDB connection for Oracle. Please help.
Check your case sensitivity. Sreekanth will not match sreekanth or SREEKANTH
Check for trailing spaces, even in variable length columns.
Those are the two biggest stumbling blocks. If those are not it, then edit your question with the source table definition, the lookup table definition and an appropriate insert statement for both so we can help identify the specifics of what is awry.
Can anyone help me with the below query. Do no of column should match in source and target table while export and import data from source to destination using datapump in Oracle 11.1
Eg: we are exporting sourcedb.tab(10 columns) and importing to targetdb.tab(11 columns).
Will this work or will give an error.
This should work but I haven't tried.
From Oracle 11.2 documentation (Can't find that for 11.1, but most likely the same):
When Data Pump detects that the source table and target table do not
match (the two tables do not have the same number of columns or the
target table has a column name that is not present in the source
table), it compares column names between the two tables. If the tables
have at least one column in common, then the data for the common
columns is imported into the table (assuming the datatypes are
compatible). The following restrictions apply:
This behavior is not > supported for network imports.
The following types of columns cannot be dropped: object columns,
object attributes, nested table columns, and ref columns based on a
primary key.
Also note that you need to set parameter TABLE_EXISTS_ACTION=APPEND (or TRUNCATE , which remove all existing data). Otherwise, data pump will take the default value of SKIP leaving the table as is.
11.2 Documentation of Data Pump Import
It won't work, as far as I can tell. Target table has to match the source table.
So, what can you do?
create a database link between those two databases and insert rows manually, e.g.
insert into target#db_link (col1, col2, ..., col10, col11)
select col1, col2, ..., col10, null
from source
drop 11th column from the target table, perform import and then alter table to re-create the 11th column
If there is a table in Oracle(or any RDBMS) which contains data that is flushed out every day.
example:
1234,Raj,Kolkata,1000,09092015
Suppose, I import this row today using a standard sqoop import and store in HDFS in flatfile. Next day, the row is deleted from the source table.But if the same record is updated(say the sal field 1000 is updated to 2000) after 7 days.
If I run again a sqoop query how will it treat the data and how will it store?
Will there be two entries of the same record or the newer value will be updated?
will this record
<1234, Raj, Kolkata, 1000, 09092015>
be replaced by this one?
<1234, Raj, Kolkata, 2000, 09092015>
If you perform incremental imports in Sqoop, you can control what happens when one of the rows is updated as well as what happens when new rows are inserted by means of using the argument --incremental. You have two options:
append (sqoop import (...) --incremental append) This option is used when new rows are continually added to your database and you want to import them. In this case, you'd need to tell Sqoop the column that it has to check (in order to detect these new rows), by means of the check-column parameter.
lastmodified (sqoop import (...) --incremental lastmodified). This option is what you want in your example, it lets you tell Sqoop that you want to check for updated rows in the table (that you already imported) and set them to the new values. You have to bear in mind that you have to specify, by means of the parameter --check-column, the column name which Sqoop will use to detect the updated rows, and also that this column is required to hold a date value (for instance, date, datetime, time or timestamp). In your example you would need an extra column holding a date value, and you should update that value every time you change the value of any of the other columns, in order for that row to be imported.
Of course, if you update a row but you don't update the field specified by check-column of that row, it will not be updated in your destination table.
I hope this helps.
Let me give an example: I exported 1TB of data yesterday. Today, the database got another 1GB of data. If I try to import the data again today, Sqoop will import 1TB+1GB of data, then I am merging it. So it's a headache. I want to import only new data and append it to the old data. In this way, on a daily basis, I'll pull the RDBMS data into HDFS.
You can use sqoop Incremental Imports:
Sqoop provides an incremental import mode which can be used to retrieve only rows newer than some previously-imported set of rows.
Incremental import arguments:
--check-column (col) Specifies the column to be examined when determining which rows to import.
--incremental (mode) Specifies how Sqoop determines which rows are new. Legal values for mode include append and last modified.
--last-value (value) Specifies the maximum value of the check column from the previous import.
Reference: https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html#_incremental_imports
For Incremental Import: You would need to specify a value in a check column against a reference value for the most recent import. For example, if the –incremental append argument was specified, along with –check-column id and –last-value 100, all rows with id > 100 will be imported. If an incremental import is run from the command line, the value which should be specified as –last-value in a subsequent incremental import will be printed to the screen for your reference. If an incremental import is run from a saved job, this value will be retained in the saved job. Subsequent runs of sqoop job –exec some Incremental Job will continue to import only newer rows than those previously imported.
For importing all the tables at one go, you would need to use sqoop-import-all-tables command, but this command must satisfy the below criteria to work
Each table must have a single-column primary key.
You must intend to import all columns of each table.
You must not intend to use non-default splitting column, nor impose any conditions via a WHERE clause.
Reference: https://hortonworks.com/community/forums/topic/sqoop-incremental-import/
I have a Column (SALARY) in Source Table from Relational DB,
for example 15000 is a record in SALARY column
and I want to format it as $15,000.00 into the Target table which is a Relational DB
using Expression Transformation.
Thankyou,
Ajay
This can be done using below steps:
Pull the required column from source to Expression transformation.
Create a derived column as the concatenation of $ and the input column by making use of the equation CONCAT('$',Source_Column)
Load this new column to the target, instead of the column from the source.
I hope this is just to learn the functionality. In real life scenarios, this is a bad practice. We need not keep these symbols and all in tables. This can be directly handled at reporting level.
may be the scenario where salary column has salary of different types say 'dollor','euro','pounds', etc in same table.
still i agree with Thomas Cherian that is not good practice to store the symbol with data itself.
if this is the scenario you can do 2 things.
--1.) add another column in table and store the value of type there or add another column which stores the conversion rate also.
--2.) convert all the values in once currency and store it without symbol.