I have seen various similar questions to this but none of the solutions seem to work for me.
I have been given a CSV file produced on a mainframe that I need to load up into Oracle. I decided to try and map it in an Oracle external table and then use this to get it inserted into Oracle.
This is my CSV:
CONTRACT_NUMBER,PRODUCTCODE,TRANSACTION_NUMBER,EFFECTIVE_DATE,AMENDMENT,TERM,ACTIVE,AGENT_NUMBER,PREMIUM,ICRATE,RCRATE,IC_ALLOW,RC_ALLOW,SPRATE,TRANSACTION_CODE,TRANSACTION_DATE,AGENT_CATEGORY,AGENT_SALES_CODE,FREQ,TOT_PREMTD,REFERENCE,ALTERNATIVE_COMMISSION_METHOD,PAXUS_REF_ID
PAXUSCT1,MAA,1,07/10/2017,NB,12,Y,2905,6000,,,1,1,,T642,,,,,6000,,,
PAXUSCT1,MAA,2,07/05/2018,INC,11,Y,2905,2400,90,3,1,1,,,,,,,8400,,,
PAXUSCT2,MAA,1,01/06/2018,NB,12,Y,T1000,540,,,1,1,,,,,,,540,,,
PAXUSCT3,MAA,1,05/06/2018,NB,12,Y,T1000,1200,,,1,1,,,,,,,1200,,,
I created this definition, and many other variations of this but I keep getting errors:
create table LD_CMS_BASIS_MIGRATION
(
contract_number VARCHAR2(8),
productcode VARCHAR2(3),
transaction_number NUMBER,
effective_date DATE,
amendment VARCHAR2(3),
term NUMBER,
active VARCHAR2(1),
agent_number VARCHAR2(5),
premium NUMBER,
icrate NUMBER,
rcrate NUMBER,
ic_allow NUMBER,
rc_allow NUMBER,
sprate NUMBER,
transaction_code VARCHAR2(4),
transaction_date DATE,
agent_category VARCHAR2(4),
agent_sales_code VARCHAR2(4),
freq VARCHAR2(1),
tot_premtd NUMBER,
reference VARCHAR2(40),
alternative_commission_method VARCHAR2(40),
paxus_ref_id VARCHAR2(8)
)
organization external
(
type ORACLE_LOADER
default directory MIGRATIONS
access parameters
(
records field names all files
fields CSV without embedded record terminators
)
location (MIGRATIONS:'CMS_BASIS_MIG.csv')
)
reject limit UNLIMITED;
When I try to read from it I get this error:
This is what is in the log file on the server:
KUP-05004: Warning: Intra source concurrency disabled because parallel select was not requested.
Field Definitions for table LD_CMS_BASIS_MIGRATION
Record format DELIMITED BY NEWLINE
Data in file has same endianness as the platform
Rows with all null fields are accepted
Fields in Data Source:
CONTRACT_NUMBER CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
PRODUCTCODE CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
TRANSACTION_NUMBER CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
EFFECTIVE_DATE CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
AMENDMENT CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
TERM CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
ACTIVE CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
AGENT_NUMBER CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
PREMIUM CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
ICRATE CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
RCRATE CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
IC_ALLOW CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
RC_ALLOW CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
SPRATE CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
TRANSACTION_CODE CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
TRANSACTION_DATE CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
AGENT_CATEGORY CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
AGENT_SALES_CODE CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
FREQ CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
TOT_PREMTD CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
REFERENCE CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
ALTERNATIVE_COMMISSION_METHOD CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
PAXUS_REF_ID CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
KUP-04117: Field name PAXUS_REF_ID
was not found in the access parameter field list or table.
KUP-04093: error processing the FIELD NAMES record in data file /u02/CAMS/MIGRATIONS/dataload/CMS_BASIS_MIG.csv
Any help is greatly appreciated.
Thanks,
Mac
################## EDIT
Answer from Tajesh below, pretty much. This is what worked. I think the Newline command is what mostly did the trick. When I had edited the CSV file and added a comma at the end of each line the it picked up the last column just fine. I did also have to add the date mask too. But, Tajesh solution means that I don't need to edit the CSV file.
create table LD_CMS_BASIS_MIGRATION
(
contract_number VARCHAR2(8),
productcode VARCHAR2(3),
transaction_number NUMBER,
effective_date DATE,
amendment VARCHAR2(3),
term NUMBER,
active VARCHAR2(1),
agent_number VARCHAR2(5),
premium NUMBER,
icrate NUMBER,
rcrate NUMBER,
ic_allow NUMBER,
rc_allow NUMBER,
sprate NUMBER,
transaction_code VARCHAR2(4),
transaction_date DATE,
agent_category VARCHAR2(4),
agent_sales_code VARCHAR2(4),
freq VARCHAR2(1),
tot_premtd NUMBER,
reference VARCHAR2(40),
alternative_commission_method VARCHAR2(40),
paxus_ref_id VARCHAR2(8)
)
ORGANIZATION EXTERNAL ( TYPE ORACLE_LOADER
DEFAULT DIRECTORY "MIGRATIONS" ACCESS PARAMETERS (
RECORDS DELIMITED BY NEWLINE
BADFILE 'CMS_BASIS_MIG_BAD.bad'
LOGFILE 'CMS_BASIS_MIG_LOG.log'
SKIP 1
FIELDS TERMINATED BY ','
DATE_FORMAT DATE MASK "dd/mm/yyyy"
MISSING FIELD VALUES ARE NULL
) LOCATION ( 'CMS_BASIS_MIG.csv' )
) REJECT LIMIT UNLIMITED
PARALLEL 5;
Can you please try with the following create table syntax?
create table LD_CMS_BASIS_MIGRATION
(
contract_number VARCHAR2(8),
productcode VARCHAR2(3),
transaction_number NUMBER,
effective_date DATE,
amendment VARCHAR2(3),
term NUMBER,
active VARCHAR2(1),
agent_number VARCHAR2(5),
premium NUMBER,
icrate NUMBER,
rcrate NUMBER,
ic_allow NUMBER,
rc_allow NUMBER,
sprate NUMBER,
transaction_code VARCHAR2(4),
transaction_date DATE,
agent_category VARCHAR2(4),
agent_sales_code VARCHAR2(4),
freq VARCHAR2(1),
tot_premtd NUMBER,
reference VARCHAR2(40),
alternative_commission_method VARCHAR2(40),
paxus_ref_id VARCHAR2(8)
)
ORGANIZATION EXTERNAL ( TYPE ORACLE_LOADER
DEFAULT DIRECTORY "MIGRATIONS" ACCESS PARAMETERS (
RECORDS DELIMITED BY NEWLINE
BADFILE 'CMS_BASIS_MIG_BAD.bad'
LOGFILE 'CMS_BASIS_MIG_LOG.log'
SKIP 1
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' MISSING FIELD VALUES ARE NULL
) LOCATION ( 'CMS_BASIS_MIG.csv' )
) REJECT LIMIT UNLIMITED
PARALLEL 5;
If the mentioned code throws an error of any type of "date conversion", then you must have to specify each column name and their format if the column's data type is the date. Example: conversion format
As the error suggests "KUP-04117: Field name PAXUS_REF_ID
was not found in the access parameter field list or table.", Oracle was unable to find a value for the column PAXUS_REF_ID, instead, it's getting a new line character as there is no value populated for this column after the last comma of the record, because of which it's throwing an error.
I have modified the second and third rows of the CSV file as below and it's generating the output as expected.
Added value 0 to the second record and white space for the third at the end of the record. After the change, External table is able to read both of these records
CONTRACT_NUMBER,PRODUCTCODE,TRANSACTION_NUMBER,EFFECTIVE_DATE,AMENDMENT,TERM,ACTIVE,AGENT_NUMBER,PREMIUM,ICRATE,RCRATE,IC_ALLOW,RC_ALLOW,SPRATE,TRANSACTION_CODE,TRANSACTION_DATE,AGENT_CATEGORY,AGENT_SALES_CODE,FREQ,TOT_PREMTD,REFERENCE,ALTERNATIVE_COMMISSION_METHOD,PAXUS_REF_ID
PAXUSCT1,MAA,1,07/10/2017,NB,12,Y,2905,6000,,,1,1,,T642,,,,,6000,,,0
PAXUSCT1,MAA,2,07/05/2018,INC,11,Y,2905,2400,90,3,1,1,,,,,,,8400,,,
PAXUSCT2,MAA,1,01/06/2018,NB,12,Y,T1000,540,,,1,1,,,,,,,540,,,
PAXUSCT3,MAA,1,05/06/2018,NB,12,Y,T1000,1200,,,1,1,,,,,,,1200,,,
To fix this, as suggested by #Hotfix, you will have to include below mentioned statement in you access parameters
missing field values are null
Also, if you face any issue with interpreting date column data, you can add below date formatter, to your access parameters.
date_format date mask "dd/mm/yyyy"
Apart from this, your dataset seems to have issues with the values of column agent_number as well for records 4 and 5, which has value T1000 for a number column.
missing values in your csv file are the problem. you Need to convert them to null. just add MISSING FIELD VALUES ARE NULL in ACCESS PARAMETERS
access parameters
(
records field names all files
fields CSV without embedded record Terminators
MISSING FIELD VALUES ARE NULL
)
Related
How can I replace a comma with a dot directly from the external table?
I have a CSV with that format:
aaa;12345.67;bbbbbb
ccc;23132;eeeee
Sometimes someone puts a line like that:
ddd;1111,22;fff
CREATE TABLE MYTAB_EXT
(
"TX1" VARCHAR2(20 BYTE),
"VAL1" NUMBER(13,3),
"TX2" VARCHAR2(20)
)
ORGANIZATION EXTERNAL
(
TYPE ORACLE_LOADER DEFAULT DIRECTORY "EXT_TABLES_FOO" ACCESS
PARAMETERS (
records delimited BY newline
SKIP 1
fields terminated BY ';' LRTRIM
missing field VALUES are NULL (
TX1 ,
VAL1 ,
TX2)
) LOCATION ( 'MYTAB.csv' )
)
REJECT LIMIT UNLIMITED;
Thanks
I have an external table which reads from a CSV file and is failing on certain rows.
External table definition:
E_ID NUMBER
A_IND VARCHAR2 (3 Byte)
B_IND VARCHAR2 (3 Byte)
E_DATE DATE
E_AMT NUMBER
F_DATE DATE
D_E_DATE DATE
I see the following info from a log file generated when I select * from the external table.
KUP-05004: Warning: Intra source concurrency disabled because parallel select was not requested.
Field Definitions for table EXTERNAL_TABLE_XTL
Record format DELIMITED BY NEWLINE
Data in file has same endianness as the platform
Rows with all null fields are accepted
Fields in Data Source:
E_ID CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
A_IND CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
B_IND CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
E_DATE CHAR (10)
Date datatype DATE, date mask MM/DD/YYYY
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
E_AMT CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
F_DATE CHAR (10)
Date datatype DATE, date mask MM/DD/YYYY
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
D_E_DATE CHAR (10)
Date datatype DATE, date mask MM/DD/YYYY
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
KUP-04021: field formatting error for field D_E_DATE
KUP-04026: field too long for datatype
KUP-04101: record 56 rejected in file /home/TEST.csv
KUP-04021: field formatting error for field D_E_DATE
KUP-04026: field too long for datatype
KUP-04101: record 61 rejected in file /home/TEST.csv
KUP-04021: field formatting error for field D_E_DATE
KUP-04026: field too long for datatype
KUP-04101: record 70 rejected in file /home/TEST.csv
The file was transferred to the server via FileZilla. From reading other posts I thought maybe it was because the file was transferred in binary mode (it was originally on Auto setting) and maybe some non-printed characters have came in. So I tried to transfer using ASCII setting but that did not work. Then I tried to delete one of the lines that caused an error and retype it in manually. That did not work either.
Failed sample data:
5560000,N,Y,,24950,10/12/2011,10/27/2011
5550001,Y,Y,11/26/2013,73813,11/18/2013,11/29/2013
5560002,Y,Y,11/6/2015,22041.28,11/6/2015,11/18/2015
5560003,Y,Y,10/10/2012,2768.66,10/10/2012,10/24/2012
5560004,N,Y,,29750,9/30/2013,10/15/2013
5560005,Y,Y,10/8/2015,76474.84,10/8/2015,10/21/2015
5560006,N,Y,,63879.28,11/16/2011,11/30/2011
5560007,N,Y,,100000,11/14/2013,11/21/2013
Successful sample data:
5560008,Y,N,11/1/2010,,,
5550009,Y,N,,,,
5550010,N,N,,,,
5550011,N,N,,,,
5560012,Y,Y,2/12/2016,50000,2/12/2016,2/23/2016
5560013,Y,N,7/22/2011,,,
My first assumption is for some reason double digit months are not being accepted for the field D_E_DATE. Please note this is successful in the dev environment but not production and both are the same database version.
The following is working fine for me.
Table Definition:
CREATE TABLE my_data (
E_ID NUMBER,
A_IND VARCHAR2 (3 Byte),
B_IND VARCHAR2 (3 Byte),
E_DATE DATE,
E_AMT NUMBER,
F_DATE DATE,
D_E_DATE DATE
)
ORGANIZATION EXTERNAL (
TYPE ORACLE_LOADER
DEFAULT DIRECTORY MY_DIR
ACCESS PARAMETERS (
RECORDS DELIMITED BY NEWLINE
FIELDS TERMINATED BY ','
MISSING FIELD VALUES ARE NULL
(
E_ID,
A_IND,
B_IND,
E_DATE date 'MM/DD/YYYY',
E_AMT,
F_DATE date 'MM/DD/YYYY',
D_E_DATE date 'MM/DD/YYYY'
)
)
LOCATION ('data.txt')
);
Sample Data:
[oracle#ora12c Desktop]$ cat data.txt
5560000,N,Y,,24950,10/12/2011,10/27/2011
5550001,Y,Y,11/26/2013,73813,11/18/2013,11/29/2013
5560002,Y,Y,11/6/2015,22041.28,11/6/2015,11/18/2015
5560003,Y,Y,10/10/2012,2768.66,10/10/2012,10/24/2012
5560004,N,Y,,29750,9/30/2013,10/15/2013
5560005,Y,Y,10/8/2015,76474.84,10/8/2015,10/21/2015
5560006,N,Y,,63879.28,11/16/2011,11/30/2011
5560007,N,Y,,100000,11/14/2013,11/21/2013
Output:
SQL> select * from my_date;
E_ID A_I B_I E_DATE E_AMT F_DATE D_E_DATE
---------- --- --- --------- ---------- --------- ---------
5560000 N Y 24950 12-OCT-11 27-OCT-11
5550001 Y Y 26-NOV-13 73813 18-NOV-13 29-NOV-13
5560002 Y Y 06-NOV-15 22041.28 06-NOV-15 18-NOV-15
5560003 Y Y 10-OCT-12 2768.66 10-OCT-12 24-OCT-12
5560004 N Y 29750 30-SEP-13 15-OCT-13
5560005 Y Y 08-OCT-15 76474.84 08-OCT-15 21-OCT-15
5560006 N Y 63879.28 16-NOV-11 30-NOV-11
5560007 N Y 100000 14-NOV-13 21-NOV-13
8 rows selected.
The answer to this question was found in the following thread:
Oracle external table date field - works in one DB and not in another
Transferring the same file from the dev server to prod server seemed to have resolved the issue. Weird, I wish I knew better exactly why this issue occurred and how to resolve it.
I have this table in oracle :
CREATE TABLE mytable
(
TSTAMP Date,
prmc1 VARCHAR2(30),
prmc2 VARCHAR2(30),
prmc3 VARCHAR2(30),
prmc4 VARCHAR2(30),
prmc5 NUMBER,
prmc5 NUMBER,
prmc6 NUMBER
)
the control file is below :
load data
append
into table mytable
fields terminated by ',' TRAILING NULLCOLS
( tstamp DATE "YYYY-MM-DD HH24:MI" TERMINATED BY ",",
prmc1 ":prmc1",
prmc2 ":prmc2",
prmc3 ":prmc3",
prmc4 ":prmc4",
prmc5 INTEGER ":prmc5",
prmc6 INTEGER ":prmc6"
)
the value of the column prmc5 in the csv file is -106.436
how do i load this into the table?
You just need to specify it as DECIMAL EXTERNAL
I am also removing specifying the format as the column itself.
load data
append
into table mytable
fields terminated by ',' TRAILING NULLCOLS
( tstamp DATE "YYYY-MM-DD HH24:MI" TERMINATED BY ",",
prmc1 ,
prmc2 ,
prmc3 ,
prmc4 ,
prmc5 DECIMAL EXTERNAL,
prmc6 DECIMAL EXTERNAL
)
More details with some Example here
I have a SQL Loader Control file,
LOAD DATA
INFILE 'test.txt'
INTO TABLE TEST replace
fields terminated "|" optionally enclosed by '"' TRAILING NULLCOLS
( DOCUMENTID INTEGER(10),
CUSTID INTEGER(10),
USERID INTEGER(10),
FILENAME VARCHAR(255),
LABEL VARCHAR(50),
DESCRIPTION VARCHAR(2000),
POSTDATE DATE "YYYY-MM-DD HH24:MI:SS" NULLIF POSTDATE="",
USERFILENAME VARCHAR(50),
STORAGEPATH VARCHAR(255)
)
and it's giving me an error when I run SQL Loader on it,
Record 1: Rejected - Error on table TEST, column FILENAME.
Variable length field exceeds maximum length.
Here's that row.. the length of that column is way under 255..
1|5001572|2|/Storage/Test/5001572/test.pdf|test.pdf||2005-01-13 11:47:49||
And here's an oddity I noticed within the log file
Column Name | Position | Len | Term | Encl | Datatype
FILENAME | NEXT | 257 | | | VARCHAR
I define the length as 255 in both my table and control file. Yet the log spits it out as 257? I've tried knocking down the length in the control file to 253, so it appears as 255 in the log file, but the same issue.
Any help? This has bugged me for two days now.
Thanks.
Don't define your data fields as VARCHAR2 and INTEGER. Use CHAR. Most of the time, when loading data from a text file, you want to use CHAR, or perhaps DATE, although even that is converted from a text form. Most of the time you don't even need a length specifier. The default length for a CHAR field is 255. Your control file should look something like:
LOAD DATA
INFILE "test.txt"
INTO TABLE TEST replace
fields terminated "|" optionally enclosed by '"' TRAILING NULLCOLS
(
DOCUMENTID,
CUSTID,
USERID ,
FILENAME,
LABEL,
DESCRIPTION CHAR(2000),
POSTDATE DATE "YYYY-MM-DD HH24:MI:SS" NULLIF POSTDATE=BLANKS,
USERFILENAME,
STORAGEPATH
)
+1 for DCookie, but to expand on that it's important to distinguish between data types as specified in a table and data types in a SQL*loader control file as they mean rather different things, confusingly.
Start with a look at the the documentation, and note that when loading regular text files you need to be using the "portable" data types.
Varchar is a "non-portable" type, in which:
... consists of a binary length subfield followed by a character string of the specified length
So as DCookie says, CHAR is the thing to go for, and INTEGER EXTERNAL is a very commonly used SQL*Loader data type which you'd probably want to specify for DOCUMENTID etc.
I am having text file contaning field in below manner.
"64252368","7489040","305762",
"64285217","12132108","787341",
I am using a below control file.
OPTIONS (SKIP=1)
LOAD DATA
TRUNCATE INTO TABLE test_table
FIELDS TERMINATED BY '",'
(
LEARNEVENT_ID,
ORGANIZATION,
COURSE_ID
)
But, I am getting the error:
Record 1: Rejected - Error on table test_table, column LEARNEVENT_ID
ORA-01722: invalid number
Kindly help me on it.
You need to change your ctl file to include OPTIONALLY ENCLOSED BY option.
OPTIONS (SKIP=1)
LOAD DATA
TRUNCATE INTO TABLE test_table
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
(
LEARNEVENT_ID,
ORGANIZATION,
COURSE_ID
)
I'd recommend reading up on SQL*Loader.
The problem lays with the encapsulation of the numbers with the quotes " " and your fields terminated by '",' simply does not strip the quotes.
Try this
OPTIONS(SKIP=1)
LOAD DATA
TRUNCATE INTO TABLE test_table
FIELDS TERMINATED BY ','
TRAILING NULLCOLS
(
LEARNEVENT_ID "replace ( :LEARNEVENT_ID ,'"', '')",
ORGINAZATION "replace ( :ORGINAZATION ,'"', '')",
COURSE_ID "replace ( :COURSE_ID ,'"', '')"
)