External Tables - Replace comma with dot in numbers - oracle

How can I replace a comma with a dot directly from the external table?
I have a CSV with that format:
aaa;12345.67;bbbbbb
ccc;23132;eeeee
Sometimes someone puts a line like that:
ddd;1111,22;fff
CREATE TABLE MYTAB_EXT
(
"TX1" VARCHAR2(20 BYTE),
"VAL1" NUMBER(13,3),
"TX2" VARCHAR2(20)
)
ORGANIZATION EXTERNAL
(
TYPE ORACLE_LOADER DEFAULT DIRECTORY "EXT_TABLES_FOO" ACCESS
PARAMETERS (
records delimited BY newline
SKIP 1
fields terminated BY ';' LRTRIM
missing field VALUES are NULL (
TX1 ,
VAL1 ,
TX2)
) LOCATION ( 'MYTAB.csv' )
)
REJECT LIMIT UNLIMITED;
Thanks

Related

ORACLE - What is the Problem with date in External Table?

I'm trying to load data from a CSV file. The date value in the file is correct.
And in the table, some values are incorrect.
CREATE TABLE "TABLE"
( "TIMESTAMP" VARCHAR2(4000)
)
ORGANIZATION EXTERNAL
( TYPE ORACLE_LOADER
DEFAULT DIRECTORY "UPLOAD"
ACCESS PARAMETERS
( records delimited by newline
CHARACTERSET CL8MSWIN1251
NODISCARDFILE
NOLOGFILE
NOBADFILE
skip 1
fields terminated BY ';' OPTIONALLY ENCLOSED BY '"'
)
LOCATION
( "UPLOAD":'table.csv'
)
)
REJECT LIMIT UNLIMITED;
What could be the mistake?

Import a CSV file into an Oracle External Table

I have seen various similar questions to this but none of the solutions seem to work for me.
I have been given a CSV file produced on a mainframe that I need to load up into Oracle. I decided to try and map it in an Oracle external table and then use this to get it inserted into Oracle.
This is my CSV:
CONTRACT_NUMBER,PRODUCTCODE,TRANSACTION_NUMBER,EFFECTIVE_DATE,AMENDMENT,TERM,ACTIVE,AGENT_NUMBER,PREMIUM,ICRATE,RCRATE,IC_ALLOW,RC_ALLOW,SPRATE,TRANSACTION_CODE,TRANSACTION_DATE,AGENT_CATEGORY,AGENT_SALES_CODE,FREQ,TOT_PREMTD,REFERENCE,ALTERNATIVE_COMMISSION_METHOD,PAXUS_REF_ID
PAXUSCT1,MAA,1,07/10/2017,NB,12,Y,2905,6000,,,1,1,,T642,,,,,6000,,,
PAXUSCT1,MAA,2,07/05/2018,INC,11,Y,2905,2400,90,3,1,1,,,,,,,8400,,,
PAXUSCT2,MAA,1,01/06/2018,NB,12,Y,T1000,540,,,1,1,,,,,,,540,,,
PAXUSCT3,MAA,1,05/06/2018,NB,12,Y,T1000,1200,,,1,1,,,,,,,1200,,,
I created this definition, and many other variations of this but I keep getting errors:
create table LD_CMS_BASIS_MIGRATION
(
contract_number VARCHAR2(8),
productcode VARCHAR2(3),
transaction_number NUMBER,
effective_date DATE,
amendment VARCHAR2(3),
term NUMBER,
active VARCHAR2(1),
agent_number VARCHAR2(5),
premium NUMBER,
icrate NUMBER,
rcrate NUMBER,
ic_allow NUMBER,
rc_allow NUMBER,
sprate NUMBER,
transaction_code VARCHAR2(4),
transaction_date DATE,
agent_category VARCHAR2(4),
agent_sales_code VARCHAR2(4),
freq VARCHAR2(1),
tot_premtd NUMBER,
reference VARCHAR2(40),
alternative_commission_method VARCHAR2(40),
paxus_ref_id VARCHAR2(8)
)
organization external
(
type ORACLE_LOADER
default directory MIGRATIONS
access parameters
(
records field names all files
fields CSV without embedded record terminators
)
location (MIGRATIONS:'CMS_BASIS_MIG.csv')
)
reject limit UNLIMITED;
When I try to read from it I get this error:
This is what is in the log file on the server:
KUP-05004: Warning: Intra source concurrency disabled because parallel select was not requested.
Field Definitions for table LD_CMS_BASIS_MIGRATION
Record format DELIMITED BY NEWLINE
Data in file has same endianness as the platform
Rows with all null fields are accepted
Fields in Data Source:
CONTRACT_NUMBER CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
PRODUCTCODE CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
TRANSACTION_NUMBER CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
EFFECTIVE_DATE CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
AMENDMENT CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
TERM CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
ACTIVE CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
AGENT_NUMBER CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
PREMIUM CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
ICRATE CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
RCRATE CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
IC_ALLOW CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
RC_ALLOW CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
SPRATE CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
TRANSACTION_CODE CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
TRANSACTION_DATE CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
AGENT_CATEGORY CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
AGENT_SALES_CODE CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
FREQ CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
TOT_PREMTD CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
REFERENCE CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
ALTERNATIVE_COMMISSION_METHOD CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
PAXUS_REF_ID CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
KUP-04117: Field name PAXUS_REF_ID
was not found in the access parameter field list or table.
KUP-04093: error processing the FIELD NAMES record in data file /u02/CAMS/MIGRATIONS/dataload/CMS_BASIS_MIG.csv
Any help is greatly appreciated.
Thanks,
Mac
################## EDIT
Answer from Tajesh below, pretty much. This is what worked. I think the Newline command is what mostly did the trick. When I had edited the CSV file and added a comma at the end of each line the it picked up the last column just fine. I did also have to add the date mask too. But, Tajesh solution means that I don't need to edit the CSV file.
create table LD_CMS_BASIS_MIGRATION
(
contract_number VARCHAR2(8),
productcode VARCHAR2(3),
transaction_number NUMBER,
effective_date DATE,
amendment VARCHAR2(3),
term NUMBER,
active VARCHAR2(1),
agent_number VARCHAR2(5),
premium NUMBER,
icrate NUMBER,
rcrate NUMBER,
ic_allow NUMBER,
rc_allow NUMBER,
sprate NUMBER,
transaction_code VARCHAR2(4),
transaction_date DATE,
agent_category VARCHAR2(4),
agent_sales_code VARCHAR2(4),
freq VARCHAR2(1),
tot_premtd NUMBER,
reference VARCHAR2(40),
alternative_commission_method VARCHAR2(40),
paxus_ref_id VARCHAR2(8)
)
ORGANIZATION EXTERNAL ( TYPE ORACLE_LOADER
DEFAULT DIRECTORY "MIGRATIONS" ACCESS PARAMETERS (
RECORDS DELIMITED BY NEWLINE
BADFILE 'CMS_BASIS_MIG_BAD.bad'
LOGFILE 'CMS_BASIS_MIG_LOG.log'
SKIP 1
FIELDS TERMINATED BY ','
DATE_FORMAT DATE MASK "dd/mm/yyyy"
MISSING FIELD VALUES ARE NULL
) LOCATION ( 'CMS_BASIS_MIG.csv' )
) REJECT LIMIT UNLIMITED
PARALLEL 5;

Can you please try with the following create table syntax?
create table LD_CMS_BASIS_MIGRATION
(
contract_number VARCHAR2(8),
productcode VARCHAR2(3),
transaction_number NUMBER,
effective_date DATE,
amendment VARCHAR2(3),
term NUMBER,
active VARCHAR2(1),
agent_number VARCHAR2(5),
premium NUMBER,
icrate NUMBER,
rcrate NUMBER,
ic_allow NUMBER,
rc_allow NUMBER,
sprate NUMBER,
transaction_code VARCHAR2(4),
transaction_date DATE,
agent_category VARCHAR2(4),
agent_sales_code VARCHAR2(4),
freq VARCHAR2(1),
tot_premtd NUMBER,
reference VARCHAR2(40),
alternative_commission_method VARCHAR2(40),
paxus_ref_id VARCHAR2(8)
)
ORGANIZATION EXTERNAL ( TYPE ORACLE_LOADER
DEFAULT DIRECTORY "MIGRATIONS" ACCESS PARAMETERS (
RECORDS DELIMITED BY NEWLINE
BADFILE 'CMS_BASIS_MIG_BAD.bad'
LOGFILE 'CMS_BASIS_MIG_LOG.log'
SKIP 1
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' MISSING FIELD VALUES ARE NULL
) LOCATION ( 'CMS_BASIS_MIG.csv' )
) REJECT LIMIT UNLIMITED
PARALLEL 5;
If the mentioned code throws an error of any type of "date conversion", then you must have to specify each column name and their format if the column's data type is the date. Example: conversion format

As the error suggests "KUP-04117: Field name PAXUS_REF_ID
was not found in the access parameter field list or table.", Oracle was unable to find a value for the column PAXUS_REF_ID, instead, it's getting a new line character as there is no value populated for this column after the last comma of the record, because of which it's throwing an error.
I have modified the second and third rows of the CSV file as below and it's generating the output as expected.
Added value 0 to the second record and white space for the third at the end of the record. After the change, External table is able to read both of these records
CONTRACT_NUMBER,PRODUCTCODE,TRANSACTION_NUMBER,EFFECTIVE_DATE,AMENDMENT,TERM,ACTIVE,AGENT_NUMBER,PREMIUM,ICRATE,RCRATE,IC_ALLOW,RC_ALLOW,SPRATE,TRANSACTION_CODE,TRANSACTION_DATE,AGENT_CATEGORY,AGENT_SALES_CODE,FREQ,TOT_PREMTD,REFERENCE,ALTERNATIVE_COMMISSION_METHOD,PAXUS_REF_ID
PAXUSCT1,MAA,1,07/10/2017,NB,12,Y,2905,6000,,,1,1,,T642,,,,,6000,,,0
PAXUSCT1,MAA,2,07/05/2018,INC,11,Y,2905,2400,90,3,1,1,,,,,,,8400,,,
PAXUSCT2,MAA,1,01/06/2018,NB,12,Y,T1000,540,,,1,1,,,,,,,540,,,
PAXUSCT3,MAA,1,05/06/2018,NB,12,Y,T1000,1200,,,1,1,,,,,,,1200,,,
To fix this, as suggested by #Hotfix, you will have to include below mentioned statement in you access parameters
missing field values are null
Also, if you face any issue with interpreting date column data, you can add below date formatter, to your access parameters.
date_format date mask "dd/mm/yyyy"
Apart from this, your dataset seems to have issues with the values of column agent_number as well for records 4 and 5, which has value T1000 for a number column.

missing values in your csv file are the problem. you Need to convert them to null. just add MISSING FIELD VALUES ARE NULL in ACCESS PARAMETERS
access parameters
(
records field names all files
fields CSV without embedded record Terminators
MISSING FIELD VALUES ARE NULL
)

Oracle loader skip lines

In a CSV, using oracle loader, how can I skip lines that aren't the header?
For exemple:
Names;Initials
SomethingForSkip
Name1;Inital1
Name2;Inital2
SomethingForSkip
Name3;Inital3
Name4;Inital4
At the moment I have this code:
CREATE TABLE t_ext_course(
UC CHAR(100),
SCIENTIFIC_FIELD CHAR(10),
DEPARTAMENT CHAR(100)
)
ORGANIZATION EXTERNAL
(
TYPE oracle_loader
DEFAULT DIRECTORY src_files
ACCESS PARAMETERS
(
RECORDS DELIMITED BY newline
BADFILE 'course.bad'
DISCARDFILE 'course.dis'
LOGFILE 'course.log'
SKIP 3
FIELDS TERMINATED BY ";" OPTIONALLY ENCLOSED BY '"' MISSING FIELD VALUES ARE NULL
(
UC CHAR(100),
SCIENTIFIC_FIELD CHAR(10),
DEPARTAMENT CHAR(100)
)
)
LOCATION ('course.csv')
)
REJECT LIMIT UNLIMITED;
Thank you in advance for your help.

You can use the LOAD WHEN parameter. You would put it in right after your SKIP 3 parameter.
With LOAD WHEN, you can specify conditions so that only the source file rows meeting that condition will be loaded.
SQL*Loader and, by extension, external tables of type oracle_loader offer a BLANKS keyword that you can use to check for empty fields. It is useful for delimited data where you do not know the field length.
You can put it all together like this:
LOAD WHEN (SCIENTIFIC_FIELD != BLANKS)

Unable to load data using external table

I'm trying to load data in an external table from a csv file.
Following is my fragment:
create table emp_ext
(
eid number,ename char(9)
)
organization external
(
TYPE ORACLE_LOADER
DEFAULT DIRECTORY test
ACCESS PARAMETERS
(
RECORDS DELIMITED BY NEWLINE
FIELDS TERMINATED BY ','
(
eid number,
ename char(9)
)
)
LOCATION('C:\Users\99002971\Desktop\empf.csv')
)
create directory test as 'C:\Users\99002971\Desktop'
grant read on directory test to matuat35 // granted using another user
When i do select * from emp_ext , i get following errors:
ORA-29913:Error in executing ODCIEXTTABLEOPEN callout
ORA-29400:data cartridge error
KUP-00554:error encountered while parsing access parameters
KUP-01005:syntax error:found ""identifier:expecting one of :"binary_float,binary_double,comma,char,date,double"
KUP-01008:the bad identifier was double
KUP-01007:at line 4 column 12
Please help

The datatype_spec section of the external table documenation shows that number isn't recognised by the loader driver. In your code that is being seen as an identifier rather than a data type, hence the error.
You can use oracle_number, or unsigned integer since it's presumably always going to be a positive integer for an employee ID; or leave it untyped and allow implicit conversion to the table column type.
Your location should also only specify the file name within the directory, not the full path:
create table emp_ext
(
eid number,ename varchar2(15)
)
organization external
(
TYPE ORACLE_LOADER
DEFAULT DIRECTORY test
ACCESS PARAMETERS
(
RECORDS DELIMITED BY NEWLINE
FIELDS TERMINATED BY ','
(
eid unsigned integer external(9),
ename char(15)
)
)
LOCATION('empf.csv')
)
Or more simply, but relying on implicit conversion:
...
ACCESS PARAMETERS
(
RECORDS DELIMITED BY NEWLINE
FIELDS TERMINATED BY ','
(
eid,
ename char(15) rteim
)
)
LOCATION('empf.csv')
)

load decimal into oracle table with sql loader

I have this table in oracle :
CREATE TABLE mytable
(
TSTAMP Date,
prmc1 VARCHAR2(30),
prmc2 VARCHAR2(30),
prmc3 VARCHAR2(30),
prmc4 VARCHAR2(30),
prmc5 NUMBER,
prmc5 NUMBER,
prmc6 NUMBER
)
the control file is below :
load data
append
into table mytable
fields terminated by ',' TRAILING NULLCOLS
( tstamp DATE "YYYY-MM-DD HH24:MI" TERMINATED BY ",",
prmc1 ":prmc1",
prmc2 ":prmc2",
prmc3 ":prmc3",
prmc4 ":prmc4",
prmc5 INTEGER ":prmc5",
prmc6 INTEGER ":prmc6"
)
the value of the column prmc5 in the csv file is -106.436
how do i load this into the table?

You just need to specify it as DECIMAL EXTERNAL
I am also removing specifying the format as the column itself.
load data
append
into table mytable
fields terminated by ',' TRAILING NULLCOLS
( tstamp DATE "YYYY-MM-DD HH24:MI" TERMINATED BY ",",
prmc1 ,
prmc2 ,
prmc3 ,
prmc4 ,
prmc5 DECIMAL EXTERNAL,
prmc6 DECIMAL EXTERNAL
)
More details with some Example here

Resources