How can i load csv file with 3.5 million records into database table?
my csv file format:
code,doi,type,j_code,title,is_no,ib_no
"ETO072","10.10ISSN","J_ID","BEJ","ABCDEF","19-38","07-93"
"ETO073","10.11ISSN","J_ID","BEJ","NSKDJJ","19-39","07-94"
"ETO074","10.12ISSN","J_ID","BEJ","WEWDSD","19-40","07-95"
"ETO075","10.13ISSN","J_ID","BEJ","UOISDJ","19-41","07-96"
table structure:
CREATE TABLE CNTNT
( ID NUMBER,
code NUMBER,
TYPE VARCHAR2(255 BYTE),
is_no VARCHAR2(13 BYTE),
ib_no VARCHAR2(13 BYTE),
J_code VARCHAR2(255 BYTE),
TITLE VARCHAR2(1000 BYTE),
JCODE VARCHAR2(20 BYTE),
DOI VARCHAR2(255 BYTE),
PRIMARY KEY (ID),
);
control file:
load data
infile 'test.csv' "str '\n'"
append
into table CNTNT
fields terminated by ','
OPTIONALLY ENCLOSED BY '"' AND '"'
trailing nullcols
(
code,
DOI ,
TYPE,
J_CODE ,
TITLE ,
is_no,
ib_no,
ID "id_seq.nextval"
)
this is what i am using to run loader:
sqlldr CONTROL=test.ctl
LOG=test.log
BAD=test.bad
skip=1;
What are the changes i need to do in control file or sqlldr command to achieve this?
Related
I have a requirement where I have to read the compressed zip blob of XLS files in the Oracle database 19c, and process the sheets inside each XLS file, using PLSQL.
I am using APEX_ZIP package and Odie's EXCELTABLE for this.
I am able to do it by first getting the individual Excel file BLOBs using apex_zip package
and passing the BLOB to EXCELTable.GetRowsto read the Excel sheets in the BLOB.
It works fine for few Excel sheets but for a few it gives me this error:
ORA-20731: Error at position 632243, expecting a [String] record
The sheets are all Character data.
This is a snippet of the code I use:
SELECT t.*
FROM Table(
ExcelTable.getRows(
(select fileblob from md_blob where lower(filename) like '%renew%')
,ExcelTableSheetList('Ownership - RENEW')
,' "COL1" varchar2(200 char)
, "COL2" varchar2(200 char)
, "COL3" varchar2(200 char)
, "COL4" varchar2(200 char)
, "COL5" varchar2(200 char)
, "COL6" varchar2(200 char)
, "COL7" varchar2(200 char)
, "COL8" varchar2(200 char)'
, '9:10'
)
)
;
Any thoughts on how to deal with it?
There is data in the spreadsheet.
If you are able to get the files as XLSX, not XLS, you can use the APEX_DATA_PARSER package to query the Excel file as if it was a table. This makes it much easier as you will not need to write any code of your own to unzip the files and dig through the contents.
I'm loading some data into oracle using an external table. One of the fields "CREATEDON" is a date but is formatted like this 20151231
How can I convert CREATEDON to a proper oracle date datatype. This is my attempt so far which doesn't work...
CREATE TABLE "AMS"."DATA"
( "BLANK" VARCHAR2(255 BYTE),
"BLANK2" VARCHAR2(255 BYTE),
"CLIENT" VARCHAR2(255 BYTE),
"MATERIAL" DATE,
"CREATEDON" VARCHAR2(255 BYTE)
)
ORGANIZATION EXTERNAL
( TYPE ORACLE_LOADER
DEFAULT DIRECTORY "DIR"
ACCESS PARAMETERS
( records delimited BY '\r\n'
skip 1
fields terminated BY '|' lrtrim missing field VALUES are NULL
)
LOCATION ( "DIR":'foo.txt' )
)
REJECT LIMIT UNLIMITED ;
You have two options:
Keep as is, and transform in your select statements, such as:
select ..., to_date(CREATEDON,'YYYYMMDD') CREATEDON
Define the format in the table DDL:
"CREATEDON" VARCHAR2(255 BYTE) DATE_FORMAT DATE MASK "YYYYMMDD"
I have this table in oracle :
CREATE TABLE mytable
(
TSTAMP Date,
prmc1 VARCHAR2(30),
prmc2 VARCHAR2(30),
prmc3 VARCHAR2(30),
prmc4 VARCHAR2(30),
prmc5 NUMBER,
prmc5 NUMBER,
prmc6 NUMBER
)
the control file is below :
load data
append
into table mytable
fields terminated by ',' TRAILING NULLCOLS
( tstamp DATE "YYYY-MM-DD HH24:MI" TERMINATED BY ",",
prmc1 ":prmc1",
prmc2 ":prmc2",
prmc3 ":prmc3",
prmc4 ":prmc4",
prmc5 INTEGER ":prmc5",
prmc6 INTEGER ":prmc6"
)
the value of the column prmc5 in the csv file is -106.436
how do i load this into the table?
You just need to specify it as DECIMAL EXTERNAL
I am also removing specifying the format as the column itself.
load data
append
into table mytable
fields terminated by ',' TRAILING NULLCOLS
( tstamp DATE "YYYY-MM-DD HH24:MI" TERMINATED BY ",",
prmc1 ,
prmc2 ,
prmc3 ,
prmc4 ,
prmc5 DECIMAL EXTERNAL,
prmc6 DECIMAL EXTERNAL
)
More details with some Example here
I can't figure out where is the syntax error in this case trying to execute the sqlloader by command line. It seems to be ok.
Command:
sqlldr myuser/mypass#myhost:1521/orcl CONTROL=tbx.ctl LOG=C:\path\to\tbx.log BAD=C:\path\to\tbx.bad skip=0
CTL file:
load data
infile 'C:\path\to\tbx.csv'
into table TBX
fields terminated by ';'
optionally enclosed by '"' AND '"'
( x,
xx,
xxx,
xxxx,
xxxxx,
xxxxxx,
xxxxxxx,
xxxxxxxx,
xxxxxxxxx
)
CSV file:
"724098100357859";"";"";"";"";"";"";""
"724098100358417";"";"";"";"";"";"";""
...
ERROR:
SQL*Loader-100: Syntax error on command-line
Table structure:
CREATE TABLE TBX
(
"x" VARCHAR2(20 BYTE),
"xx" VARCHAR2(80 BYTE),
"xxx" VARCHAR2(80 BYTE),
"xxxx" VARCHAR2(80 BYTE),
"xxxxx" VARCHAR2(60 BYTE),
"xxxxxx" VARCHAR2(60 BYTE),
"xxxxxxx" VARCHAR2(60 BYTE),
"xxxxxxxx" VARCHAR2(60 BYTE),
"xxxxxxxxx" VARCHAR2(80 BYTE)
)
If your paths have spaces in them then SQL*Loader will see a path as more than one argument, usually generating an LRM-00112 error. You haven't shown that but from comments that does seem to be the issue. You need to enclose the paths in quotes:
sqlldr myuser/mypass#myhost:1521/orcl CONTROL='tbx.ctl' LOG='C:\path\with spaces\tbx.log' BAD='C:\path\with spaces\tbx.bad' skip=0
Off-topic from the original question, but picking up from a comment... When you use #myhost:1521/orcl as your connect string, you're using an easy connection identifier (link is for SQL*Plus, but the same applies here). The final element of that is the service name for the database, which may not be the same as the SID - it could be orcl.example.com, for instance. On the database server you can run lsnrctl status or lsnrctl services to see what service names are valid. If you already have a working tnsnames.ora, though, you can use the TNS alias instead, e.g. sqlldr myusr/mypass#orcl.
I'm having an issue with an oracle object I created. The goal of the stored procedure is to either retreive all wires contained on an airplane or compare the list of wires with another one (past airplane wire configuration). You will understand that this can be quite big... I did a first shot with a table of objects. I make a select statement and loop through a cursor to insert values in my objects. The problem comes from when I try this:
OPEN P_CURSOR FOR
SELECT * FROM TABLE(CAST(GET_WIRES(P_ACTUAL_HAR_LIST, P_END_IDENT_TYPES) AS TBL_TP_WIRE_LIST));
I get the ORA-22813: operand value exceeds system limits. So, I started looking on the internet and I found that I should have used a PIPELINED function, which I tryied. But, the PIPELINED function is not allowed in a PL/SQL context, so that doesn't fit my problem, since I need to rework the object afterward. So, I tryied a Global Temporary Table and I have now performance issue. I also tryied to contact the DBA if he could increase the PGA size, which he did (it's now 150mb) but I have the same problem. Maybe it's still not big enough, but I don't want him to reincrease it if it's not going to work... Just so you have an idea, I have 1 487 761 of those:
CREATE OR REPLACE
TYPE TP_WIRE_LIST_ENTRY AS OBJECT (
ATA_NUM VARCHAR2(8 BYTE),
EFFECTIVITIES VARCHAR2(4000 BYTE),
WIRE_ID VARCHAR2(50 BYTE),
DRAWING_NO VARCHAR2(25 BYTE),
END_IDENT_1 VARCHAR2(25 BYTE),
END_IDENT_2 VARCHAR2(25 BYTE),
PIN_1 VARCHAR2(25 BYTE),
PIN_2 VARCHAR2(25 BYTE),
TERM_1 VARCHAR2(25 BYTE),
TERM_2 VARCHAR2(25 BYTE),
DES_LENGTH NUMBER(6,2),
TECH_PUBS_REMARKS VARCHAR2(500 BYTE),
WIRE_CODE VARCHAR2(25 BYTE),
W_CODE VARCHAR2(3 BYTE),
AWG VARCHAR2(3 BYTE),
COLOR VARCHAR2(4 BYTE),
GA VARCHAR2(6 BYTE),
END_IDENT_DESC_1 VARCHAR2(50 BYTE),
END_IDENT_DESC_2 VARCHAR2(50 BYTE),
TECH_PUBS_REMARKS_1 VARCHAR2(500 BYTE),
TECH_PUBS_REMARKS_2 VARCHAR2(500 BYTE),
IDENT_TYPE_ID_1 VARCHAR2(10 BYTE),
IDENT_TYPE_ID_2 VARCHAR2(10 BYTE),
BUS_NAME_1 VARCHAR2(20 BYTE),
PROPERTY_COLLECTION_ID_1 NUMBER(10,0),
BUS_NAME_2 VARCHAR2(20 BYTE),
PROPERTY_COLLECTION_ID_2 NUMBER(10,0),
X_1 NUMBER(8,3),
Y_1 NUMBER(8,3),
Z_1 NUMBER(8,3),
UOM_1 VARCHAR2(10 BYTE),
X_2 NUMBER(8,3),
Y_2 NUMBER(8,3),
Z_2 NUMBER(8,3),
UOM_2 VARCHAR2(10 BYTE),
COORD_CODE_1 VARCHAR2(10 BYTE),
COORD_CODE_2 VARCHAR2(10 BYTE),
UOM_DESC_1 VARCHAR2(50 BYTE),
UOM_DESC_2 VARCHAR2(50 BYTE),
PART_NO VARCHAR2(25 BYTE),
PART_DESC VARCHAR2(50 BYTE),
WIRE_GAUGE VARCHAR2(4 BYTE),
SPEC_REFERENCE VARCHAR2(60 BYTE),
MPS_NUM VARCHAR2(25 BYTE),
PPS_NUM VARCHAR2(25 BYTE),
INSERTION_TOOL VARCHAR2(25 BYTE),
EXTRACTION_TOOL VARCHAR2(25 BYTE)
);
/
We work with Oracle9i Enterprise Edition Release 9.2.0.6.0
Best regards,
I ran into a similar situation in a data warehousing environment at a past job.
Did your DBA increate pga_aggregate_target or _pga_max_size or both? Keep in mind that when the DBA increases pga_aggregate_target to 150MB that doesn't mean that your session would be able to grab all of that 150MB. Oracle will only allow a given session to take a small fraction of that space. Increasing pga_aggregate_target from 150MB to 200MB will only give your session a few extra megs of memory to play with for in-memory operations IIRC.
If your DB server has more available memory, you'll have to have your DBA increase both pga_aggregate_target and _pga_max_size, otherwise you'll have to look at using a GTT or creating a scratch table to dump your data into and then work on it from there.
Just wanted to add this tidbit in case it helps someone else. We suddenly started getting this error in our PL/SQL when the DBA set RESULT_CACHE_MODE to AUTO as a test for another process. We didnt know he did this until much later. Anyway, setting it back to MANUAL fixed this problem.