I am developing stored procedure for Oracle 10g and I am hitting pretty heavy performance bottle neck while trying to pass list of about 2-3k items into procedure. Here's my code:
TYPE ty_id_list
AS TABLE OF NUMBER(11);
----------------------------------------------------------
PROCEDURE sp_performance_test (
p_idsCollection IN schema.ty_id_list )
AS
TYPE type_numeric_table IS TABLE OF NUMBER(11) INDEX BY BINARY_INTEGER;
l_ids type_numeric_table;
data type_numeric_table;
empty type_numeric_table;
BEGIN
EXECUTE IMMEDIATE
'ALTER TABLE schema.T_TEST_TABLE NOLOGGING';
COMMIT;
SELECT COLUMN_VALUE BULK COLLECT INTO l_ids
FROM TABLE(p_idsCollection);
FOR j IN 1 .. l_ids.COUNT LOOP
data(data.count+1) := l_ids(j);
IF(MOD(data.COUNT,500) = 0 ) THEN
FORALL i IN 1 .. data.COUNT
INSERT INTO schema.T_TEST_TABLE (REF_ID, ACTIVE)
VALUES (data(i), 'Y');
data := empty;
END IF;
END LOOP;
IF(data.count IS NOT NULL) THEN
FORALL i IN 1 .. data.COUNT
INSERT INTO schema.T_TEST_TABLE (REF_ID, ACTIVE)
VALUES (data(i), 'Y');
END IF;
COMMIT;
EXECUTE IMMEDIATE
'ALTER TABLE schema.T_TEST_TABLE LOGGING';
COMMIT;
END sp_performance_test;
So the thing that adds up to the process quite drastically seems to be this part: data(data.count+1) := l_ids(j); If I skip getting element from the collection and change this line to data(data.count+1) := j;, procedure execution time will be 3-4 times faster (from over 30s to 8-9s for 3k items) - but this logic obviously is not the one i want.
Can You guys give me a hint where could I improve my code to get better performance on inserting data? If any improvements can be done really.
Thanks,
Przemek
I don't follow your logic.
You accept a collection. You copy it to another collection:
SELECT COLUMN_VALUE BULK COLLECT INTO l_ids
FROM TABLE(p_idsCollection);
And then you copy it once again, in a loop:
FOR j IN 1 .. l_ids.COUNT LOOP
data(data.count+1) := l_ids(j);
And only after that you manage to perform your 500-row-chunk bulk insert. What is wrong with bulk inserting p_idsCollection immediately?
p.s. You don't need commits after 'ALTER TABLE', ddl statements issue them implicitly.
that whole block after the DDL can be rewritten as
insert into schema.T_TEST_TABLE (REF_ID, ACTIVE)
select COLUMN_VALUE, 'Y' FROM TABLE(p_idsCollection);
You can also add hint into insert operation.
Insert /*+ append */ into tab (...) values (...)
It's change oracle work logic and it will work faster.
http://www.dba-oracle.com/t_insert_tuning.htm
Related
Hello I have this situation
I have a table with 400 millions of records I want to migrate some data to another table for get better performance.
I know that the process could be slow and heavy and I want to execute something like this
INSERT INTO TABLLE_2 SELECT NAME,TAX_ID,PROD_CODE FROM TABLE_1;
But I want this in a procedure that iterates the table from 50000 to 50000 records.
I saw this but it DELETE the rows and the target is SQLServer and not Oracle
WHILE 1 = 1
BEGIN
DELETE TOP (50000) FROM DBO.VENTAS
OUTPUT
Deleted.AccountNumber,
Deleted.BillToAddressID,
Deleted.CustomerID,
Deleted.OrderDate,
Deleted.SalesOrderNumber,
Deleted.TotalDue
INTO DATOS_HISTORY.dbo.VENTAS
WHERE OrderDate >= '20130101'
and OrderDate < '20140101'
IF ##ROWCOUNT = 0 BREAK;
END;
If it's just for a one-time migration, don't paginate/iterate at all. Just use a pdml append insert-select:
BEGIN
EXECUTE IMMEDIATE 'alter session enable parallel dml';
INSERT /*+ parallel(8) nologging append */ INTO table_2
SELECT name,tax_id,prod_code from table_1;
COMMIT;
EXECUTE IMMEDIATE 'alter session disable parallel dml';
END;
Or if the table doesn't yet exist, CTAS it to be even simpler:
CREATE OR REPLACE TABLE table_2 PARALLEL (DEGREE 8) NOLOGGING AS
SELECT name,tax_id,prod_code FROM table_1;
If you absolutely must iterate due to other business needs you didn't mention, you can use PL/SQL for that as well:
DECLARE
CURSOR cur_test
IS
SELECT name,tax_id,prod_code
FROM table_1;
TYPE test_tabtype IS TABLE OF cur_test%ROWTYPE;
tab_test test_tabtype;
var_limit integer := 50000;
BEGIN
OPEN cur_test;
FETCH cur_test BULK COLLECT INTO tab_test LIMIT var_limit;
LOOP
-- do whatever else you need to do
FORALL i IN tab_test.FIRST .. tab_test.LAST
INSERT INTO table_2
(name,tax_id,prod_code)
VALUES (tab_test(i).name, tab_test(i).tax_id, tab_test(i).prod_code);
COMMIT;
EXIT WHEN tab_test.COUNT < var_limit;
FETCH cur_test BULK COLLECT INTO tab_test LIMIT var_limit;
END LOOP;
CLOSE cur_test;
END;
That won't be nearly as fast, though. If you need even faster, you can create a SQL object and nested table type instead of the PL/SQL types you see here and then you can do a normal INSERT /*+ APPEND */ SELECT ... statement that will use direct path. But honestly, I doubt you really need iteration at all..
I wrote a stored procedure, but I don't think it is performing at all. How can I make it work better? Thanks.
Table A has 800k records. Table B has 36k records. Is test data, there will be more records in production environment.
Table B has customer_no column index defined.
I ran it once and it took 22 minutes.
create or replace procedure SP_KVKK
is
TYPE json_data_type IS RECORD
(
del_data CLOB
);
customer_no number(10);
r_del_data C%ROWTYPE;
l_deleted_cur SYS_REFCURSOR;
l_deleted_rec json_data_type;
l_sel_sql VARCHAR2 (500);
cursor mbb_list is
select customer_no from A;
begin
open mbb_list;
loop
fetch mbb_list into customer_no;
exit when mbb_list%notfound;
l_sel_sql := 'SELECT JSON_OBJECT(* RETURNING CLOB) AS DEL_DATA
FROM B where customer_no=' || customer_no;
open l_deleted_cur for l_sel_sql;
loop
fetch l_deleted_cur into l_deleted_rec;
exit when l_deleted_cur%notfound;
r_del_data.DELETED_DOCUMENT_JSON := l_deleted_rec.del_data;
r_del_data.DELETE_DATE := SYSTIMESTAMP;
Insert into C
values r_del_data;
end loop;
close l_deleted_cur;
end loop;
close mbb_list;
end;
The best thing you can do is deal with this construct as a single SQL statement. Stop thinking in terms of row-by-row nested loops which will always be slow, and look to SQL batch operations to handle this:
begin
-- insert your two columns into table C using the data
-- from table B returned as a json_object and systimestamp,
-- where only records from table B that have a customer_no
-- from table A will be selected
insert into C (deleted_document_json, delete_date)
select json_object(B.* returning clob), systimestamp
from B
inner join A using (customer_no);
commit;
end;
I have around 5 million of records which needs to be copied from table of one schema to table of another schema(in the same database). I have prepared a script but it gives me the below error.
ORA-06502: PL/SQL: numeric or value error: Bulk bind: Error in define
Following is my script
DECLARE
TYPE tA IS TABLE OF varchar2(10) INDEX BY PLS_INTEGER;
TYPE tB IS TABLE OF SchemaA.TableA.band%TYPE INDEX BY PLS_INTEGER;
TYPE tD IS TABLE OF SchemaA.TableA.start_date%TYPE INDEX BY PLS_INTEGER;
TYPE tE IS TABLE OF SchemaA.TableA.end_date%TYPE INDEX BY PLS_INTEGER;
rA tA;
rB tB;
rD tD;
rE tE;
f number :=0;
BEGIN
SELECT col1||col2||col3 as main_col, band, effective_start_date as start_date, effective_end_date as end_date
BULK COLLECT INTO rA, rB, rD, rE
FROM schemab.tableb;
FORALL i IN rA.FIRST..rE.LAST
insert into SchemaA.TableA(main_col, BAND, user_type, START_DATE, END_DATE, roll_no)
values(rA(i), rB(i), 'C', rD(i), rE(i), 71);
f:=f+1;
if (f=10000) then
commit;
end if;
end;
Could you please help me in finding where the error lies?
Why not a simple
insert into SchemaA.TableA (main_col, BAND, user_type, START_DATE, END_DATE, roll_no)
SELECT col1||col2||col3 as main_col, band, 'C', effective_start_date, effective_end_date, 71
FROM schemab.tableb;
This
f:=f+1;
if (f=10000) then
commit;
end if;
does not make any sense. f becomes 1 - that's it. f=10000 will never be true, thus you don't make a COMMIT.
Following script worked for me and i was able to load around 5 millions of data within 15 minutes.
ALTER SESSION ENABLE PARALLEL DML
/
DECLARE
cursor c_p1 is
SELECT col1||col2||col3 as main_col, band, effective_start_date as start_date, effective_end_date as end_date
FROM schemab.tableb;
TYPE TY_P1_FULL is table of c_p1%rowtype
index by pls_integer;
v_P1_FULL TY_P1_FULL;
v_seq_num number;
BEGIN
open c_p1;
loop
fetch c_p1 BULK COLLECT INTO v_P1_FULL LIMIT 10000;
exit when v_P1_FULL.count = 0;
FOR i IN 1..v_P1_FULL.COUNT loop
INSERT /*+ APPEND */ INTO schemaA.tableA VALUES (v_P1_FULL(i));
end loop;
commit;
end loop;
close c_P1;
dbms_output.put_line('Load completed');
end;
-- Disable parallel mode for this session
ALTER SESSION DISABLE PARALLEL DML
/
ORA-06502: PL/SQL: numeric or value error: Bulk bind: Error in define
You get that error because you have a literal in the VALUES clause of the INSERT. The FORALL expects everything to be bind to an array.
Your program has a massive problem, literally. You have no LIMIT on the BULK COLLECT clause, so that's going to try to load all five million records from TableB into your collections. That will blow your session's memory limit.
The point of using BULK COLLECT and FORALL is to bite off chunks of a bigger data set and process it in batches. For that you need a loop. The loop has no FOR condition: instead test whether the fetch returned anything and exit when the array has zero records.
DECLARE
TYPE recA IS RECORD (
main_col SchemaA.TableA.main_col%TYPE
, band SchemaA.TableA.band%TYPE
, start_date date
, end_date date
, roll_ni number);
TYPE recsA is table of recA
nt_a recsA;
f number :=0;
CURSOR cur_b is
SELECT col1||col2||col3 as main_col,
band,
effective_start_date as start_date,
effective_end_date as end_date ,
71 as roll_no
FROM schemab.tableb;
BEGIN
open cur_b;
loop
fetch curb_b bulk collect into nt_a limit 1000;
exit when nt_a.count() = 0;
FORALL i IN rA.FIRST..rE.LAST
insert into SchemaA.TableA(main_col, BAND, user_type, START_DATE, END_DATE, roll_no)
values nt_a(i);
f := f + sql%rowcount;
if (f > = 10000) then
commit;
f := 0;
end if;
end loop;
commit;
close cur_b;
end;
Please note that issuing commits inside a loop is contraindicated. You lay yourself open to runtime errors such as ORA-01002 and ORA-01555. If your program does crash half-way through you will have great difficulty in resuming it without problems. By all means persist if you have problems with UNDO tablespace, but the correct answer is to get the DBA to enlarge the UNDO tablespace not weaken your code.
"i am using bulk insert because it gives better performance"
It is true that BULK COLLECT and FORALL ... INSERT is more performative than a CURSOR FOR loop with row-by-row single inserts. It is not more efficient than a pure SQL INSERT INTO ... SELECT. The value of the construct is that it allows us to manipulate the contents of the array before inserting it. This is handle if we have complex business rules which can only be applied programmatically.
Please try after changing first 2 line of your code with below:
DECLARE
TYPE tA IS TABLE OF SchemaA.TableA.main_col%TYPE INDEX BY PLS_INTEGER;
...
...
This may be because of data type/length mismatch. In declaration section you have missed to declare one to inherit type from table.
Also as mentioned, f logic for commit will not do the magic for you. Better you should use LIMIT with BULL COLLECT
What's the best way of getting and outputting how many rows have been inserted in the FORALL statement I have below. I've seen the SQL%BULK_ROWCOUNT but I'm not sure how that would work in the below statement.
is it
DBMS_OUTPUT.('rows inserted '||SQL%BULK_ROWCOUNT||'');
Does the above need to go in another FORALL statement? For the code below how would I achieve this?
DECLARE
TYPE t_arc_act_plus_trigger1 IS TABLE OF arc_act_plus_triggers1%ROWTYPE;
v_arc_act_plus_triggers1 t_arc_act_plus_trigger1;
CURSOR c_arc_act_plus_triggers1 IS
SELECT /*+ PARALLEL */ apt.*
FROM act_plus_triggers1 apt
WHERE NOT EXISTS
(SELECT 1
FROM act_plus_triggers_copy1 aptc
WHERE aptc.surr_id = apt.surr_id)
AND apt.status IN ('EXT', 'EXP');
BEGIN
OPEN c_arc_act_plus_triggers1;
LOOP
FETCH c_arc_act_plus_triggers1 BULK COLLECT INTO v_arc_act_plus_triggers1 LIMIT 10000; -- limit to 10k to avoid out of memory
FORALL i IN 1..v_arc_act_plus_triggers1.COUNT
INSERT /*+ APPEND_VALUES */ INTO arc_act_plus_triggers1 values v_arc_act_plus_triggers1(i);
Com0932.get_parameter ('ACT_ARCHIVE_TRIGGER_STOP_YN',l_STOP_PROGRAM_YN);
IF l_STOP_PROGRAM_YN = 'Y' THEN
p_location('insert_into_arc_act_plus - STOP_PROGRAM_YN flag = '||l_STOP_PROGRAM_YN||' so ROLLBACK');
ROLLBACK;
EXIT;
END IF;
-- **************************************************
-- Output how many records have been inserted here???
-- **************************************************
-- commit after every 10000 records into arc_act_plus_triggers1
COMMIT;
EXIT WHEN c_arc_act_plus_triggers1%NOTFOUND;
END LOOP;
CLOSE c_arc_act_plus_triggers1;
END;
I haven't checked as I have nothing to test against so please forgive any 'missing semi-colon type errors' and I'm afraid I'm not in a position to performance check this.
Your code seems to select which rows to insert to the archive table based on there non-existence in the archive. Therefore simply use an INSERT based on a SELECT limited by a suitable ROWNUM value. Once you commit then the next time round the loop it wont try getting already archived rows as you just committed them.
I think this should be as quick if not quicker than bulkifying the inserts with the advantage that its simpler - Occams Razor and all that.
DECLARE
l_commit_count NUMBER := 10000;
l_rows_copied NUMBER := 0;
BEGIN
DBMS_OUTPUT.PUT_LINE('Started at '||TO_DATE(SYSDATE, 'DD_MON_YYY HH24:MI:SS');
LOOP
INSERT /*+APPEND */
INTO c_arc_act_plus_triggers1
SELECT /*+ PARALLEL */ apt.*
FROM act_plus_triggers1 apt
WHERE NOT EXISTS
(SELECT 1
FROM act_plus_triggers_copy1 aptc
WHERE aptc.surr_id = apt.surr_id)
AND apt.status IN ('EXT', 'EXP')
AND rownum < l_commit_count;
COMMIT;
l_rows := l_rows + SQL%ROWCOUNT;
EXIT WHEN SQL%ROWCOUNT < 1;
END LOOP
DBMS_OUTPUT.PUT_LINE('Finished at '||TO_DATE(SYSDATE, 'DD_MON_YYY HH24:MI:SS');
DBMS_OUTPUT.PUT_LINE(TO_CHAR(l_rows)||' rows copied to the archive table');
END;
I need to do some inserts in a cursor over about 300000 rows, this is however running slowly, any ideas on how i can make it run faster? Can i speed it up by batching the commits? So for example i would perform a commit after the 1000th row
DECLARE
CURSOR test_cursor IS
SELECT a from database.mytable
BEGIN
FOR curRow IN test_cursor LOOP
insert into tableb (testval)
values ('something');
commit;
END LOOP;
END;
300000 rows is not that many rows. Unless the rows are each extremely large, you should not commit in the middle of the batch.
Intermediate commits will only achieve:
additional overhead because each commit creates additional work,
loss of restartability in case of error (and loss of transactional integrity),
greater chance of running into ORA-1555
If your process is really a cursor with a single insert inside the loop, you should run a single statement:
BEGIN
INSERT INTO tableb (col1..coln) (SELECT col1..coln FROM database.mytable);
END;
If you still need extra performance, you could look into direct insert and parallel operation but It might be over-optimization with "only" 300k rows.
By far the single greatest optimization available to you is to think in term of sets instead of the traditional procedural approach that consists of batches of single row statements.
Or you can try this:
DECLARE
CURSOR test_cursor IS
SELECT col1 from table_a;
TYPE fetch_array IS TABLE OF test_cursor%ROWTYPE;
test_array fetch_array;
l_errors PLS_INTEGER;
l_dml_errors EXCEPTION;
PRAGMA EXCEPTION_INIT(l_dml_errors, -24381);
BEGIN
open test_cursor;
loop
fetch test_cursor bulk collect into test_array limit 10000;
forall i in 1..test_array.count save exceptions
insert into table_b(col1)
values(test_array(i).col1);
exit when test_cursor%notfound;
end loop;
close test_cursor;
commit;
EXCEPTION
WHEN l_dml_errors THEN
l_errors := SQL%BULK_EXCEPTIONS.COUNT;
dbms_output.put_line('Number of INSERT statements that failed: ' || l_errors);
FOR i IN 1 .. l_errors
LOOP
dbms_output.put_line('Error #' || i || ' at '|| 'iteration #' || SQL%BULK_EXCEPTIONS(i).ERROR_INDEX);
dbms_output.put_line('Error message is ' || SQLERRM(-SQL%BULK_EXCEPTIONS(i).ERROR_CODE));
END LOOP;
END;
I would not recommend a cursor approach for this. I use append parallel hints for situations like this. Most of the time your query literally runs N times as fast where N is the parallel degree. It is occasionally a good idea to bypass disaster recovery with nologging / noarchivelog.
For truly large migrations (dozens to hundreds of GB), I've found it's a good idea to batch on a table's natural key (date, usually). Some small amount of state around it can let you cancel + resume the migration at will if necessary.
May Be This will help you please try this
DECLARE
i number;
CURSOR test_cursor IS
SELECT a from database.mytable
BEGIN
FOR curRow IN test_cursor LOOP
insert into tableb (testval)
values ('something');
i:i+1;
if mod(i,1000)=0 then
commit;
end if;
END LOOP;
commit;
END;