I have a stored procedure which performs some transactions (insert / update) and want to know which of these two options run "COMMIT" more efficiently:
OPTION 1:
BEGIN
OPEN myCursor;
LOOP
FETCH myCursor INTO AUX_ID, AUX_VAR1, AUX_VAR2;
EXIT WHEN myCursor%NOTFOUND;
SELECT count(*) INTO myCount FROM myTable WHERE code = AUX_ID;
IF myCount > 0 THEN
UPDATE myTable
SET VAR1 = AUX_VAR1, VAR2 = AUX_VAR2
WHERE code = AUX_ID_BD;
COMMIT;
ELSE
INSERT INTO myTable(code, VAR1, VAR2)
VALUES(AUX_ID, AUX_VAR1, AUX_VAR2)
COMMIT;
END IF;
END LOOP;
CLOSE myCursor;
END;
OR OPTION 2:
BEGIN
OPEN myCursor;
LOOP
FETCH myCursor INTO AUX_ID, AUX_VAR1, AUX_VAR2;
EXIT WHEN myCursor%NOTFOUND;
SELECT count(*) INTO myCount FROM myTable WHERE code = AUX_ID;
IF myCount > 0 THEN
UPDATE myTable
SET VAR1 = AUX_VAR1, VAR2 = AUX_VAR2
WHERE code = AUX_ID_BD;
ELSE
INSERT INTO myTable(code, VAR1, VAR2)
VALUES(AUX_ID, AUX_VAR1, AUX_VAR2)
END IF;
END LOOP;
COMMIT;
CLOSE myCursor;
END;
it's okay? or is there a better way?
Option #2 is definitely more efficient, although it's hard to tell if it will be noticeable in your case.
Every COMMIT requires a small amount of physical I/O; Oracle must ensure all the data is written to disk, the system change number (SCN) is written to disk, and there are probably other consistency checks I'm not aware of. In practice, it takes a huge number of COMMITs from multiple users to significantly slow down a database. When that happens you may see unusual wait events involving REDO, control files, etc.
Before a COMMIT is issued, Oracle can make the changes in memory or asynchronously. This may allow the performance to be equivalent to an in-memory database.
An even better option is to avoid the issue entirely by using a single MERGE statement, as Sylvain Leroux suggested. If the processing must be done in PL/SQL, at least replace the OPEN/FETCH cursor syntax with a simpler cursor FOR-loop. A cursor FOR-loop will automatically bulk collect data, significantly improving read performance.
Related
I have a Main cursor that is working fine.
declare
v_firm_id number;
amount number;
v_total_sum TABLE_TEMP.TOTAL_SUM%TYPE;
CURSOR MT_CURSOR IS
SELECT firm_id FROM t_firm;
BEGIN
OPEN MT_CURSOR;
LOOP
FETCH MT_CURSOR INTO v_firm_id;
EXIT WHEN MT_CURSOR%NOTFOUND;
DBMS_OUTPUT.PUT_LINE(to_char(sysdate, 'mi:ss') ||'--- '|| v_firm_id)
INSERT INTO TABLE_TEMP(TOTAL_SUM) VALUES(v_firm_id)
COMMIT;
END LOOP;
DBMS_LOCK.SLEEP(20);
BEGIN
FOR loop_emp IN
(SELECT TOTAL_SUM INTO v_total_sum FROM TABLE_TEMP)
LOOP
dbms_output.put_line(to_char(sysdate, 'mi:ss') ||'--- '|| v_total_sum || '-TEST--');
END LOOP loop_emp;
END;
end;
Everything Works fine except dbms_output.put_line(v_total_sum || '---');
I do not get any data there. I get the correct number of rows. which it inserted.
The problem is the cursor FOR loop has a redundant into clause which it appears the compiler silently ignores, and so v_total_sum is never used.
Try this:
begin
for r in (
select firm_id from t_firm
)
loop
insert into table_temp (total_sum) values (r.firm_id);
end loop;
dbms_lock.sleep(20);
for r in (
select total_sum from table_temp
)
loop
dbms_output.put_line(r.total_sum || '---');
end loop;
commit;
end;
If this had been a stored procedure rather than an anonymous block and you had PL/SQL compiler warnings enabled with alter session set plsql_warnings = 'ENABLE:ALL'; (or the equivalent preference setting in your IDE) then you would have seen:
PLW-05016: INTO clause should not be specified here
I also moved the commit to the end so you only commit once.
To summarise the comments below, the Cursor FOR loop construction declares, opens, fetches and closes the cursor for you, and is potentially faster because it fetches in batches of 100 (or similar - I haven't tested in recent versions). Simpler code has less chance of bugs and is easier to maintain in the future, for example if you need to add a column to the cursor.
Note the original version had:
for loop_emp in (...)
loop
...
end loop loop_emp;
This is misleading because loop_emp is the name of the record, not the cursor or the loop. The compiler is ignoring the text after end loop although really it should at least warn you. If you wanted to name the loop, you would use a label like <<LOOP_EMP>> above it. (I always name my loop records r, similar to the i you often see used in numeric loops.)
What's the best way of getting and outputting how many rows have been inserted in the FORALL statement I have below. I've seen the SQL%BULK_ROWCOUNT but I'm not sure how that would work in the below statement.
is it
DBMS_OUTPUT.('rows inserted '||SQL%BULK_ROWCOUNT||'');
Does the above need to go in another FORALL statement? For the code below how would I achieve this?
DECLARE
TYPE t_arc_act_plus_trigger1 IS TABLE OF arc_act_plus_triggers1%ROWTYPE;
v_arc_act_plus_triggers1 t_arc_act_plus_trigger1;
CURSOR c_arc_act_plus_triggers1 IS
SELECT /*+ PARALLEL */ apt.*
FROM act_plus_triggers1 apt
WHERE NOT EXISTS
(SELECT 1
FROM act_plus_triggers_copy1 aptc
WHERE aptc.surr_id = apt.surr_id)
AND apt.status IN ('EXT', 'EXP');
BEGIN
OPEN c_arc_act_plus_triggers1;
LOOP
FETCH c_arc_act_plus_triggers1 BULK COLLECT INTO v_arc_act_plus_triggers1 LIMIT 10000; -- limit to 10k to avoid out of memory
FORALL i IN 1..v_arc_act_plus_triggers1.COUNT
INSERT /*+ APPEND_VALUES */ INTO arc_act_plus_triggers1 values v_arc_act_plus_triggers1(i);
Com0932.get_parameter ('ACT_ARCHIVE_TRIGGER_STOP_YN',l_STOP_PROGRAM_YN);
IF l_STOP_PROGRAM_YN = 'Y' THEN
p_location('insert_into_arc_act_plus - STOP_PROGRAM_YN flag = '||l_STOP_PROGRAM_YN||' so ROLLBACK');
ROLLBACK;
EXIT;
END IF;
-- **************************************************
-- Output how many records have been inserted here???
-- **************************************************
-- commit after every 10000 records into arc_act_plus_triggers1
COMMIT;
EXIT WHEN c_arc_act_plus_triggers1%NOTFOUND;
END LOOP;
CLOSE c_arc_act_plus_triggers1;
END;
I haven't checked as I have nothing to test against so please forgive any 'missing semi-colon type errors' and I'm afraid I'm not in a position to performance check this.
Your code seems to select which rows to insert to the archive table based on there non-existence in the archive. Therefore simply use an INSERT based on a SELECT limited by a suitable ROWNUM value. Once you commit then the next time round the loop it wont try getting already archived rows as you just committed them.
I think this should be as quick if not quicker than bulkifying the inserts with the advantage that its simpler - Occams Razor and all that.
DECLARE
l_commit_count NUMBER := 10000;
l_rows_copied NUMBER := 0;
BEGIN
DBMS_OUTPUT.PUT_LINE('Started at '||TO_DATE(SYSDATE, 'DD_MON_YYY HH24:MI:SS');
LOOP
INSERT /*+APPEND */
INTO c_arc_act_plus_triggers1
SELECT /*+ PARALLEL */ apt.*
FROM act_plus_triggers1 apt
WHERE NOT EXISTS
(SELECT 1
FROM act_plus_triggers_copy1 aptc
WHERE aptc.surr_id = apt.surr_id)
AND apt.status IN ('EXT', 'EXP')
AND rownum < l_commit_count;
COMMIT;
l_rows := l_rows + SQL%ROWCOUNT;
EXIT WHEN SQL%ROWCOUNT < 1;
END LOOP
DBMS_OUTPUT.PUT_LINE('Finished at '||TO_DATE(SYSDATE, 'DD_MON_YYY HH24:MI:SS');
DBMS_OUTPUT.PUT_LINE(TO_CHAR(l_rows)||' rows copied to the archive table');
END;
Is there a quick way to flush REF CURSOR into a temporary table in oracle
CURSOR size will vary between 5K to 100K rows
currently I am using a for loop, but its very slow
There is no way to change the OUT parameter for SOME_FUNCTION_CALL function.
DECLARE
v_return SYS_REFCURSOR;
VAR_A varchar2(100);
VAR_B varchar2(100);
BEGIN
v_return := SOME_FUNCTION_CALL();
LOOP
FETCH v_return into VAR_A,VAR_B
EXIT WHEN v_return%NOTFOUND
INSERT INTO temp_table(a,b) values (VAR_A,VAR_B);
END LOOP;
CLOSE v_return;
END LOOP;
Probably not.
You could potentially make the code more efficient by doing a BULK COLLECT into local collections rather than doing row-by-row fetches. Something like
DECLARE
TYPE vc100_tbl IS TABLE OF VARCHAR2(100);
l_return SYS_REFCURSOR;
l_var_a_tbl vc100_tbl;
l_var_b_tbl vc100_tbl;
BEGIN
l_return := some_function_call();
LOOP
FETCH l_return
BULK COLLECT INTO l_var_a_tbl, l_var_b_tbl
LIMIT 100;
EXIT WHEN l_var_a_tbl.count = 0;
FORALL i IN 1 .. l_var_a_tbl.count
INSERT INTO temp_table( a, b )
VALUES( l_var_a_tbl(i), l_var_b_tbl(i) );
END LOOP;
close l_return;
END;
That will reduce the number of context shifts taking place between the SQL and PL/SQL engines. But depending on your definition of "very slow", it seems unlikely that context shifts are your most pressing problem. Were I to guess, I'd wager that if your code is "very slow" for most definitions of that term, the problem is likely that the query that is being used to open the cursor in some_function is slow and that most of your time is being spent executing that query in order to generate the rows that you want to fetch. If that's the case, you'd want to optimize the query.
Architecturally, this approach also seems rather problematic. Even if you can't change the definition of the function, can't you do something to factor out the query the function is executing so that you can use the same code without calling the function? For example, could you not factor out the query the function calls into a view and then call that view from your code?
I have a routine written in T-SQL for SQL Server. We are migrating to Oracle so I am trying to port it to PL/SQL. Here is the T-SQL routine (simplified); note the use of the table-valued variable which, in Oracle, will become a "nested table" type PL/SQL variable. The main thrust of my question is on the best ways of working with such "collection" objects within PL/SQL. Several operations in the ported code (second code sample, below) are quite awkward, where they seemed a lot easier in the SQL Server original:
DECLARE #MyValueCollection TABLE( value VARCHAR(4000) );
DECLARE #valueForThisRow VARCHAR(4000);
DECLARE #dataItem1Val INT, #dataItem2Val INT, #dataItem3Val INT, #dataItem4Val INT;
DECLARE theCursor CURSOR FAST_FORWARD FOR
SELECT DataItem1, DataItem2, DataItem3, DataItem4 FROM DataTable;
OPEN theCursor;
FETCH NEXT FROM theCursor INTO #dataItem1Val, #dataItem2Val, #dataItem3Val, #dataItem4Val;
WHILE ##FETCH_STATUS = 0
BEGIN
-- About 50 lines of logic that evaluates #dataItem1Val, #dataItem2Val, #dataItem3Val, #dataItem4Val and constructs #valueForThisRow
SET #valueForThisRow = 'whatever';
-- !!! This is the row that seems to have no natural Oracle equivalent
INSERT INTO #MyValueCollection VALUES(#valueForThisRow);
FETCH NEXT FROM theCursor INTO #dataItem1Val, #dataItem2Val, #dataItem3Val, #dataItem4Val;
END;
CLOSE theCursor;
DEALLOCATE theCursor;
-- !!! output all the results; this also seems harder than it needs to be in Oracle
SELECT * FROM #MyValueCollection;
I have been able to port pretty much everything, but in two places (see comments in the code), the logic is a lot more complex than the old SQL Server way, and I wonder if there might be, in Oracle, some more graceful way that is eluding me:
set serveroutput on; -- needed for DBMS_OUTPUT; see below
DECLARE
TYPE StringList IS TABLE OF VARCHAR2(4000);
myValueCollection StringList;
dummyTempCollection StringList; -- needed for my kludge; see below
valueForThisRow VARCHAR2(4000);
BEGIN
-- build all the sql statements
FOR c IN (
SELECT DataItem1, DataItem2, DataItem3, DataItem4 FROM DataTable;
)
LOOP
-- About 50 lines of logic that evaluates c.DataItem1, c.DataItem2, c.DataItem3, c.DataItem4 and constructs valueForThisRow
valueForThisRow := 'whatever';
-- This seems way harder than it should be; I would rather not need an extra dummy collection
SELECT valueForThisRow BULK COLLECT INTO dummyTempCollection FROM dual; -- overwrites content of dummy temp
myValueCollection := myValueCollection MULTISET UNION dummyTempCollection; -- merges into main collection
END LOOP;
-- output all the results... again, there's no shorter/easier/more-compact/single-line equivalent?
IF myValueCollection.COUNT > 0
THEN
FOR indx IN myValueCollection.FIRST .. myValueCollection.LAST
LOOP
DBMS_OUTPUT.PUT_LINE(myValueCollection(indx));
END LOOP;
END IF;
END;
/
Thanks in advance for any help!
Personally, I'd take the "50 lines of logic", move it into a function that you call in your SQL statement, and then do a simple BULK COLLECT to load the data into your local collection.
Assuming that you really want to load data element-by-element into the collection, you can simplify the code that loads the collection
DECLARE
TYPE StringList IS TABLE OF VARCHAR2(4000);
myValueCollection StringList := StringList();
valueForThisRow VARCHAR2(4000);
BEGIN
-- build all the sql statements
FOR c IN (
SELECT DataItem1, DataItem2, DataItem3, DataItem4 FROM DataTable;
)
LOOP
-- About 50 lines of logic that evaluates c.DataItem1, c.DataItem2, c.DataItem3, c.DataItem4 and constructs valueForThisRow
valueForThisRow := 'whatever';
myValueCollection.extend();
myValueCollection( myValueCollection.count ) := valueForThisRow;
END LOOP;
-- output all the results... again, there's no shorter/easier/more-compact/single-line equivalent?
IF myValueCollection.COUNT > 0
THEN
FOR indx IN myValueCollection.FIRST .. myValueCollection.LAST
LOOP
DBMS_OUTPUT.PUT_LINE(myValueCollection(indx));
END LOOP;
END IF;
END;
/
If you declare the collection as an associative array, you could avoid calling extend to increase the size of the collection. If you know the number of elements that you are going to load into the collection, you could pass that to a single extend call outside the loop. Potentially, you can also eliminate the valueForThisRow local variable and just operate on elements in the collection.
As for the code that processes the collection, what is it that you are really trying to do? It would be highly unusual for production code to write to dbms_output and expect that anyone will see the output during normal processing. That will influence the way that you would write that code. Assuming that your intention is really to just call dbms_output, knowing that will generally send the data into the ether
FOR indx IN 1 .. myValueCollection.count
LOOP
dbms_output.put_line( myValueCollection(indx) );
END LOOP;
This works when you have a dense collection (all indexes between 1 and the count of the collection exist and have values). If you might have a sparse collection, you would want to use FIRST, NEXT, and LAST in a loop but that's a bit more code.
I need to do some inserts in a cursor over about 300000 rows, this is however running slowly, any ideas on how i can make it run faster? Can i speed it up by batching the commits? So for example i would perform a commit after the 1000th row
DECLARE
CURSOR test_cursor IS
SELECT a from database.mytable
BEGIN
FOR curRow IN test_cursor LOOP
insert into tableb (testval)
values ('something');
commit;
END LOOP;
END;
300000 rows is not that many rows. Unless the rows are each extremely large, you should not commit in the middle of the batch.
Intermediate commits will only achieve:
additional overhead because each commit creates additional work,
loss of restartability in case of error (and loss of transactional integrity),
greater chance of running into ORA-1555
If your process is really a cursor with a single insert inside the loop, you should run a single statement:
BEGIN
INSERT INTO tableb (col1..coln) (SELECT col1..coln FROM database.mytable);
END;
If you still need extra performance, you could look into direct insert and parallel operation but It might be over-optimization with "only" 300k rows.
By far the single greatest optimization available to you is to think in term of sets instead of the traditional procedural approach that consists of batches of single row statements.
Or you can try this:
DECLARE
CURSOR test_cursor IS
SELECT col1 from table_a;
TYPE fetch_array IS TABLE OF test_cursor%ROWTYPE;
test_array fetch_array;
l_errors PLS_INTEGER;
l_dml_errors EXCEPTION;
PRAGMA EXCEPTION_INIT(l_dml_errors, -24381);
BEGIN
open test_cursor;
loop
fetch test_cursor bulk collect into test_array limit 10000;
forall i in 1..test_array.count save exceptions
insert into table_b(col1)
values(test_array(i).col1);
exit when test_cursor%notfound;
end loop;
close test_cursor;
commit;
EXCEPTION
WHEN l_dml_errors THEN
l_errors := SQL%BULK_EXCEPTIONS.COUNT;
dbms_output.put_line('Number of INSERT statements that failed: ' || l_errors);
FOR i IN 1 .. l_errors
LOOP
dbms_output.put_line('Error #' || i || ' at '|| 'iteration #' || SQL%BULK_EXCEPTIONS(i).ERROR_INDEX);
dbms_output.put_line('Error message is ' || SQLERRM(-SQL%BULK_EXCEPTIONS(i).ERROR_CODE));
END LOOP;
END;
I would not recommend a cursor approach for this. I use append parallel hints for situations like this. Most of the time your query literally runs N times as fast where N is the parallel degree. It is occasionally a good idea to bypass disaster recovery with nologging / noarchivelog.
For truly large migrations (dozens to hundreds of GB), I've found it's a good idea to batch on a table's natural key (date, usually). Some small amount of state around it can let you cancel + resume the migration at will if necessary.
May Be This will help you please try this
DECLARE
i number;
CURSOR test_cursor IS
SELECT a from database.mytable
BEGIN
FOR curRow IN test_cursor LOOP
insert into tableb (testval)
values ('something');
i:i+1;
if mod(i,1000)=0 then
commit;
end if;
END LOOP;
commit;
END;