I have a pl/sql procedure (Oracle 11g) that takes a long time to execute
(3h for 195.000 rows).
So the goal is to speed it up.
The code uses bulk collect and for all supposedly to speed up dmls, however there are some computations that are done on bulk collected data. These are done inside a classical for loop. And this for loop also accesses some other tables (performs selects) in order to perform computations needed. I think that part is one that slows everything down.
Consider the code below (it is modified and stripped down version of the real code - just to give you a gist of what is going on):
procedure long_runnig_task is
TYPE my_record is RECORD(key1 number, key2 number, key3 number ,
key4 number, p1 number ,p2 number, place_holder1 number,
place_holder2 number,place_holder3 number );
TYPE my_record_table IS TABLE OF my_record;
l_data my_record_table;
cursor c is
select key1, key2 , key3 ,key4 ,
(select param from paramtable where param_id=1) p1,
(select param from paramtable where param_id=2) p2,
0 place_holder1 ,0 place_holder2, 0 place_holder3 from mytable where myflag=4;
begin
open c;
loop
begin
fetch c bulk collect into l_data limit 1000;
savepoint mysp;
FOR indx IN 1 .. l_data.COUNT
loop
--computations per record
select max(amount) into myValue1 from table3 where
key1=l_data(indx).kay1 and
key2=l_data(indx).kay2 and key3=l_data(indx).kay3 and key4=l_data(indx).kay4;
select amount into myValue2 from table4 where key1=l_data(indx).kay1
and
key2=l_data(indx).kay2 and key3=l_data(indx).kay3 and key4=l_data(indx).kay4;
select amount into myValue3 from table5 where key1=l_data(indx).kay1
and
key2=l_data(indx).key2 and key3=l_data(indx).key3 and key4=l_data(indx).key4;
l_data(indx).place_holder1 := myValue1;
l_data(indx).place_holder2 := someFunction(myValue2,l_data(indx).p1);
l_data(indx).palce_holder3 := myValue3*l_data(indx).p2;
end loop;
forall indx IN 1 .. l_data.COUNT
update table6 set v= l_data(indx).place_holder1 where key1=l_data(indx).key1
and
key2=l_data(indx).key2 and key3=l_data(indx).key3 and key4=l_data(indx).key4;
forall indx IN 1 .. l_data.COUNT
insert into table7(col1,col2,col3) values (l_data(indx).place_holder3,sysdate,l_data(indx).place_holder2/10);
exception when others then
rollback to mysp;
raise;
end;
exit when c%notfound;
end loop;
exception when others then
rollback;
raise;
end;
Any suggestions on how to optimise the performance of the above are welcome.
Thanks.
This could be done in two sql statements:
BEGIN
MERGE INTO table6 tgt
USING (SELECT key1,
key2,
key3,
key4,
(SELECT MAX(amount)
FROM table3 t3
WHERE t3.key1 = mt.key1
AND t3.key2 = mt.key2
AND t3.key3 = mt.key3
AND t3.key4 = mt.key4) place_holder1
FROM mytable mt
WHERE myflag = 4) src
ON (tgt.key1 = src.key1
AND tgt.key2 = src.key2
AND tgt.key3 = src.key3
AND tgt.key4 = src.key4)
WHEN MATCHED THEN
UPDATE SET tgt.v = src.place_holder1;
INSERT INTO table7
(col1,
col2,
col3)
SELECT place_holder3 * mt.p2 val3,
SYSDATE dt,
somefunction(place_holder2, mt.p1) val2,
FROM (SELECT (SELECT param
FROM paramtable
WHERE param_id = 1) p1,
(SELECT param
FROM paramtable
WHERE param_id = 2) p2,
(SELECT amount
FROM table4
WHERE t4.key1 = mt.key1
AND t4.key2 = mt.key2
AND t4.key3 = mt.key3
AND t4.key4 = mt.key4) place_holder2,
(SELECT amount
FROM table5
WHERE t5.key1 = mt.key1
AND t5.key2 = mt.key2
AND t5.key3 = mt.key3
AND t5.key4 = mt.key4)/10 place_holder3
FROM mytable mt
WHERE myflag = 4);
EXCEPTION
WHEN OTHERS THEN
ROLLBACK;
RAISE;
END;
/
N.B. this does assume that for each key1, key2, key3 and key4 in mytable there is a corresponding row in table3, table4 and table5. By doing scalar subqueries, if there isn't a row present in those tables, null will be returned, rather than an error being generated (no_data_found) in your current code. You'll have to amend the above procedure to generate an error if such a condition occurs.
The way I did this was to take the cursor sql and then add in the subsequent sql statements as scalar subqueries in the cursor's sql.
Then I noticed that the insert and update were using different values (the update used the place_holder1 value, the insert the place_holder2 and place_holder3 values (amended appropriately). Then it was just a matter of converting those into a MERGE statement to do the update, and an insert statement.
Related
The below block tries to update a large table t1 with data found in table t2.It seems fine when i update for comment code that has 500 records but takes 30 minutes to update more than 1000 records. I tried the bulk collect update and index on comment code there is not much of time difference.
DECLARE
lv_row_count NUMBER(9) := 0;
lv_total_count NUMBER(9) := 0;
lv_commit_cnt SIMPLE_INTEGER:=0;
BEGIN
FOR rec in
(SELECT
a.t1_id,
a.t1_orig_code,
t2_orig_code,
a.t1_comment_code,
t2_code,
a.t1_restrict_update_ind,
t2_restrict_update_ind,
a.t1_data_origin,
t2_data_origin,
a.t1_purge_ind,
t2_purge_ind,
a.t1_created_date,
a.rowid
FROM t1 a
JOIN t2 ON t2_code = a.t1_comment_code
WHERE a.t1_comment_code in ('A','B','C','C1','D3')
AND ( a.t1_orig_code != t2_orig_code OR a.t1_restrict_update_ind !=t2_restrict_update_ind
OR a.t1_data_origin != t2_data_origin OR a.t1_purge_ind != t2_purge_ind)
)
LOOP
lv_total_count := lv_total_count + 1;
UPDATE t1
SET t1_ORIG_CODE= rec.t2_orig_code
t1_RESTRICT_UPDATE_IND = 'Y',,
t1_DATA_ORIGIN = rec.t2_data_origin,
t1_PURGE_IND =rec.t2_purge_ind
WHERE t1.rowid =rec.rowid ;
lv_commit_cnt:=lv_commit_cnt+1;
IF MOD(lv_commit_cnt,lv_limit)=0 THEN
lv_commit_cnt:=0;
COMMIT;
END IF;
dbms_output.put_line('a.t1_pidm -'||rec.t1_pidm ||
'a.t1_orig_code -'||rec.t1_orig_code ||'Updated');
END LOOP;
COMMIT;
dbms_output.put_line('Total_count- '||lv_total_count);
-- dbms_output.put_line('No record');
END;
Appreciate inputs on this .
No surprise it takes ages; row-by-row processing is slow-by-slow.
How about merge, instead?
merge into t1
using t2
on (t1.t1_comment_code = t2.t2_code)
when matched then update set
t1.t1_orig_code = t2.t2_orig_code,
t1.t1_restrict_update_ind = 'Y',
t1.t1_data_origin = t2.t2_data_origin,
t1.t1_purge_ind = t2.t2_purge_ind
where t1.t1_comment_code in ('A', 'B', 'C', 'C1', 'D3');
Just like that; no PL/SQL, no loop, no commit in the middle of the loop ... nothing. Just merge.
I have created one Stored Procedure. In that Stored Proc I want if the value of col1 & col2 match with employee then insert the unique record of the employee. If not found then match the value of col1, col2 & col3 with employee match then insert the value. If also not found while match all these column then insert the record by using another column.
Also one more thing that i want find list of values like emp_id by passing the another column value and if a single record can not match then make emp_id as NULL.
create or replace procedure sp_ex
AS
empID_in varchar2(10);
fname_in varchar2(20);
lname_in varchar2(30);
---------
type record is ref cursor return txt%rowtype; --Staging table
v_rc record;
rc rc%rowtype;
begin
open v_rc for select * from txt;
loop
fetch v_rc into rc;
exit when v_rc%notfound;
loop
select col1 from tbl1
Where EXISTS (select col1 from tbl1 where tbl1.col1 = rc.col1);
IF txt.col1 = rc.col1 AND txt.col2 = rc.col2 THEN
insert into main_table select distinct * from txt where txt.col2 = rc.col2;
ELSIF txt.col1 = rc.col1 AND txt.col2 = rc.col2 AND txt.col3 = rc.col3 THEN
insert into main_table select distinct * from txt where txt.col2 = rc.col2;
ELSE
insert into main_table select * from txt where txt.col4 = rc.col4;
end if;
end loop;
close v_rc;
end sp_ex;
I found an error while compile this Store Procedure PLS-00357: Table,View Or Sequence reference not allowed in this context. How to resolve this issue and how to insert value from staging to main table while using CASE or IF ELSIF statement. Could you please help me so that i can compile the Stored Proc.
Since I don't have your database to work with it's difficult to be 100% certain, but to my eye the line which reads
rc rc%rowtype;
should say
rc txt%rowtype;
You've defined the cursor v_rc as returning txt%rowtype, and your SQL statement used with this cursor is select * from txt, but that data type is at odds with the definition of rc. Thus, it appears you need to change rc as shown.
It also looks like the LOOP statement which comes immediately after exit when v_rc%notfound; should be removed, as there's nothing after that which would terminate that loop.
In addition, you have many references to columns in the txt table, e.g. IF txt.col1 = rc.col1. You can't refer to values in a table in this manner. I'm not quite sure what you're trying to do here so I can't really suggest anything.
Also, the statement
select col1 from tbl1
Where EXISTS (select col1 from tbl1 where tbl1.col1 = rc.col1);
is selecting a column from the database, but isn't putting it anywhere. This should be either a singleton SELECT (SELECT..INTO) or a cursor.
One more thing: you can't use distinct *. You need to use a column list with distinct.
Perhaps the following would be close to what you're trying to do:
create or replace procedure sp_ex
AS
begin
FOR rc IN (SELECT * FROM TXT)
LOOP
FOR t1 IN (SELECT *
FROM TBL1
WHERE TBL1.COL1 = rc.COL1)
LOOP
IF t1.COL1 = rc.COL1 AND
t1.COL2 = rc.COL2
THEN
insert into main_table
select *
from txt
where txt.col2 = rc.col2;
ELSIF t1.col1 = rc.col1 AND
t1.col2 = rc.col2 AND
t1.col3 = rc.col3
THEN
insert into main_table
select *
from txt
where txt.col2 = rc.col2;
ELSE
insert into main_table
select *
from txt
where txt.col4 = rc.col4;
END IF;
END LOOP; -- t1
END LOOP; -- rc
end sp_ex;
Best of luck.
I have nested for loop which iterates same table. In inner loop I update a column in same table. But in for loop condition I check that updated column and I need to check this column not in the beginning but dynamically, so my for loop iterations will maybe greatly decrease.
Am I doing this correct or is for statement will not see updated column?
declare
control number(1);
dup number(10);
res varchar2(5);--TRUE or FALSE
BEGIN
dup :=0;
control :=0;
FOR aRow IN (SELECT MI_PRINX, geoloc,durum, ROWID FROM ORAHAN where durum=0)
LOOP
FOR bRow IN (SELECT MI_PRINX, geoloc, ROWID FROM ORAHAN WHERE ROWID>aRow.ROWID AND durum=0)
LOOP
BEGIN
--dbms_output.put_line('aRow' || aRow.Mi_Prinx || ' bRow' || bRow.Mi_Prinx);
select SDO_GEOM.RELATE(aRow.geoloc,'anyinteract', bRow.Geoloc,0.02) into res from dual;
if (res='TRUE')
THEN
Insert INTO ORAHANCROSSES values (aRow.MI_PRINX,bRow.MI_PRINX);
UPDATE ORAHAN SET DURUM=1 where rowid=bRow.Rowid;
control :=1;
--dbms_output.put_line(' added');
END IF;
EXCEPTION
WHEN DUP_VAL_ON_INDEX
THEN
dup := dup+1;
--dbms_output.put_line('duplicate');
--continue;
END;
END LOOP;
IF(control =1)
THEN
UPDATE ORAHAN SET DURUM=1 WHERE rowid=aRow.Rowid;
END IF;
control :=0;
END LOOP;
dbms_output.put_line('duplicate: '||dup);
END ;
Note: I use oracle 11g and pl/sql developer
Sorry my english.
Yes, the FOR statement will not see the updated DURUM column because the FOR statement will see all data as they were when the query was started! This is called read consistency and Oracle accomplishes this by using the generated UNDO data. That means it'll have more and more work to do (==run slower) as your FOR loop advances and the base table is updated!
It also means that your implementation will eventually run into a ORA-01555: snapshot too old error when the UNDO tablespace is exhausted.
You'll be probably better off using a SQL MERGE statement which should also run much faster.
e.g.:
Merge Into ORAHANCROSSES C
Using (Select aROW.MI_PRINX aROW_MI_PRIX,
aROW.GEOLOC aROW_GEOLOC,
bROW.MI_PRINX bROW_MI_PRIX,
bROW.GEOLOC bROW_GEOLOC,
SDO_GEOM.RELATE(aRow.geoloc,'anyinteract', bRow.Geoloc,0.02) RES
From ORAHAN aROW,
ORAHAN bROW
Where aROW.ROWID < bROW.ROWID
) Q
On (C.MI_PRIX1 = Q.aROW_MI_PRIX
and C.MI_PRIX2 = Q.bROW_MI_PRIX)
When Matched Then
Delete Where Q.RES = 'FALSE'
When Not Matched Then
Insert Values (Q.aROW_MI_PRIX, Q.bROW_MI_PRIX)
Where Q.RES = 'TRUE'
;
I'm not sure what you're trying to accomplish by ROWID>aRow.ROWID though
To use a certain order (in this case MI_PRINX) use the following technique:
Merge Into ORAHANCROSSES C
Using (With D as (select T.*, ROWNUM RN from (select MI_PRINX, GEOLOC from ORAHAN order by MI_PRINX) T)
Select aROW.MI_PRINX aROW_MI_PRIX,
aROW.GEOLOC aROW_GEOLOC,
bROW.MI_PRINX bROW_MI_PRIX,
bROW.GEOLOC bROW_GEOLOC,
SDO_GEOM.RELATE(aRow.geoloc,'anyinteract', bRow.Geoloc,0.02) RES
From D aROW,
D bROW
Where aROW.RN < bROW.RN
) Q
On (C.MI_PRIX1 = Q.aROW_MI_PRIX
and C.MI_PRIX2 = Q.bROW_MI_PRIX)
When Matched Then
Delete Where Q.RES = 'FALSE'
When Not Matched Then
Insert Values (Q.aROW_MI_PRIX, Q.bROW_MI_PRIX)
Where Q.RES = 'TRUE'
;
In case the query is taking too long, you might select * from v$session_longops where seconds_remaining >0 to find out when it'll be finished.
I have the following pl/sql procedure:
create or replace procedure processData (a_date Date, r_offset Number, r_limit Number) as
begin
for r in (select * from (select a.*, ROWNUM rnum from (select* from TABLE1 T1 where T1.date=a_date) a
where rownum <= r_limit) where rnum >= r_offset) loop
if (/*some condition on column values */) then
/* insert into A*/
else
/*insert into B*/
end if;
end loop;
end;
as you can see it is made with for each.
I was now wondering about doing that with for-all statement: this would involve gathering data inside of table variables and then, after populating those variables, perform 2 forall statements: one for table A and one for table B.
Would this greatly improve my program's performances?
edit:
I just noticed that this program is, in general, very very slow! I tried processing 10k records and it almost took 30secs! Where could the general problem be?
Why just don't use plain SQL - conditional insert:
http://docs.oracle.com/cd/B19306_01/server.102/b14200/statements_9014.htm
I bet that it will be at least 100 times faster than any PL/SQL loop, even FORALL.
INSERT FIRST
WHEN /*some condition on column values */
THEN INTO /* insert into A */
ELSE INTO /*insert into B*/
select * from (select a.*, ROWNUM rnum from (select* from TABLE1 T1 where T1.date=a_date) a
where rownum <= r_limit) where rnum >= r_offset
Simple demo: http://sqlfiddle.com/#!4/d2019/1
I need to update a non-unique field. I have a table tbl:
create table tbl (A number(5));
Values in tbl: 1, 2, 2, 2 .. 2.
I need to replace all 2 with new non-unique values
New values: 1, 100, 101, 102, 103 ..
I wrote:
DECLARE
sql_stmt VARCHAR2(500);
cursor curs is
select A from tbl group by A having count(*)>1;
l_row curs%ROWTYPE;
i number(5);
new_mail VARCHAR2(20);
BEGIN
i:=100;
open curs;
loop
fetch curs into l_row;
exit when curs%notfound;
SQL_STMT := 'update tbl set a='||i||' where a='||l_row.A;
i:=i+1;
EXECUTE IMMEDIATE sql_stmt;
end loop;
close curs;
END;
/
But I got:
A
----------
1
100
...
100
What can be wrong? Why doesn't the loop work?
what about
update tbl
set a = 100 + rownum
where a in (
select a
from tbl
group by a
having count(*) > 1 )
the subquery finds duplicated A fields and the update gives them the unique identifier starting from 100. (you got other problems here like , what if id 100, 101.... already exists ).
first rule of PLSQL says that what ever you can do with SQL always do with SQL. writing straight up for loop cause allot of context switches between the sql and pl/sql engine. even if oracle automatically converts this to a bulk statement (10g<) it will still be faster with pure SQL.
Your cursor gets one row per unique value of A:
select A from tbl group by A having count(*)>1;
You need to get all the distinct rows that match those values. One way is to do this:
select a, r from (
select a, rowid as r, count(*) over (partition by a) as c
from tbl
) where c > 1;
... and then use the rowid values to do the update. I'm not sure why you're using dynamic SQL as it is not at all necessary, and you can simplify (IMO) the loop:
declare
i number(5);
begin
i:=100;
for l_row in (
select a, r from (
select a, rowid as r, count(*) over (partition by a) as c
from tbl
) where c > 1) loop
update tbl set a=i where rowid = l_row.r;
i:=i+1;
end loop;
end;
/
I've kept this as PL/SQL to show what was wrong with what you were attempting, but #haki is quite correct, you should (and can) do this in plain SQL if at all possible. Even if you need it to be PL/SQL because you're doing other work in the loop (as the new_mail field might suggest) then you might be able to still do a single update within the procedure, rather than one update per iteration around the loop.