I run into an interesting and unexpected issue when processing records in Oracle (11g) using BULK COLLECT.
The following code was running great, processing through all million plus records with out an issue:
-- Define cursor
cursor My_Data_Cur Is
Select col1
,col2
from My_Table_1;
…
-- Open the cursor
open My_Data_Cur;
-- Loop through all the records in the cursor
loop
-- Read the first group of records
fetch My_Data_Cur
bulk collect into My_Data_Rec
limit 100;
-- Exit when there are no more records to process
Exit when My_Data_Rec.count = 0;
-- Loop through the records in the group
for idx in 1 .. My_Data_Rec.count
loop
… do work here to populate a records to be inserted into My_Table_2 …
end loop;
-- Insert the records into the second table
forall idx in 1 .. My_Data_Rec.count
insert into My_Table_2…;
-- Delete the records just processed from the source table
forall idx in 1 .. My_Data_Rec.count
delete from My_Table_1 …;
commit;
end loop;
Since at the end of processing each group of 100 records (limit 100) we are deleting the records just read and processed, I though it would be a good idea to add the “for update” syntax to the cursor definition so that another process couldn’t update any of the records between the time the data was read and the time the record is deleted.
So, the only thing in the code I changed was…
cursor My_Data_Cur
is
select col1
,col2
from My_Table_1
for update;
When I ran the PL/SQL package after this change, the job only processes 100 records and then terminates. I confirmed this change was causing the issue by removing the “for update” from the cursor and once again the package processed all of the records from the source table.
Any ideas why adding the “for update” clause would cause this change in behavior? Any suggestions on how to get around this issue? I’m going to try starting an exclusive transaction on the table at the beginning of the process, but this isn’t an idea solution because I really don’t want to lock the entire table which processing the data.
Thanks in advance for your help,
Grant
The problem is that you're trying to do a fetch across a commit.
When you open My_Data_Cur with the for update clause, Oracle has to lock every row in the My_Data_1 table before it can return any rows. When you commit, Oracle has to release all those locks (the locks Oracle creates do not span transactions). Since the cursor no longer has the locks that you requested, Oracle has to close the cursor since it can no longer satisfy the for update clause. The second fetch, therefore, must return 0 rows.
The most logical approach would almost always be to remove the commit and do the entire thing in a single transaction. If you really, really, really need separate transactions, you would need to open and close the cursor for every iteration of the loop. Most likely, you'd want to do something to restrict the cursor to only return 100 rows every time it is opened (i.e. a rownum <= 100 clause) so that you wouldn't incur the expense of visiting every row to place the lock and then every row other than the 100 that you processed and deleted to release the lock every time through the loop.
Adding to Justin's Explantion.
You should have seen the below error message.Not sure, if your Exception handler suppressed this.
And the message itself explains a Lot!
For this kind of Updates, it is better to create a shadow copy of the main table, and let the public synonym point to it. While some batch id, creates a private synonym to our main table and perform the batch operations, to keep it simpler for maintenance.
Error report -
ORA-01002: fetch out of sequence
ORA-06512: at line 7
01002. 00000 - "fetch out of sequence"
*Cause: This error means that a fetch has been attempted from a cursor
which is no longer valid. Note that a PL/SQL cursor loop
implicitly does fetches, and thus may also cause this error.
There are a number of possible causes for this error, including:
1) Fetching from a cursor after the last row has been retrieved
and the ORA-1403 error returned.
2) If the cursor has been opened with the FOR UPDATE clause,
fetching after a COMMIT has been issued will return the error.
3) Rebinding any placeholders in the SQL statement, then issuing
a fetch before reexecuting the statement.
*Action: 1) Do not issue a fetch statement after the last row has been
retrieved - there are no more rows to fetch.
2) Do not issue a COMMIT inside a fetch loop for a cursor
that has been opened FOR UPDATE.
3) Reexecute the statement after rebinding, then attempt to
fetch again.
Also, you can change you Logic by Using rowid
An Example for Docs:
DECLARE
-- if "FOR UPDATE OF salary" is included on following line, an error is raised
CURSOR c1 IS SELECT e.*,rowid FROM employees e;
emp_rec employees%ROWTYPE;
BEGIN
OPEN c1;
LOOP
FETCH c1 INTO emp_rec; -- FETCH fails on the second iteration with FOR UPDATE
EXIT WHEN c1%NOTFOUND;
IF emp_rec.employee_id = 105 THEN
UPDATE employees SET salary = salary * 1.05 WHERE rowid = emp_rec.rowid;
-- this mimics WHERE CURRENT OF c1
END IF;
COMMIT; -- releases locks
END LOOP;
END;
/
You have to fetch a record row by row!! update it using the ROWID AND COMMIT immediately
. And then proceed to the next row!
But by this, you have to give up the Bulk Binding option.
Related
I have the following cursor in a procedure :
procedure Run is
Cur Cursor is select * from table where condition;
R Cur%rowtype;
Open Cur;
loop
fetch Cur into R;
exit when Cur%notfound;
-- Run some time consuming operations here
something...
end loop;
Close Cur;
end;
This cursor is run a scheduled job.
Assume when running this cursor there are 100 rows that satisfy the where condition.
If, while the procedure is running, I have a new rows inserted in the table that satisfies the same where condition, Is there any way that cursor picks also these new row please ?
Thanks.
Cheers,
No.
The set of rows the cursor will return is determined at the time the cursor is opened. At that point, Oracle knows the current SCN (system change number) and will return the data as it existed at that point in time.
Depending on the nature of the problem, you could write a loop that just keeps asking for a single row that meets the criteria (assuming your time-consuming operation updates some data so that you know what needs to be processed). Something like
loop
begin
select some_id
into l_some_id
from your_table
where needs_processing = 'Y'
order by some_id
fetch first row only;
exception
when no_data_found
then
l_some_id := null;
end;
exit when l_some_id is null;
some_slow_operation( l_some_id );
end loop;
assuming that some_slow_operation changes the needs_processing flag to N. And assuming that you are using the default read committed transaction isolation level.
You can have commit inside loop so that select query fetches latest records from table in every iteration.
No, a cursor can't do that. The transactions are consistent and your cursor is a snapshot of the data you've extracted.
If you want consistent results you could either:
Lock the table so that there will be no changes,
Use other mechanism e.g. move the logic to a trigger, which will execute on each new piece of data that satisfies your conditions (and bring overhead too so very situational)
I have 5 rows in table. And some of the rows are locked in some sessions .
I don't want to generate any error, Just want to wait until any row will become free for further processing
I tired with nowait and skip locked:-
nowait , But there is a problem with nowait. query has written in cursor , when I used "nowait" under cursor , query will return null and control will go out with an error by saying- resource busy
I tried with skip locked with for update-But if table contain 5 rows and all
5 rows are locked then it is giving error.
CURSOR cur_name_test IS
SELECT def.id , def.name
FROM def_map def
WHERE def.id = In_id
FOR UPDATE skip locked;
Why don't use select for update only ? The below is being test locally in plsql developer.
in first sesssion I do the below
SELECT id , name
FROM ex_employee
FOR UPDATE;
in second session i run the below however it hang.
SET serveroutput ON size 2000
/
begin
declare curSOR cur_name_test IS
SELECT id , name
FROM ex_employee
WHERE id = 1
FOR UPDATE ;
begin
for i in cur_name_test loop
dbms_output.put_line('inside cursor');
end loop;
end;
end;
/
commit
/
when I commit in the first session , the lock will be released and the second session will do its work. i guess that you want , infinite wait.
However such locking mechanism (pessimistic locking) can lead to deadlocks if its not managed correctly and carefully ( first session waiting second session , and second session waiting first session).
As for the nowait its normal to have error resource busy, because you are saying for the query don't wait if there are locking. you can instead wait 30, which will wait 30 second then output error but thats not what you want(I guess).
As for skip locked, the select will skip the locked data for example you have 5 rows , and one of them is locked then the select will not read this row . thats why when all the data is locked its throwing error because nothing can be skipped. and I guess is not what you want in your scenario.
This sounds like you need to think about transaction control.
If you are doing work in a transaction then the implication is that that unit of work needs to complete in order to be valid.
What you are saying is that some of the work in my update transaction doesn't need to complete in order for the transaction to be committed.
Not only that but you have two transactions running at the same time performing operations against the same object. In itself that may be valid but
if it is then you really need to go back to the first sentence and think hard about transaction control and process flow and see if there's a way you can have the second transaction only attempt to update rows that aren't being updated in the first transaction.
Why we need WHERE CURRENT OF clause in Oracle PL/SQL? We all know that FETCH retrieves only one row at a time and hence FETCH is used in LOOP to process all the rows of a cursor. Then why do we exclusively need a WHERE CURRENT OF clause? We can anyhow lock the cursor rows using FOR UPDATE or FOR UPDATE OF.
Can rows be unlocked (which are locked by FOR UPDATE or FOR UPDATE OF) once we close the cursor? Or do we need to COMMIT or ROLLBACK the transaction to unlock the rows?
Have a look at this block:
DECLARE
CURSOR c1 IS
SELECT course_number, ROWID AS RID
FROM courses_tbl
FOR UPDATE;
begin
FOR aCourse IN c1 LOOP
UPDATE courses_tbl SET course_number = aCourse.course_number + 1
WHERE CURRENT OF c1;
UPDATE courses_tbl SET course_number = aCourse.course_number + 1
WHERE ROWID = aCourse.RID
end loop;
end;
The two UPDATE statements are equivalent, WHERE CURRENT OF ... is just a shortcut for WHERE ROWID = ..., you can use either of them.
Actually your question should be "Why we need FOR UPDATE ...?" The reason is, the ROWID may change by other operations, e.g. ALTER TABLE ... SHRINK SPACE, moving tablespace or big DML's. FOR UPDATE locks the row, i.e. ensures that ROWID does not change until you finished your transaction.
No, you can release the lock only by finishing the transaction, i.e. ROLLBACK or COMMIT
May be this will convince you ...
When we want to update or delete the cursor fetched row(s) from the
database, we don’t have to form a UPDATE or a DELETE statement with a
primary key mapping in its WHERE clause, instead, the WHERE CURRENT OF
clause comes in handy.
Source:
http://www.dba-oracle.com/t_adv_plsql_for_update_where_current.htm
You must commit or rollback your transaction to release locks.
PL SQL moves older versions of data from a transaction table to a history table of same structure and archive for a certain period -
for each record
insert into tab_hist (select older_versions of current row);
delete from tab (select older_versions of current row);
END
ps: earlier we were not archiving(no insert) - but after adding the insert it has doubled the run time - so can we accomplish insert and delete with a single select statement? as there is large data to be processed and across multiple table
This is a batch operation, right? In which case you should avoid Row By Row and use set processing. SQL is all about The Joy Of Sets.
Oracle has fantastic bulk SQL processing capabilities. The pseudo code you paosted would look something like this:
declare
cursor c_oldrecs is
select * from your_table
where criterion between some_date and some_other_date;
type rec_nt is table of your_table%rowtype;
oldrecs_coll rec_nt;
begin
open c_oldrecs;
loop
fetch c_oldrecs into oldrecs_coll limit 1000;
exit when oldrecs_coll.count() = 0;
forall i in oldrecs_coll.first() oldrecs_coll.last()
insert into your_table_hist
values oldrecs_coll(i);
forall i in oldrecs_coll.first() oldrecs_coll.last()
delete from your_table
where pk_col = oldrecs_coll(i).pk_col;
end loop;
end;
/
This bulk processing is faster because it sends one thousand statements to the database at a time, instead of switching between PL/SQL and SQL one thousand times. The LIMIT 1000 clause is there to prevent a really huge selection blowing the PGA. This safeguard may not be necessary in your case, or perhaps you can work with a higher value.
I think your current implementation is wrong. It is better to keep only the current version in the live table, and to keep all the historical versions in a separate table from the off. Use triggers to maintain the history as part of every transaction.
It may be that the slowness you are seeing is due to the logic that selects which rows are to be moved. If so, you might get better results by doing the select once to get the rowids into a nested table in memory, then doing the insert and the delete based on that list; or alternatively, driving your loop with a query that selects the rows to be moved.
You might instead consider creating a trigger on insert that will move the existing rows that "match" the row being inserted. This will slow down the inserts somewhat, but would mean you don't need any process to move the old rows in bulk.
If you are on Enterprise edition with the partitioning option, look at partition exchange.
As simple as this
CREATE BACKUP_TAB AS SELECT * FROM TAB
If you are deleting a lot of rows you will be hitting your undo tablespace and a delete which deletes say 100k rows can cause performance issues. You are better of deleting by batch say 5k rows at a time and committing.
BEGIN
-- Where condition on insert and delete must be the same
loop
INSERT INTO BACKUP_TAB SELECT * FROM TAB WHERE 1=1 and rownum < 5000; --Your condition here
exit when SQL%rowcount < 4999;
commit;
end loop;
loop
DELETE FROM TAB
where 1=1--Your condition here
and rownum < 5000;
exit when SQL%rowcount < 4999;
commit;
end loop;
commit;
END;
Using Oracle, is it possible to indicate which rows are currently locked (and which are not) when performing a select statement (I don't want to lock any rows, just be able to display which are locked)?
For example, a pseudo column that would return the lock/transaction against the row:
SELECT lockname FROM emp;
One thing you could do is this - although it is not terribly efficient and so I wouldn't want to do use it for large data sets. Create a row-level function to try and lock the row. If it fails, then the row is already locked
CREATE OR REPLACE FUNCTION is_row_locked (v_rowid ROWID, table_name VARCHAR2)
RETURN varchar2
IS
x NUMBER;
PRAGMA AUTONOMOUS_TRANSACTION;
BEGIN
EXECUTE IMMEDIATE 'Begin
Select 1 into :x from '
|| table_name
|| ' where rowid =:v_rowid for update nowait;
Exception
When Others Then
:x:=null;
End;'
USING OUT x, v_rowid;
-- now release the lock if we got it.
ROLLBACK;
IF x = 1
THEN
RETURN 'N';
ELSIF x IS NULL
THEN
RETURN 'Y';
END IF;
END;
/
And then you could
Select field1, field2, is_row_locked(rowid, 'MYTABLE') from mytable;
It will work, but it isn't pretty nor efficient.
Indeed, it has exactly one redeeming quality - it will work even if you don't have select privs on the various v$ tables required in the linked document. If you have the privs, though, definitely go the other route.
is it possible to indicate which rows are currently locked (and which are not) when performing a select statement
A SELECT statement will never lock any rows - unless you ask it to by using FOR UPDATE.
If you want to see locks that are held due to a SELECT ... FOR UPDATE (or a real update), you can query the v$lock system view.
See the link that OMG Pony posted for an example on how to use that view.
I think #Michael Broughton's answer is the only way that will always work. This is because V$LOCK is not accurate 100% of the time.
Sessions don't wait for a row, they wait for the end of the transaction that modified that row. Most of the time those two concepts are the same thing, but not when you start using savepoints.
For example:
Session 1 creates a savepoint and modifies a row.
Session 2 tries to modify that same
row, but sees session 1 already has that row,
and waits for session 1 to finish.
Session 1 rolls back to the
savepoint. This removes its entry
from the ITL but does not end the
transaction. Session 2 is still
waiting on session 1. According to
V$LOCK session 2 is still waiting on
that row, but that's not really true
because now session 3 can modify that
row. (And if session 1 executes a
commit or rollback, session 2 will
wait on session 3.)
Sorry if that's confusing. You may want to step through the link provided by OMG Ponies, and then try it again with savepoints.