I have table with 6 million records I am running archival script to delete around 5 million records.
My script1 will do the deletion but the DBA said my script1 getting into more buffer gets and
he recommended the approach which is script2. I am confused how come script2 is better than script1.
So please review the scripts and reply which is the best approach and why.
script1 :
PROCEDURE archival_charging_txn(p_no_hrs IN NUMBER,
p_error_code OUT NUMBER,
p_error_msg OUT VARCHAR2) IS
v_sysdate DATE := SYSDATE - p_no_hrs / 24;
TYPE t_txn_id IS TABLE OF scg_charging_txn.txn_id%TYPE INDEX BY PLS_INTEGER;
v_txn_id t_txn_id;
CURSOR c IS
SELECT txn_id FROM scg_charging_txn WHERE req_time < v_sysdate; /* non unique index */
BEGIN
OPEN c;
LOOP
FETCH c BULK COLLECT
INTO v_txn_id LIMIT 10000;
IF v_txn_id.COUNT > 0 THEN
FORALL i IN v_txn_id.FIRST .. v_txn_id.LAST
DELETE FROM scg_charging_txn WHERE txn_id = v_txn_id(i); /* Primary key based */
END IF;
COMMIT;
EXIT WHEN c%NOTFOUND;
END LOOP;
CLOSE c;
COMMIT;
p_error_code := 0;
EXCEPTION
WHEN OTHERS THEN
p_error_code := 1;
p_error_msg := substr(SQLERRM, 1, 200);
END archival_charging_txn;
script 2:
PROCEDURE archival_charging_txn_W(p_no_hrs IN NUMBER,
p_error_code OUT NUMBER,
p_error_msg OUT VARCHAR2) IS
BEGIN
DELETE FROM scg_charging_txn WHERE req_time < SYSDATE - p_no_hrs / 24;
COMMIT;
p_error_code := 0;
EXCEPTION
WHEN OTHERS THEN
p_error_code := 1;
p_error_msg := substr(SQLERRM, 1, 200);
END archival_charging_txn_W;
the 1st script reads the table entries using the cursor, therefore more buffer gets, while the 2nd script just deletes the table entries.
I would prefer the 2nd script. in case you get troubles with locking the table for too log time, make a LOOP UNTIL SQL%NOTFOUND with DELETE ... AND ROWNUM <= 10000; COMMIT; inside.
If you're deleting 5 million of 6 million rows, performance will suck.
You're better off doing CTAS.
Something like:
create table new_scg_charging_txn nologging as select * from scg_charging_txn WHERE req_time >= SYSDATE - p_no_hrs / 24;
If downtime is unacceptable, you may be able to do something similar, but wrap it with DBMS_REDEFINITION.
Related
I have written a package for building a reporting table. The simplified code for the function I am testing follows:
function do_build return integer is
V_RESULT PLS_INTEGER := 0;
cursor all_entities is
select e.id_number
from entity e
;
BEGIN
c_count := 0; -- this variable is declared at the package level outside of this function
for rec in all_entities LOOP
BEGIN
insert into reporting (
select *
from table(get_report_data(rec.id_number))
);
c_count := c_count + 1;
if MOD(c_count, 1000) = 0 Then
-- record status to table
commit;
end if;
EXCEPTION
WHEN OTHERS THEN
-- record exception to table
END;
END LOOP;
return V_RESULT;
END;
A little background: get_report_data is a function that returns a dataset with all of the input entity's reporting data.
About 1000 records out of 1 million are missing from the "reporting" table when the build completes. No exceptions are thrown and other than the missing records, everything appears to have been successful (function returns 0 to caller).
When I run the get_report_data for the entity records that do not have their reporting data recorded, the records show up fine. In fact, I can do an adhoc "insert into reporting (select * from table(get_reporting_data(missing_id))" and the information will be inserted.
Why would these records be skipped/fail to insert? Should I be looping a different way? Any better way to do it?
You're only committing every 1000 rows. You're not committing the last batch. Add a commit after the END LOOP;
BEGIN
c_count := 0; -- this variable is declared at the package level outside of this function
for rec in all_entities LOOP
BEGIN
insert into reporting (
select *
from table(get_report_data(rec.id_number))
);
c_count := c_count + 1;
if MOD(c_count, 1000) = 0 Then
-- record status to table
commit;
end if;
EXCEPTION
WHEN OTHERS THEN
-- record exception to table
END;
END LOOP;
COMMIT; -- <-- Add this commit to pick up last few records
return V_RESULT;
END;
Can this be a concurrency issue? If the records are committed in the ENTITY table while you loop is running they won't be processed.
BTW: Using WHEN OTHERS in this way is asking for trouble.
BTW2: Why not simply use:
INSERT INTO reporting
SELECT rep.*
FROM entity e
CROSS JOIN table(get_report_data(e.id_number)) rep;
I need to 1 billion data which 10 years before records from a table tblmail , for that I have created the below procedure.
I am doing through batch size.
CREATE OR REPLACE PROCEDURE PURGE_Data AS
batch_size INTEGER := 1000;
pvc_procedure_name CONSTANT VARCHAR2(50) := 'Purge_data';
pvc_info_message_num CONSTANT NUMBER := 1;
pvc_error_message_type CONSTANT VARCHAR2(5) := 'ERROR';
v_message schema_mc.db_msg_log.message%TYPE;
v_msg_num schema_mc.db_msg_log.msg_num%TYPE;
/*
Purpose: Provide stored procedures to be used to purge unwanted archives.
*/
BEGIN
Delete from tblmail where createdate_dts < (SYSDATE - INTERVAL '10' YEAR) and ROWNUM <= batch_size;
COMMIT;
EXCEPTION
WHEN OTHERS THEN
ROLLBACK;
v_msg_num := SQLCODE;
v_message := 'Error deleting from tblmail table';
INSERT INTO error_log
(date, num, type, source, mail)
VALUES
(systimestamp, v_msg_num, pvc_error_message_type,pvc_procedure_name, v_message);
COMMIT;
END;
Do I need to use bulk collect and delete? What is the best way to do this?
As always in computing it depends. Provided that you have an index on createdate_dts your procedure should work, but how do you know when to stop calling it? I tend to use a loop:
loop
delete /*+ first_rows */ from tblmail where createdate_dts <
(SYSDATE - INTERVAL '10' YEAR) and ROWNUM <= batch_size;
v_rows := SQL%ROWCOUNT;
commit;
exit when v_rows = 0;
end loop;
You could also return the number of deleted records if you want to keep the loop outside of the procedure. Without an index on createdate_dts it may be cheaper to collect the primary keys for the rows to delete in one pass first and then loop over them, deleting batch size records per commit with bulk collect or something. However, when possible it is always nice to use a simple solution! You may want to experiment a bit in order to find the best batch size.
Please help me to solve this ORACLE PL/SQL problem.
I have the following code:
BEGIN
DECLARE
P_COMMIT_STEP NUMBER := 10000; -- Commit every 10000 record copied
V_QUERY VARCHAR2 (4000) := NULL;
MY_CURSOR SYS_REFCURSOR;
TYPE FETCH_ARRAY IS TABLE OF MY_TABLE_BACKUP%ROWTYPE;
S_ARRAY FETCH_ARRAY;
BEGIN
V_QUERY := 'SELECT * FROM MY_TABLE_BACKUP';
OPEN MY_CURSOR FOR V_QUERY;
LOOP
FETCH MY_CURSOR
BULK COLLECT INTO S_ARRAY
LIMIT P_COMMIT_STEP;
FORALL I IN 1 .. S_ARRAY.COUNT
INSERT INTO MY_TABLE_BIS /*+ APPEND */
VALUES S_ARRAY (I);
COMMIT;
EXIT WHEN MY_CURSOR%NOTFOUND;
END LOOP;
CLOSE MY_CURSOR;
COMMIT;
END;
END;
Since the commit step is 10000, the copy works for a multiple of 10000 record.
So, if the original table has 1000010 records, only 1000000 records will be copied.
Where is the error?
The code seems correct, in my opinion.
Thank you very much for considering my request.
As noted in this article, you shouldn't rely on %NOTFOUND with bulk collect and forall. Check how many rows were fetched:
LOOP
FETCH MY_CURSOR
BULK COLLECT INTO S_ARRAY
LIMIT P_COMMIT_STEP;
FORALL I IN 1 .. S_ARRAY.COUNT
INSERT INTO MY_TABLE_BIS /*+ APPEND */
VALUES S_ARRAY (I);
COMMIT;
EXIT WHEN S_ARRAY.COUNT < P_COMMIT_STEP;
END LOOP;
Your first thousand iterations will get 10000 rows, so the count will equal your limit for all of those, and it will continue. The next one will get only 10 rows, so it will exit after the forall.
It should still work with the %NOTFOUND check at the end of the loop - the article is really talking about it being an issue if you use it to exit before processing the partial batch - and the documentation shows that pattern; but in some circumstances it seems not to. Having said that, I can't reproduce the issue from your code in 11.2.0.3.
Incidentally, committing inside a loop is generally a bad idea unless you've made the block restartable.
HI small modification to your Forall loop
FORALL I IN S_ARRAY.first .. S_ARRAY.last
INSERT INTO MY_TABLE_BIS /*+ APPEND */
VALUES S_ARRAY (I);
I think it will work for you.
I have to write a dynamic sql cursor where there are several possibilities in which the select query will be generated. Hence I am chosing dynamic and I am Using DBMS_SQL package to dynamically create a cursor and dynamically fetch the data.
However , Result set is going to be huge . around 11GB (there are 2.4 million records and the select statement will be approx 80 cols long assumning about 50Byte varchar per column)
Hence I cannot open the cursor at once . I want to know if there is a feature wherein i can fetch the data from the curosr keeping the curosr open for Blocks of say 1000 records at time(I will have to do this dynamically)
Please find the code attached which only fetches and prints the value of the columns (one sample case ) I want to use bul collect here \
Thanks
---------------code sample--------------------------------------
--create or replace type TY_DIMDEAL AS TABLE OF VARCHAR2(50) ;
create or replace procedure TEST_PROC (po_recordset out sys_refcursor)
as
v_col_cnt INTEGER;
v_ind NUMBER;
rec_tab DBMS_SQL.desc_tab;
v_cursor NUMBER;
lvar_output number:=0;
lvar_output1 varchar2(100);
lvar_output3 varchar2(100);
lvar_output2 varchar2(100);
LVAR_TY_DIMDEAL TY_DIMDEAL;
lvarcol varchar2(100);
begin
--
LVAR_TY_DIMDEAL := TY_DIMDEAL();
lvar_output1 := '';
v_cursor := dbms_sql.open_cursor;
dbms_sql.parse(v_cursor, 'select to_char(Field1) , to_char(fiel2) , to_char(field3) from table,table2 ', dbms_sql.native);
dbms_sql.describe_columns(v_cursor, v_col_cnt, rec_tab);
FOR v_pos in 1..rec_tab.LAST LOOP
LVAR_TY_DIMDEAL.EXTEND();
DBMS_SQL.define_column( v_cursor, v_pos ,LVAR_TY_DIMDEAL(v_pos),20);
END LOOP;
-- DBMS_SQL.define_column( v_cursor, 1 ,lvar_output1,20);
--DBMS_SQL.define_column( v_cursor, 2 ,lvar_output2,20);
--DBMS_SQL.define_column( v_cursor, 3 ,lvar_output3,20);
v_ind := dbms_sql.execute( v_cursor );
LOOP
v_ind := DBMS_SQL.FETCH_ROWS( v_cursor );
EXIT WHEN v_ind = 0;
lvar_output := lvar_output+1;
dbms_output.put_line ('row number '||lvar_output) ;
FOR v_col_seq IN 1 .. rec_tab.COUNT LOOP
LVAR_TY_DIMDEAL(v_col_seq):= '';
DBMS_SQL.COLUMN_VALUE( v_cursor, v_col_seq,LVAR_TY_DIMDEAL(v_col_seq));
dbms_output.put_line (LVAR_TY_DIMDEAL(v_col_seq));
END LOOP;
END LOOP;
end TEST_PROC;
Fetching data from a cursor in blocks of reasonable size, while keeping the cursor open, is one of PL/SQL Best Practices.
The above document (see Code 38 item) sketches an approach for when the select list is not known until runtime. Basically:
Define an appropriate type to fetch results into. Let's assume that all the returned columns will by of of type VARCHAR2:
-- inside DECLARE
Ty_FetchResults IS TABLE OF DBMS_SQL.VARCHAR2_TABLE;
lvar_results Ty_FetchResults;
Before each call to DBMS_SQL.FETCH_ROWS, call DBMS_SQL.DEFINE_ARRAY to enable batch fetching.
Call DBMS_SQL.FETCH_ROWS to fetch 1000 rows from the cursor.
Call DBMS_SQL.COLUMN_VALUE to copy the fetched data into your result array.
Process the results, record by record, in a FOR loop. Don't worry about the number of fetched records: if there are records to process, the FOR loop will run correctly; if the result array is empty, the FOR loop will not run.
Exit from the loop when the number of fetched records is less than the expected size.
Remember to DBMS_SQL.CLOSE the cursor.
Your loop body should look like this:
LOOP
FOR j IN 1..v_col_cnt LOOP
DBMS_SQL.DEFINE_ARRAY(v_cursor, j, lvar_results(j), 1000, 1);
END LOOP;
v_ind := DBMS_SQL.FETCH_ROWS(v_cursor);
FOR j IN 1..v_col_cnt LOOP
lvar_results(j).DELETE;
DBMS_SQL.COLUMN_VALUE(v_cursor, j, lvar_results(j));
END LOOP;
-- process the results, record by record
FOR i IN 1..lvar_results(1).COUNT LOOP
-- process a single record...
-- your logic goes here
END LOOP;
EXIT WHEN lvar_results(1).COUNT < 1000;
END LOOP;
-- don't forget: DBMS_CLOSE(v_cursor);
See also Doing SQL from PL/SQL: Best and Worst Practices.
LIMIT CLAUSE CAN COME TO RESCUE!
PL/SQL collections are essentially arrays in memory, so massive
collections can have a detrimental effect on system performance due to
the amount of memory they require. In some situations, it may be
necessary to split the data being processed into chunks to make the
code more memory-friendly. This “chunking” can be achieved using the
LIMIT clause of the BULK COLLECT syntax.
YOU CAN USE LIMIT CLAUSE AFTER BULK COLLECT INTO CLAUSE TO LIMIT YOUR RS.
AFTER YOU EXCEED TO LIMIT YOU CAN FETCH REMAINING ROWS.
SEE THIS ARTICLE
http://www.dba-oracle.com/plsql/t_plsql_limit_clause.htm
Here's my cursor:
CURSOR C1 IS SELECT * FROM MY_TABLE WHERE SALARY < 50000 FOR UPDATE;
I immediately open the cursor in order to lock these records for the duration of my procedure.
I want to raise an application error in the event that there are < 2 records in my cursor. Using the C1%ROWCOUNT property fails because it only counts the number which have been fetched thus far.
What is the best pattern for this use case? Do I need to create a dummy MY_TABLE%ROWTYPE variable and then loop through the cursor to fetch them out and keep a count, or is there a simpler way? If this is the way to do it, will fetching all rows in my cursor implicitly close it, thus unlocking those rows, or will it stay open until I explicitly close it even if I've fetched them all?
I need to make sure the cursor stays open for a variety of other tasks beyond this count.
NB: i just reread you question.. and you want to fail if there is ONLY 1 record..
i'll post a new update in a moment..
lets start here..
From Oracle® Database PL/SQL User's Guide and Reference
10g Release 2 (10.2)
Part Number B14261-01
reference
All rows are locked when you open the cursor, not as they are fetched. The rows are unlocked when you commit or roll back the transaction. Since the rows are no longer locked, you cannot fetch from a FOR UPDATE cursor after a commit.
so you do not need to worry about the records unlocking.
so try this..
declare
CURSOR mytable_cur IS SELECT * FROM MY_TABLE WHERE SALARY < 50000 FOR UPDATE;
TYPE mytable_tt IS TABLE OF mytable_cur %ROWTYPE
INDEX BY PLS_INTEGER;
l_my_table_recs mytable_tt;
l_totalcount NUMBER;
begin
OPEN mytable_cur ;
l_totalcount := 0;
LOOP
FETCH mytable_cur
BULK COLLECT INTO l_my_table_recs LIMIT 100;
l_totalcount := l_totalcount + NVL(l_my_table_recs.COUNT,0);
--this is the check for only 1 row..
EXIT WHEN l_totalcount < 2;
FOR indx IN 1 .. l_my_table_recs.COUNT
LOOP
--process each record.. via l_my_table_recs (indx)
END LOOP;
EXIT WHEN mytable_cur%NOTFOUND;
END LOOP;
CLOSE mytable_cur ;
end;
ALTERNATE ANSWER
I read you answer backwards to start and thought you wanted to exit if there was MORE then 1 row.. not exactly one.. so here is my previous answer.
2 simple ways to check for ONLY 1 record.
Option 1 - Explicit Fetchs
declare
CURSOR C1 IS SELECT * FROM MY_TABLE WHERE SALARY < 50000 FOR UPDATE;
l_my_table_rec C1%rowtype;
l_my_table_rec2 C1%rowtype;
begin
open C1;
fetch c1 into l_my_table_rec;
if c1%NOTFOUND then
--no data found
end if;
fetch c1 into l_my_table_rec2;
if c1%FOUND THEN
--i have more then 1 row
end if;
close c1;
-- processing logic
end;
I hope you get the idea.
Option 2 - Exception Catching
declare
CURSOR C1 IS SELECT * FROM MY_TABLE WHERE SALARY < 50000 FOR UPDATE;
l_my_table_rec C1%rowtype;
begin
begin
select *
from my_table
into l_my_table_rec
where salary < 50000
for update;
exception
when too_many_rows then
-- handle the exception where more than one row is returned
when no_data_found then
-- handle the exception where no rows are returned
when others then raise;
end;
-- processing logic
end;
Additionally
Remember: with an explicit cursor.. you can %TYPE your variable off the cursor record rather then the original table.
this is especially useful when you have joins in your query.
Also, rememebr you can update the rows in the table with an
UPDATE table_name
SET set_clause
WHERE CURRENT OF cursor_name;
type statement, but I that will only work if you haven't 'fetched' the 2nd row..
for some more information about cursor FOR loops.. try
Here
If you're looking to fail whenver you have more than 1 row returned, try this:
declare
l_my_table_rec my_table%rowtype;
begin
begin
select *
from my_table
into l_my_table_rec
where salary < 50000
for update;
exception
when too_many_rows then
-- handle the exception where more than one row is returned
when no_data_found then
-- handle the exception where no rows are returned
when others then raise;
end;
-- processing logic
end;
If this is the way to do it, will
fetching all rows in my cursor
implicitly close it, thus unlocking
those rows
The locks will be present for the duration of the transaction (ie until you do a commit or rollback) irrespective of when (or whether) you close the cursor.
I'd go for
declare
CURSOR C1 IS SELECT * FROM MY_TABLE WHERE SALARY < 50000 FOR UPDATE;;
v_1 c1%rowtype;
v_cnt number;
begin
open c_1;
select count(*) into v_cnt FROM MY_TABLE WHERE SALARY < 50000 and rownum < 3;
if v_cnt < 2 then
raise_application_error(-20001,'...');
end if;
--other processing
close c_1;
end;
There's a very small chance that, between the time the cursor is opened (locking rows) and the select count, someone inserts one or more rows into the table with a salary under 50000. In that case the application error would be raised but the cursor would only process the rows present when the cursor was opened. If that is a worry, at the end do another check on c_1%rowcount and, if that problem was experienced, you'd need to rollback to a savepoint.
Create a savepoint before you iterate through the cursor and then use a partial rollback when you find there are < 2 records returned.
You can start transaction and check if SELECT COUNT(*) MY_TABLE WHERE SALARY < 50000 greater than 1.