I have a table that is 5 GB, now I was trying to delete like below:
delete from tablename
where to_char(screatetime,'yyyy-mm-dd') <'2009-06-01'
But it's running long and no response. Meanwhile I tried to check if anybody is blocking with this below:
select l1.sid, ' IS BLOCKING ', l2.sid
from v$lock l1, v$lock l2
where l1.block =1 and l2.request > 0
and l1.id1=l2.id1
and l1.id2=l2.id2
But I didn't find any blocking also.
How can I delete this large data without any problem?
5GB is not a useful measurement of table size. The total number of rows matters. The number of rows you are going to delete as a proportion of the total matters. The average length of the row matters.
If the proportion of the rows to be deleted is tiny it may be worth your while creating an index on screatetime which you will drop afterwards. This may mean your entire operation takes longer, but crucially, it will reduce the time it takes for you to delete the rows.
On the other hand, if you are deleting a large chunk of rows you might find it better to
Create a copy of the table using
'create table t1_copy as select * from t1
where screatedate >= to_date('2009-06-01','yyyy-mm-dd')`
Swap the tables using the rename command.
Re-apply constraints, indexs to the new T1.
Another thing to bear in mind is that deletions eat more UNDO than other transactions, because they take more information to rollback. So if your records are long and/or numerous then your DBA may need to check the UNDO tablespace (or rollback segs if you're still using them).
Finally, have you done any investigation to see where the time is actually going? DELETE statements are just another query, and they can be tackled using the normal panoply of tuning tricks.
Use a query condition to export necessary rows
Truncate table
Import rows
If there is an index on screatetime your query may not be using it. Change your statement so that your where clause can use the index.
delete from tablename where screatetime < to_date('2009-06-01','yyyy-mm-dd')
It runs MUCH faster when you lock the table first. Also change the where clause, as suggested by Rene.
LOCK TABLE tablename IN EXCLUSIVE MODE;
DELETE FROM tablename
where screatetime < to_date('2009-06-01','yyyy-mm-dd');
EDIT: If the table cannot be locked, because it is constantly accessed, you can choose the salami tactic to delete those rows:
BEGIN
LOOP
DELETE FROM tablename
WHERE screatetime < to_date('2009-06-01','yyyy-mm-dd')
AND ROWNUM<=10000;
EXIT WHEN SQL%ROWCOUNT=0;
COMMIT;
END LOOP;
END;
Overall, this will be slower, but it wont burst your rollback segment and you can see the progress in another session (i.e. the number of rows in tablename goes down). And if you have to kill it for some reason, rollback won't take forever and you haven't lost all work done so far.
Related
I have to update table which has around 93 mln records, at the beginning DB updated 10 k records per 5 seconds, now after around 60 mln updated records, update next 10k records take 30-60 s, don't know why I have to update columns which are null.
I use loop with commit each 10 k records:
LOOP
UPDATE TABLE
SET DATE_COLUMN = v_hist_date
WHERE DATE_COLUMN IS NULL
AND ROWNUM <= c_commit_limit
AND NOT_REMOVED IS NULL;
EXIT WHEN SQL%ROWCOUNT = 0;
COMMIT;
END LOOP;
Have you any ideas why it slow down so much and how is possible to speed up this update ?
Updates are queries too. You haven't posted an explain plan but given you are filtering on columns which are null it seems probable that your statement is executing a Full Table Scan. That certainly fits the behaviour you describe.
What happens is this. The first loop the FTS finds 10000 rows which fit the WHERE criteria almost immediately. Then you exit the loop and start again. This time the FTS reads the same blocks again, including the ones it updated in the previous iteration before it finds the next 10000 rows it can update. And so on. Each loop takes longer because the full table scan has to read more of the table for each loop.
This is one of the penalties of randomly committing inside a loop. It may be too late for you now, but a better approach would be to track an indexed column such as a primary key. Using such a tracking key will allow an index scan to skip past the rows you have already visited.
I am copying a lot of data from one Table to another in a procedure while using cursors to iterate over the data of one table, holding them in an array and befilling the other afterwards in a limited batchsize.
I do realize that there are better ways to do this but i devloped it this way because of language restrictions. To my Code:
PROCEDURE copy_tableA_into_tableB IS
TYPE tableA_array IS TABLE OF tableA%ROWTYPE;
tableA_initialized_array TableIwantToCopyFrom_array;
CURSOR table_a_cursor IS
SELECT *
FROM TABLE_A; --get all data from Table_A
BEGIN
OPEN table_a_cursor;
LOOP
FETCH table_a_cursor BULK COLLECT
INTO test_copy LIMIT 10000;
FORALL i IN 1 .. table_a_cursor.COUNT
INSERT INTO TABLE_B
VALUES tableA_initialized_array (i);
COMMIT;
EXIT WHEN table_a_cursor%NOTFOUND;
END LOOP;
CLOSE table_a_cursor;
END copy_tableA_into_tableB;
This Code works just fine by just one weird thing that is happening,
when i execute it a couple of times with a bulk of data like 1.5 million table rows the UNDO tablespace is getting bigger and bigger and even though the procedure is done its still allocated by my procedure. Eventually my UNDO tablespace is full and i get an Exception. In fact i can only drop my UNDO tablespace and rebuild it to get it unoccupied and emptry again.
As you can clearly see i am commiting actually every time i got though the array so why would the UNDO table space even still be allocated after the transaction is done?
I am not an oracle expert to understand the underlying concept but i thought my cursor is closed and deallocated when hes closed, so i don't think that hes the culprit, i am using Oracle Version 11g if this is of any concern.
i expected that when i am done with the Procedure that my UNDO table space is deallocated again and i am checking the undo table space that is still left untouched
EDIT for Questions unanswered:
I am checking how much UNDO tablespace i have left the numbers of data to occupation added up and i am the only one running programms,
Exception: ORA-01555: snapshot too old: rollback segment number with name "" too small
I am testing the procedure in the PL/SQL developer under a stored procedure test
I am not reseting anything just emptying the tables that i wanna copy into with a truncate.
First off, I don't understand what you mean by "language restrictions". Why would that prevent you from doing a straight insert?
Secondly, you're commiting across a loop - seriously, a bad, bad idea. You could find yourself getting snapshot too old errors.
Thirdly, for your answer, Oracle doesn't immediately free up space in UNDO, once you've finished with it - it's marked as no longer needed (something that you do with that commit, hence why fetching across a loop is a bad idea!) and if another session needs that space then it's overwritten.
Tom Kyte, as ever, says it best
Is it not because the opening statement is:
CURSOR table_a_cursor IS
SELECT *
FROM TABLE_A; --get all data from Table_A
Therefore undo is generated so that this query can succeed. The lock on these rows isn't released until after it has completed (by which time the UNDO tablespace is full).
Something like this should work: (some modification to the loop may be required - I personally would check against 'x' processed before commited rather than this method.
PROCEDURE Copy_tableA_into_tableB AS
l_count integer
BEGIN
-- Count rows in tableA
l_count_total := 'select count(*) from table A';
EXECUTE IMMEDIATE l_sql_regexp_count
INTO l_pancount;
WHILE l_count > 0
LOOP
FETCH table_a_cursor BULK COLLECT
INTO cdplzstb_copy_batch LIMIT 10000;
FORALL i IN 1 .. table_a_cursor.COUNT
INSERT INTO TABLE_B
VALUES tableA_initialized_array (i);
COMMIT;
l_count:=l_count-1;
END LOOP;
END Copy_tableA_into_tableB;
Oracle will not reuse space while there is an active transaction there.
that explain why your tablespace completely occupied
More info in this Undo tablespace keeps growing
If you want to know how much space is occuipied
select tablespace_name,sum(bytes) from dba_segments group by tablespace_name;
this way you can find the actual size
select tablespace_name,sum(bytes) from dba_data_files group by tablespace_name;
I am inserting millions of records in a table
In other session I want to know the status of the same that is how many records have been processed,
No commit is issued in first session
Query v$session_longops. Not 100% but gives a good sense of the session state.
If you have oracle 11g i propose usage of DBMS_PARALLEL_EXECUTE package which allows you to create task chunks and monitoring the process flow (in your case number processed rows). I found tutorial resolving similar case to your situation: http://www.oraclefrontovik.com/2013/05/getting-started-with-dbms_parallel_execute/
Create a procedure with pragma AUTONOMOUS_TRANSACTION which update your counters in another table. Other sessions will see the counters but your main session still not commit.
You may update the counter when you reach 1000 records or how frequently you need it. Don't forget to update the counters when you rollback in main session.
example of procedure
If you're doing inserts in a loop try this construct:
FOR cur IN (SELECT rownum rn, something ... FROM something_else ...)
LOOP
INSERT INTO somewhere ...
dbms_application_info.set_module('Doing inserts', cur.rn || ' rows done');
END LOOP;
Then check module and action columns in v$session view.
First, find SQL_ID of Your session in GV$SQL_MONITOR. Then use it to find Your query's progress and details:
SELECT SQL_ID, status, PROCESS_NAME, FIRST_REFRESH_TIME, LAST_CHANGE_TIME
,PLAN_LINE_ID,PLAN_DEPTH,LPAD(' ',PLAN_DEPTH)||PLAN_OPERATION AS OPER
,PLAN_OPTIONS, PLAN_OBJECT_NAME, PLAN_OBJECT_TYPE
,OUTPUT_ROWS
FROM GV$SQL_PLAN_MONITOR
WHERE SQL_ID IN ( '0u063q75nbjt7' ) -- Your SQL_ID
order by sql_id,process_name,plan_line_id;
Once You found SQL_ID of Your query You can monitor progress in (G)V$SQL_PLAN_MONITOR and processed rows in OUTPUT_ROWS column.
The status can sometimes be derived from the table's segment size.
This is usually not the best approach. But often times it is too late to add logging, and tools like v$session_longops and SQL Monitoring frequently fail to provide accurate estimates, or any estimates at all.
Estimating by the segment size can be very tricky. You must be able to estimate the final size, perhaps based on another environment. Progress may not be linear; some operations may take an hour, and then writing to the table takes the last hour. Direct-path writes will not write to the table's segment initially, instead there will be a temporary segment with an unusual name.
Start with a query like this:
select sum(bytes)/1024/1024 mb
from dba_segments
where segment_name = '[table name]';
PL SQL moves older versions of data from a transaction table to a history table of same structure and archive for a certain period -
for each record
insert into tab_hist (select older_versions of current row);
delete from tab (select older_versions of current row);
END
ps: earlier we were not archiving(no insert) - but after adding the insert it has doubled the run time - so can we accomplish insert and delete with a single select statement? as there is large data to be processed and across multiple table
This is a batch operation, right? In which case you should avoid Row By Row and use set processing. SQL is all about The Joy Of Sets.
Oracle has fantastic bulk SQL processing capabilities. The pseudo code you paosted would look something like this:
declare
cursor c_oldrecs is
select * from your_table
where criterion between some_date and some_other_date;
type rec_nt is table of your_table%rowtype;
oldrecs_coll rec_nt;
begin
open c_oldrecs;
loop
fetch c_oldrecs into oldrecs_coll limit 1000;
exit when oldrecs_coll.count() = 0;
forall i in oldrecs_coll.first() oldrecs_coll.last()
insert into your_table_hist
values oldrecs_coll(i);
forall i in oldrecs_coll.first() oldrecs_coll.last()
delete from your_table
where pk_col = oldrecs_coll(i).pk_col;
end loop;
end;
/
This bulk processing is faster because it sends one thousand statements to the database at a time, instead of switching between PL/SQL and SQL one thousand times. The LIMIT 1000 clause is there to prevent a really huge selection blowing the PGA. This safeguard may not be necessary in your case, or perhaps you can work with a higher value.
I think your current implementation is wrong. It is better to keep only the current version in the live table, and to keep all the historical versions in a separate table from the off. Use triggers to maintain the history as part of every transaction.
It may be that the slowness you are seeing is due to the logic that selects which rows are to be moved. If so, you might get better results by doing the select once to get the rowids into a nested table in memory, then doing the insert and the delete based on that list; or alternatively, driving your loop with a query that selects the rows to be moved.
You might instead consider creating a trigger on insert that will move the existing rows that "match" the row being inserted. This will slow down the inserts somewhat, but would mean you don't need any process to move the old rows in bulk.
If you are on Enterprise edition with the partitioning option, look at partition exchange.
As simple as this
CREATE BACKUP_TAB AS SELECT * FROM TAB
If you are deleting a lot of rows you will be hitting your undo tablespace and a delete which deletes say 100k rows can cause performance issues. You are better of deleting by batch say 5k rows at a time and committing.
BEGIN
-- Where condition on insert and delete must be the same
loop
INSERT INTO BACKUP_TAB SELECT * FROM TAB WHERE 1=1 and rownum < 5000; --Your condition here
exit when SQL%rowcount < 4999;
commit;
end loop;
loop
DELETE FROM TAB
where 1=1--Your condition here
and rownum < 5000;
exit when SQL%rowcount < 4999;
commit;
end loop;
commit;
END;
We have an Oracle database, and the customer account table has about a million rows. Over the years, we've built four different UIs (two in Oracle Forms, two in .Net), all of which remain in use. We have a number of background tasks (both persistent and scheduled) as well.
Something is occasionally holding a long lock (say, more than 30 seconds) on a row in the account table, which causes one of the persistent background tasks to fail. The background task in question restarts itself once the update times out. We find out about it a few minutes after it happens, but by then the lock has been released.
We have reason to believe that it might be a misbehaving UI, but haven't been able to find a "smoking gun".
I've found some queries that list blocks, but that's for when you've got two jobs contending for a row. I want to know which rows have locks when there's not necessarily a second job trying to get a lock.
We're on 11g, but have been experiencing the problem since 8i.
Oracle's locking concept is quite different from that of the other systems.
When a row in Oracle gets locked, the record itself is updated with the new value (if any) and, in addition, a lock (which is essentially a pointer to transaction lock that resides in the rollback segment) is placed right into the record.
This means that locking a record in Oracle means updating the record's metadata and issuing a logical page write. For instance, you cannot do SELECT FOR UPDATE on a read only tablespace.
More than that, the records themselves are not updated after commit: instead, the rollback segment is updated.
This means that each record holds some information about the transaction that last updated it, even if the transaction itself has long since died. To find out if the transaction is alive or not (and, hence, if the record is alive or not), it is required to visit the rollback segment.
Oracle does not have a traditional lock manager, and this means that obtaining a list of all locks requires scanning all records in all objects. This would take too long.
You can obtain some special locks, like locked metadata objects (using v$locked_object), lock waits (using v$session) etc, but not the list of all locks on all objects in the database.
you can find the locked tables in Oracle by querying with following query
select
c.owner,
c.object_name,
c.object_type,
b.sid,
b.serial#,
b.status,
b.osuser,
b.machine
from
v$locked_object a ,
v$session b,
dba_objects c
where
b.sid = a.session_id
and
a.object_id = c.object_id;
Rather than locks, I suggest you look at long-running transactions, using v$transaction. From there you can join to v$session, which should give you an idea about the UI (try the program and machine columns) as well as the user.
Look at the dba_blockers, dba_waiters and dba_locks for locking. The names should be self explanatory.
You could create a job that runs, say, once a minute and logged the values in the dba_blockers and the current active sql_id for that session. (via v$session and v$sqlstats).
You may also want to look in v$sql_monitor. This will be default log all SQL that takes longer than 5 seconds. It is also visible on the "SQL Monitoring" page in Enterprise Manager.
The below PL/SQL block finds all locked rows in a table. The other answers only find the blocking session, finding the actual locked rows requires reading and testing each row.
(However, you probably do not need to run this code. If you're having a locking problem, it's usually easier to find the culprit using GV$SESSION.BLOCKING_SESSION and other related data dictionary views. Please try another approach before you run this abysmally slow code.)
First, let's create a sample table and some data. Run this in session #1.
--Sample schema.
create table test_locking(a number);
insert into test_locking values(1);
insert into test_locking values(2);
commit;
update test_locking set a = a+1 where a = 1;
In session #2, create a table to hold the locked ROWIDs.
--Create table to hold locked ROWIDs.
create table locked_rowids(the_rowid rowid);
--Remove old rows if table is already created:
--delete from locked_rowids;
--commit;
In session #2, run this PL/SQL block to read the entire table, probe each row, and store the locked ROWIDs. Be warned, this may be ridiculously slow. In your real version of this query, change both references to TEST_LOCKING to your own table.
--Save all locked ROWIDs from a table.
--WARNING: This PL/SQL block will be slow and will temporarily lock rows.
--You probably don't need this information - it's usually good enough to know
--what other sessions are locking a statement, which you can find in
--GV$SESSION.BLOCKING_SESSION.
declare
v_resource_busy exception;
pragma exception_init(v_resource_busy, -00054);
v_throwaway number;
type rowid_nt is table of rowid;
v_rowids rowid_nt := rowid_nt();
begin
--Loop through all the rows in the table.
for all_rows in
(
select rowid
from test_locking
) loop
--Try to look each row.
begin
select 1
into v_throwaway
from test_locking
where rowid = all_rows.rowid
for update nowait;
--If it doesn't lock, then record the ROWID.
exception when v_resource_busy then
v_rowids.extend;
v_rowids(v_rowids.count) := all_rows.rowid;
end;
rollback;
end loop;
--Display count:
dbms_output.put_line('Rows locked: '||v_rowids.count);
--Save all the ROWIDs.
--(Row-by-row because ROWID type is weird and doesn't work in types.)
for i in 1 .. v_rowids.count loop
insert into locked_rowids values(v_rowids(i));
end loop;
commit;
end;
/
Finally, we can view the locked rows by joining to the LOCKED_ROWIDS table.
--Display locked rows.
select *
from test_locking
where rowid in (select the_rowid from locked_rowids);
A
-
1
Given some table, you can find which rows are not locked with SELECT FOR UPDATESKIP LOCKED.
For example, this query will lock (and return) every unlocked row:
SELECT * FROM mytable FOR UPDATE SKIP LOCKED
References
Ask TOM "How to get ROWID for locked rows in oracle".