for the archiving purpose i need to delete the data from a table but as delete will not free up space a lot of data is occupied .To save this i have figured out a solution to use shrink space command provided by oracle.
But this requires row movement to be enabled.
So my question are below:
1. is it a good idea to enable row movement for using shrink space command.
2. Can we just enable the row movement for running shrink space command and then disable it again
3. or we should leave row movement enabled and run shrink space command as and when required(say once a week).
Are you deleting all of the data on the table or just some part of it? If you are deleting all of it you could just truncate it, freeing up all of the space allocated for the tables and indexes in a very fast way:
truncate table t;
If you are not deleting of all it, the row movement approach should be ok (any of the 3 options) but you would have to test concurrent access to this table. Is there a chance someone else would try to update/insert this table at the same time of your maintenance? My guess is that this could be a problem.
So another approach could be partition the table based on your purge criteria. For example, if you will erase the data older than 3 months, you could have the table partitioned by month and delete only the partitions you wont't need anymore. This is in fact what partitions were made for, to easy maintenance of data.
Related
I needed to truncate and reload a table.
I learned that truncate needs stats gathering on the table as its successor process so the database gets the actual statistics, otherwise previous stats are not cleared by the truncate statement.
After doing these two operations (truncate and stats gathering on the empty table), ran the insert... but don't see new statistics in all_tab_statistics table for my table. Sample_size is still 0.
Why is that? Shouldn't have Oracle done the automatic stats gathering after the insert?
Do I need to rerun the stats or is it just fine considering the performance around this table (please note it's going to truncate and reload each time)?
Consider the following approach. It has the advantage of the table always being present.
Create an empty new table like the old one.
Load the data into the new table. This is the slowest step.
Do whatever cleanup you might need, such as refreshing the statistics.
RENAME tables to swap the new table in place. This step is fast enough so you won't notice.
I know it's a long time since I posted my question above. But recently, we again faced the similar situation and this time below steps worked towards a much better performance on a table with 800 million rows.
Take a backup of the original table.
Truncate the original table.
Gather stats on the truncated table, so that statistics show 0 in the DB. Us CASCADE=>TRUE in the command to also include indexes in the process.
Drop the indexes on the truncated table and Insert the required data from the backup table.
Recreate the indexes and gather stats again (ofcourse, with CASCADE=>TRUE; however recreation of the indexes should ideally have calculated the appropriate stats).
Drop the backup table if not needed.
I have created some tables in Greenplum, performing insert update and delete operation. Regularly I am also performing vacuum operation. I Found bloat in it. Found solution to remove bloat https://discuss.pivotal.io/hc/en-us/articles/206578327-What-are-the-different-option-to-remove-bloat-from-a-table
However, if I truncate the table and reinsert the data, it removes bloat. Is it good practice to truncate the data from the table?
If you are performing UPDATE and DELETE statements on a heap table (default storage) and running VACUUM regularly, you will get some bloat by design. Heap storage, which is similar to the default PostgreSQL storage mechanism, provides read consistency using Multi-Version Concurrency Control (MVCC).
When you UPDATE or DELETE a record, the old value is still in the table and is able to be read by transactions that are still inflight and started before you issued the UPDATE or DELETE command. This provides the read consistency to the table.
When you execute a VACUUM statement, the database will mark the stale rows as available to be overwritten. It doesn't shrink the files. It just marks rows so they can be overwritten. The next time you execute an INSERT or UPDATE, the stale rows are now able to be used for the new data.
So if you UPDATE or DELETE 10% of a table between running VACUUM, you will probably have about 10% bloat.
Greenplum also has Append-Optimized (AO) storage which doesn't use MVCC and uses a visibility map instead. The files are bit smaller too so you should get better performance. The stale rows are hidden with the visibility map and VACUUM won't do anything until you hit the gp_appendonly_compaction_threshold percentage. The default is 10%. When you have 10% bloat in an AO table and execute VACUUM, the table will automatically get rebuilt for you.
Append-Optimized is called "appendonly" for backwards compatibility reasons but it does allow UPDATE and DELETE. Here is an example of an AO table:
CREATE TABLE sales
(txn_id int, qty int, date date)
WITH (appendonly=true)
DISTRIBUTED BY (txn_id);
Instead of truncate it is better to use drop the table, create the table and then insert the data.
I am using basic compression in Oracle to archive seemingly unused tables as a first step to dropping them. I used these commands:
alter table table1 compress basic;
alter table table1 move;
The move invalidates the index. Does the invalid index still take up space? The index no longer shows up in a query of the USER_SEGMENTS table.
This would be useful to know whether I need to drop or rebuild and compress the index to save even more space.
ALTER TABLE ... MOVE marks the indexes built on that table UNUSABLE.
In earlier versions, UNUSABLE indexes still had their segments and the allocated space was not freed up. Starting from version 11.2, UNUSABLE index segments are automatically removed.
A non-existent segment consumes less space than a compressed index segment :)
If you want to use those indexes again, you have to rebuild them. Otherwise, just drop them.
it is said that we should always truncate a large table before dropping, it improves performance. Is it true?
IMO in general if you simply want to drop a table then DROP is appropriate. It will release space the same way as TRUNCATE would and it will have the advantage of being atomic (no query will have the opportunity to see the table "empty").
From 10g+, a dropped table won't be deleted immediately however: if there is sufficient space it will be put in the recycle bin. If you truncate a table first, no data will remain in the recycle bin. This may be why you have been told to truncate first (?).
In any case, if you want to bypass the recycle bin you could issue DROP TABLE your_table PURGE and this statement will be atomic.
It entirely depends if you want to be able to roll back if something goes wrong.
Deletion of data records the deletion against the transaction logs of the database until you commit the change.
Truncation removes all the data from the table without recording those logs, so there can be a significant performance improvement in doing this. Just be sure you know what you are doing, as there's no way back.
It may be a good idea in order to reset the high water mark.
We have a mature Oracle database application (in production for over 10 years), and during that time, we have been using scripts of our own devising to remove old data that is no longer needed. They work by issuing delete statements against the appropriate tables, in a loop with frequent commits, in order to avoid overloading the system with i/o or using too much undo space.
They work fine, for the most part. They run daily, and it takes about an hour to remove the oldest days worth of data from the system. The main concerns I have are the effects on tables and indexes that all this deleting may have, and the fact that even though they don't overly load the system, deleting one day's worth of data in that short time does have the effect of blowing out the instances buffer cache, resulting in subsequent queries running slightly slower for the next few hours as the cache is gradually restored.
For years we've been considering better methods. In the past, I had heard that people used partitioned tables to manage old data reaping - one month per partition, for example, and dropping the oldest partition on a monthly basis. The main drawback to this approach is that our reaping rules go beyond "remove month X". Users are allowed to specify how long data must stay in the system, based on key values (e.g., in an invoice table, account foo can be removed after 3 months, but account bar may need to remain for 2 years).
There is also the issue of referential integrity; Oracle documentation talks about using partitions for purging data mostly in the context of data warehouses, where tables tend to be hypercubes. Ours is closer to the OLTP end of things, and it is common for data in month X to have relationships to data in month Y. Creating the right partitioning keys for these tables would be ticklish at best.
As for the cache blowouts, I have read a bit about setting up dedicated buffer caches, but it seems like it's more on a per-table basis, as opposed to a per-user or per-transaction basis. To preserve the cache, I'd really like the reaping job to only keep one transaction's worth of data in the cache at any time, since there is no need to keep the data around once deleted.
Are we stuck using deletes for the foreseeable future, or are there other, more clever ways to deal with reaping?
For the most part I think that you're stuck doing deletes.
Your comments on the difficulty of using partitions in your case probably do prevent them being used effectively (different delete dates being used depending on the type of record) but it it possible that you could create a "delete date" column on the records that you could partition on? It would have the disadvantage of making updates quite expensive as a change in the delete date might cause row migration, so your update would really be implemented as a delete and insert.
It could be that even then you cannot use DDL partition operations to remove old data because of the referential integrity issues, but partitioning still might serve the purpose of physically clustering the rows to be deleted so that fewer blocks need to be modified in order to delete them, mitigating the impact on the buffer cache.
Delete's aren't that bad, provided that you rebuild your indexes. Oracle will recover the pages that no longer contain data.
However, as-of 8i (and quite probably still), it would not properly recover index pages that no longer contained valid references. Worse, since the index leaves were chained, you could get into a situation where it would start walking the leaf nodes to find a row. This would cause a rather significant drop in performance: queries that would normally take seconds could take minutes. The drop was also very sudden: one day it would be fine, the next day it wouldn't.
I discovered this behavior (there was an Oracle bug for it, so other people have too) with an application that used increasing keys and regularly deleted data. Our solution was to invert portions of the key, but that's not going to help you with dates.
What if you temporarily deactivate indexes, perform the deletes and then rebuild them? Would it improve the performance of your deletes? Of course, in this case you have to make sure the scripts are correct and ensure proper delete order and referential integrity.
We have the same problem, using the same strategy.
If the situation becomes really bad (very fragmented allocation of indexes, tables, ...), we try to apply space reclamation actions.
Tables have to allow row movement (like for the flashback):
alter table TTT enable row movement;
alter table TTT shrink space;
and then rebuild all indexes.
I don't know how you are with maintenance windows, if the application has to be usable all the time, it is harder, if not, you can do some "repacking" when it is off-line. "alter table TTT move tablespace SSSS" does a lot of work cleaning up the mess as the table is rewritten. You can also specify new storage parameters such as extent management, sizes, ... take a look in the docs.
I use a script like this to create a script for the whole database:
SET SQLPROMPT "-- "
SET ECHO OFF
SET NEWPAGE 0
SET SPACE 0
SET PAGESIZE 0
SET FEEDBACK OFF
SET HEADING OFF
SET TRIMSPOOL ON
SET TERMOUT OFF
SET VERIFY OFF
SET TAB OFF
spool doit.sql
select 'prompt Enabling row movement in '||table_name||'...'||CHR (10)||'alter table '||table_name||' enable row movement;' from user_tables where table_name not like '%$%' and table_name not like '%QTAB' and table_name not like 'SYS_%';
select 'prompt Setting initial ext for '||table_name||'...'||CHR (10)||'alter table '||table_name||' move storage (initial 1m);' from user_tables where table_name not like '%$%' and table_name not like '%QTAB' and table_name not like 'SYS_%';
select 'prompt Shrinking space for '||table_name||'...'||CHR (10)||'alter table '||table_name||' shrink space;' from user_tables where table_name not like '%$%' and table_name not like '%QTAB' and table_name not like 'SYS_%';
select 'prompt Rebuilding index '||index_name||'...'||CHR (10)||'alter index '||index_name||' rebuild;' from user_indexes where status = 'UNUSABLE';
spool off
prompt now check and then run #doit.sql
exit