My boss has a concern that creating LOB columns in our database tables could result in effectively permanent storage overhead if we do not partition the table and drop the partitions over time (as opposed to just deleting the records). He thinks that data for the LOB will be stored separately when the record is created, which seems to be correct, but that Oracle may not delete the data for the LOB column when the row is deleted.
Is this correct or even possible? I find it hard to believe that the DBMS would handle object storage so poorly. We are using Oracle 11g if that changes anything.
I eventually found this ask tom question which covers the same topic.
When the database creates an extent to store the LOBs it is associated directly with the table. When rows containing those LOBs are deleted, the LOBs are removed from the extent which remaining the same size (it does not shrink just because a LOB was removed). However, the table is able to reuse the space from the removed LOB as one would expect.
So while the database at large is not able to reuse the free space in an extent created for a table's LOB storage, the space IS free for use by the table. The extent will exist (and therefore take up space that would otherwise be completely available to the database) until table is dropped or truncated, or the extent is explictly deallocated.
In the end, unless the extent(s) remaining allocated to the table is a concern (or there are other concerns that may be addressed through partition), creating partitions solely for this purpose is unnecessary.
Related
I have a friend who has a website and asked me for help.
I often use MySQL databases but never Oracle databases.
And unfortunately he has an Oracle database, so I can't find a solution.
The available disk space is slowly decreasing... I delete a lot of lines from the table but that doesn't solve his problem.
The database continues to take up disk space slowly.
I read that LOB files do not return disk space, even if you delete data.
How can I reorganize LOB files easily with a simple request?
(or/and) How can I recover disk space on Oracle?
SELECT DISTINCT VERSION FROM PRODUCT_COMPONENT_VERSION
12.1.0.1.0
The BLOB column exists within the table blocks along with data even after deletion. It is only marked as unused. You can use the following command to free up space from the BLOB table:
ALTER TABLE <YOUR_TABLE_NAME> MODIFY
LOB <LOB_COLUMN_NAME>
( SHRINK SPACE );
Now, Table must have released some space and it is now available to be used within the tablespace.
Further, you can just alter the data file and reduce the size of the data file accordingly to free up space from Disk. (Note: Space allocated to the data file will not be automatically reduced. It must be done manually)
Cheers!!
I would like to use the power of bitmap indexes for tables where the indexed column has lots of repeating values, however I am in an OLTP environment. The rule is that you must not use bitmap indexes in an OLTP environment, since it causes block level locking when rows in the table are updated. I definitely do not want block level locking.
My table is however very static, that is it is very unlikely that a row will be added to it. Also two people would not add rows at the same time, so locking for that table is not that important.
My concern however is whether the locking on the table with the bitmap indexes could cause locking on other tables if the table is used in DML, DDL or sql statements along with other tables.
So is it safe to use a bitmap index in this scenario and can it cause locking in tables other than the table with the bitmap index.
For example:
Table per_person_types:
person_type_id description internal_name
1 Contractor EMP
2 Employee EMP
3 Terminated TERM
where I want the bitmap index on internal_name.
Yes, it's safe to use a bitmap index here.
The rule is per table, not per environment. It doesn't matter how volatile the database is, as long as the relevant table is static then bitmap indexes won't cause locking issues.
From the manual, part of the conditions for using a bitmap index are:
The indexed table is either read-only or not subject to significant
modification by DML statements.
Oracle stores lock information with the data. It does not track all locks in a global table. Locking unrelated objects will generally not cause too many locks, lock escalation, or other problems.
Also, bitmap indexes don't specifically lock per-block. The bitmaps can be compressed, and updating a single row can potentially lock the entire table.
If you're still nervous about bitmap indexes you could try using btree index compression instead. For highly repetitive values it can significantly shrink the size of the index, making it faster for some operations.
I noticed today that SQL command that is used to shrink LOBs in oracle does not work in 12c.
ALTER TABLE SAMPLE_TABLE MODIFY lob (LOB_COLUMN) (SHRINK SPACE)
This returns oracle error
ORA-10635: Invalid segment or tablespace type
In the oracle documentation it is mentioned that the SHRINK option is not supported for SecureFiles LOBs.
I want to know how blob compresses in secure files. Does oracle handles that internally?
Thanks
ALTER TABLE SAMPLE_TABLE MOVE LOB(LOB_COLUMN) STORE AS (TABLESPACE USERS)
Note: this is, unlike how it can be read, a move lob operation. It is a move TABLE operation, and while at it, moving a lob too.
This is why it invalidates indexes, - because it moves the whole table not just the lob. And of course it can take a very long time and it will consume 2x space during the operation, because oracle makes a copy of the data and only after it's complete it frees the old segments.
If you want to shrink LOBs using SecureFiles, use this statement:
ALTER TABLE SAMPLE_TABLE MOVE LOB(LOB_COLUMN) STORE AS (TABLESPACE USERS)
Be careful using it - this command invalidates all indexes on SAMPLE_TABLE, so you should rebuild them after you're finished with LOBs:
ALTER INDEX <index_name> REBUILD;
I have tried to find examples but they are all simple with a single where clause. Here is the situation. I have a bunch of legacy data transferred from another database. I also have the "good" tables in that same database. I need to transfer (data-conversion) data from the legacy tables to thew tables. Because this is a different set of tables the data-conversion requires complex joins to put the old data into the new tables correctly.
So, old tables old data.
New tables must have the old data but it requires lots of joins to get that old data into the new tables correctly.
Can I use direct path with lots of joins like this? INSERT SELECT (lots of joins)
Does direct path apply to tables that are already on the same database (transfer between tables)? Is it only for loading tables from say a text file?
Thank you.
The query in your SELECT can be as complex as you'd like with a direct-path insert. The direct-path refers only to the destination table. It has nothing to do with the way that data is read or processed.
If you're doing a direct-path insert, you're asking Oracle to insert the new data above the high water mark of the table so you bypass the normal code that reuses space in existing blocks for new rows to be inserted. It also has to block other inserts since you can't have the high water mark of the table change during a direct-path insert. This probably isn't a big deal if you've got a downtime window in which to do the load but it would be quite problematic if you wanted the existing tables to be available for other applications during the load.
No, on the contrary, it means you need to do a backup after a NOLOGGING load, not that you can't backup the database.
Allow me to elaborate a bit. Normally, when you do DML in Oracle, the before images of the changes you are are making get logged in UNDO, and all the changes (including the UNDO changes) are first written to REDO. This is how Oracle manages transactions, instance recovery, and database recovery. If a transaction is aborted or rolled back, Oracle uses the information in UNDO to undo the changes your transaction made. If the instance crashes, then on instance restart, Oracle will use the information in REDO and UNDO to recover up to the last committed transaction. First, Oracle will read the REDO and roll forward, then, use UNDO to roll back all the transactions that were not committed at the time of the crash. In this way, Oracle is able to recover up to the last committed transaction.
Now, when you specify an APPEND hint on an insert statement, Oracle will execute the INSERT with direct load. This means that data is loaded into brand new, never before used blocks, from above the highwater mark. Because the blocks being loaded are brand new, there is no "before image", so, Oracle can avoid writing UNDO, which improves performance. If the database is in NOARCHIVELOG mode, then Oracle will also not write REDO. On a database in ARCHIVELOG mode, Oracle will still write REDO, unless, before you do the insert /*+ append */, you set the table to NOLOGGING, (i.e. alter table tab_name nologging;). In that case, REDO logging is disabled for the table. However, this is where you could run into backup/recovery implications. If you do a NOLOGGING direct load, and then you suffer a media failure, and the datafile containing the segment with the nologging operation is restored from a backup taken before the nologging load, then the redo log will not contain the changes required to recover that segment. So, what happens? Well, when you do a NOLOGGING load, Oracle writes extent invaldation records to the redo log, instead of the actual changes. Then, if you use that redo in recovery, those data blocks will be marked logically corrupt. Any subsequent queries against that segment will get an ORA-26040 error.
So, how to avoid this? Well, you should always take a backup imediately following any NOLOGGING direct load. If you restore/recover from a backup taken after the nologging load, there is no problem, because the data will be in the datablocks in the file that was restored.
Hope that's clear,
-Mark
Yes, there should not be any arbitrary limits on query complexity.
If you do
insert /*+ APPEND */ into target_table select .... from source1, source2..., sourceN where
It should work fine. Consider though, that the performance of the load will be limited by the performance of that query, so, be sure it's well-tuned, if you're expecting good performance.
Finally, consider whether setting NOLOGGING on the target table would improve performance significantly. But, also consider the backup recovery implications, if you decide to implement NOLOGGING.
Hope that helps,
-Mark
I need to perform a query 2.5 million times. This query generates some rows which I need to AVG(column) and then use this AVG to filter the table from all values below average. I then need to INSERT these filtered results into a table.
The only way to do such a thing with reasonable efficiency, seems to be by creating a TEMPORARY TABLE for each query-postmaster python-thread. I am just hoping these TEMPORARY TABLEs will not be persisted to hard drive (at all) and will remain in memory (RAM), unless they are out of working memory, of course.
I would like to know if a TEMPORARY TABLE will incur disk writes (which would interfere with the INSERTS, i.e. slow to whole process down)
Please note that, in Postgres, the default behaviour for temporary tables is that they are not automatically dropped, and data is persisted on commit. See ON COMMIT.
Temporary table are, however, dropped at the end of a database session:
Temporary tables are automatically dropped at the end of a session, or
optionally at the end of the current transaction.
There are multiple considerations you have to take into account:
If you do want to explicitly DROP a temporary table at the end of a transaction, create it with the CREATE TEMPORARY TABLE ... ON COMMIT DROP syntax.
In the presence of connection pooling, a database session may span multiple client sessions; to avoid clashes in CREATE, you should drop your temporary tables -- either prior to returning a connection to the pool (e.g. by doing everything inside a transaction and using the ON COMMIT DROP creation syntax), or on an as-needed basis (by preceding any CREATE TEMPORARY TABLE statement with a corresponding DROP TABLE IF EXISTS, which has the advantage of also working outside transactions e.g. if the connection is used in auto-commit mode.)
While the temporary table is in use, how much of it will fit in memory before overflowing on to disk? See the temp_buffers option in postgresql.conf
Anything else I should worry about when working often with temp tables? A vacuum is recommended after you have DROPped temporary tables, to clean up any dead tuples from the catalog. Postgres will automatically vacuum every 3 minutes or so for you when using the default settings (auto_vacuum).
Also, unrelated to your question (but possibly related to your project): keep in mind that, if you have to run queries against a temp table after you have populated it, then it is a good idea to create appropriate indices and issue an ANALYZE on the temp table in question after you're done inserting into it. By default, the cost based optimizer will assume that a newly created the temp table has ~1000 rows and this may result in poor performance should the temp table actually contain millions of rows.
Temporary tables provide only one guarantee - they are dropped at the end of the session. For a small table you'll probably have most of your data in the backing store. For a large table I guarantee that data will be flushed to disk periodically as the database engine needs more working space for other requests.
EDIT:
If you're absolutely in need of RAM-only temporary tables you can create a table space for your database on a RAM disk (/dev/shm works). This reduces the amount of disk IO, but beware that it is currently not possible to do this without a physical disk write; the DB engine will flush the table list to stable storage when you create the temporary table.