Repartitioning a table on a different column - oracle

We need to change the structure of our DB by re-partitioning a parent table and adding partitions to the child tables. The partitions will be based on range of a date-time field (which is the time of insertion).
The goal is to implement the changes with as little downtime as possible. So we have though on doing it on 2 phases: do the main load while the application is running and load the delta after the application has been shutdown.
I would like to know if the following steps are correct for the plan I mentioned, focusing on the part of re-partitioning the parent table. I have based on this example from Oracle: https://docs.oracle.com/database/121/SUTIL/GUID-56D8448A-0106-4B8C-85E0-11CFED8C71B1.htm
1. create a directory for the export/import of dumps
-- user: SYS
CREATE OR REPLACE DIRECTORY dumps AS '/data/tmp';
GRANT READ, WRITE ON DIRECTORY dumps TO APP_USR;
2. export the table while the application is running
# nohup expdp APP_USR/pwd parfile=export.par &
directory=dumps
dumpfile=parent_table.dmp
parallel=8
logfile=export.log
tables=parent_table
query=parent_table:"WHERE VERSION_TSP <= sysdate"
3. create a duplicate of the table with the new partitioning
CREATE TABLE "PARENT_TABLE_REPARTITIONED"
(
-- same fields as original
CONSTRAINT PK_TABLE_ PRIMARY KEY (foo, boo) -- with different name than original
)
PARALLEL 8
PARTITION BY RANGE (VERSION_TSP)
INTERVAL (NUMTOYMINTERVAL(1, 'MONTH'))
(
PARTITION PARENT_TABLE_PARTITION_ -- with different name than original (should I?)
VALUES LESS THAN (TO_DATE('2007-07-01', 'YYYY-MM-DD')) COMPRESS FOR OLTP
)
ENABLE ROW MOVEMENT
;
-- with new local indexes (named differently)
CREATE INDEX IDX_VERSION_TSP_ ON PARENT_TABLE_REPARTITIONED (VERSION_TSP) LOCAL;
4. populate the re-partitioned table while the application is running
# nohup impdp APP_USR/pwd parfile=import.par &
directory=dumps
dumpfile=parent_table.dmp
parallel=8
logfile=import.log
remap_table=PARENT_TABLE:PARENT_TABLE_REPARTITIONED
table_exists_action=APPEND
5. Application shutdown (and backup the DB)
6. populate the delta
-- execute with nohup
declare
startDate DATE;
begin
select max(version_tsp) into startDate from PARENT_TABLE_REPARTITIONED;
insert into PARENT_TABLE_REPARTITIONED select /*+ parallel(8) */ s.* from PARENT_TABLE s where s.VERSION_TSP > startDate;
commit;
end;
7. switch tables
drop table parent_table cascade constraints;
rename parent_table_repartitioned to parent_table;
8. deploy the new version of the application
Is ok to export ALL from the table or should I export DATA_ONLY?
But most importantly, are those steps a valid approach?

Are you on 12.2 or higher? It could be as easy as:
SQL> create table t
2 partition by range ( created )
3 (
4 partition p1 values less than ( date '2020-01-01' ),
5 partition p2 values less than ( date '2021-01-01' ),
6 partition p3 values less than ( date '2022-01-01' ),
7 partition p4 values less than ( date '2023-01-01' )
8 )
9 as select * from dba_objects
10 where created is not null
11 and last_ddl_time is not null;
Table created.
SQL>
SQL> create index t_ix on t ( owner ) local;
Index created.
SQL>
SQL>
SQL> alter table t modify
2 partition by range ( last_ddl_time )
3 (
4 partition p1 values less than ( date '2020-01-01' ),
5 partition p2 values less than ( date '2021-01-01' ),
6 partition p3 values less than ( date '2022-01-01' ),
7 partition p4 values less than ( date '2023-01-01' )
8 ) online ;
Table altered.

Related

ORA-30657 when creating an interval (or auto list) external partition table

When trying to create an Interval (or auto list) external partitioned table with DBMS_CLOUD, I am getting ORA-30657: operation not supported on external organized table.
What am I missing?
SQL> BEGIN
2 DBMS_CLOUD.CREATE_EXTERNAL_PART_TABLE(
3 table_name =>'PETX',
4 credential_name =>'MY_CRED',
5 format => json_object('delimiter' value '#'),
6 column_list => 'deptno number,dname char(14),loc char(13)',
7 partitioning_clause => 'partition by range (deptno) interval (15)
8 (
9 partition xp1 values less than (15) location(''https://swiftobjectstorage.XXXX/xp1_15.txt'') ,
10 partition xp2 values less than (30) location (''https://swiftobjectstorage.XXXX/xp2_30.txt'')
11 )'
12 );
13 END;
14 /
Error starting at line : 5 in command -
BEGIN
DBMS_CLOUD.CREATE_EXTERNAL_PART_TABLE(
table_name =>'PETX',
credential_name =>'MY_CRED',
format => json_object('delimiter' value '#'),
column_list => 'deptno number,dname char(14),loc char(13)',
partitioning_clause => 'partition by range (deptno) interval (15)
(
partition xp1 values less than (15) location(''https://swiftobjectstorage.XXXX/xp1_15.txt'') ,
partition xp2 values less than (30) location (''https://swiftobjectstorage.XXXX/xp2_30.txt'')
)'
);
END;
Error report -
ORA-20000: ORA-30657: operation not supported on external organized table
ORA-06512: at "C##CLOUD$SERVICE.DBMS_CLOUD", line 1289
ORA-06512: at "C##CLOUD$SERVICE.DBMS_CLOUD", line 4115
Interval and auto-list partitioning is functionality that creates new partitions as-needed, based on new data inserted into the database where the partition key(s) of the new data do not map to any existing partition. With external tables the database does not control the data (or insertion of it), thus the concept of auto-generation of a partition at insertion is not applying to external tables.
Adding a new partition to a partitioned external table is equivalent to having new data files that the table has to point at. You have to tell the database about this, which you are doing by adding a new partition to a table (manually or programmatically).
Short example of how to make the above-example work and how to add a new partition (pointing to additional data files):
SQL>
SQL> BEGIN
2 DBMS_CLOUD.CREATE_EXTERNAL_PART_TABLE(
3 table_name =>'PETX',
4 credential_name =>'MY_CRED',
5 format => json_object('delimiter' value '#'),
6 column_list => 'deptno number,dname char(14),loc char(13)',
7 partitioning_clause => 'partition by range (deptno)
8 (
9 partition xp1 values less than (15) location(''https://swiftobjectstorage.XXXX/xp1_15.txt'') ,
10 partition xp2 values less than (30) location (''https://swiftobjectstorage.XXXX/xp2_30.txt'')
11 )'
12 );
13 END;
14 /
PL/SQL procedure successfully completed.
SQL>
SQL> alter table petx add partition xp3 values less than (50) location ('https://swiftobjectstorage.XXXX/xp3_45.txt');
Table PETX altered.
SQL> select count(*) from petx partition (xp3);
COUNT(*)
----------
49999999
PS: The exact same behavior applies to standard Oracle Database.

How to create an incremental view where old data is not updated?

Is it possible to create a materialized view where it is only incremental?
I would like the old data already inserted not to be updated, only new insertions should be included in the view.
If possible, how could I do it?
Is there any documentation or any place I can use as a guide?
If you want to see every row at the time of insert in an MV the short answer is:
You can't.
A materialized view stores the result of a query as it exists now, not some time in the past. So after updating a row you can either see its current values or exclude it from the MV.
If you want to preserve/view the state of data at insert you have a few options:
Make the table insert-only (possibly storing change history within the table)
Capture the data when its added (e.g. via triggers) into another table
Use Flashback Data Archive to store change history and view it with Flashback Query
Which of these is most appropriate depends on why you need view the data at insert.
Technically it is possible to view data at the time of insert - up to a point.
With Flashback Version Query you can see changes to the table over time. So you can do something like this:
create table t (
c1 int, c2 int,
insert_date timestamp,
update_date timestamp
);
exec dbms_session.sleep(10);
insert into t values ( 1, 1, systimestamp, systimestamp );
insert into t values ( 2, 2, systimestamp, systimestamp );
commit;
create materialized view mv
as
select t.*
from t
versions between scn minvalue
and maxvalue
where versions_operation = 'I';
exec dbms_session.sleep(10);
update t
set c2 = 9999,
update_date = systimestamp
where c1 = 2;
insert into t values ( 3, 3, systimestamp, systimestamp );
commit;
exec dbms_mview.refresh ( 'mv' );
select * from t;
C1 C2 INSERT_DATE UPDATE_DATE
1 1 26-JUL-2021 13.49.51.666954000 26-JUL-2021 13.49.51.666954000
2 9999 26-JUL-2021 13.49.51.712259000 26-JUL-2021 13.50.08.421872000
3 3 26-JUL-2021 13.50.08.462300000 26-JUL-2021 13.50.08.462300000
select *
from mv;
C1 C2 INSERT_DATE UPDATE_DATE
3 3 26-JUL-2021 13.50.08.462300000 26-JUL-2021 13.50.08.462300000
2 2 26-JUL-2021 13.49.51.712259000 26-JUL-2021 13.49.51.712259000
1 1 26-JUL-2021 13.49.51.666954000 26-JUL-2021 13.49.51.666954000
This uses undo to reconstruct history. Eventually older changes will drop off and you're back to only seeing the current state. By default you only get 15 minutes worth of changes!
If you want to store history for long period of time, Flashback Data Archive is the way to go.

Oracle 12 get changed tables (inserted, deleted oder updated records)

Is it possible in an Oracle 12/19 database to get a list of tables and its records which have changed/added/deleted within the last X month? The problem here is, that the tables do not have any kind of timestamp per record. Is this possible?
You can use Flashback Query to find the changes to a table in period of time/range of SCNs:
create table t (
c1 int
);
exec dbms_lock.sleep ( 2 );
insert into t values ( 1 );
commit;
select current_scn from v$database;
CURRENT_SCN
19279246
exec dbms_lock.sleep ( 2 );
update t set c1 = 2;
commit;
delete t;
exec dbms_lock.sleep ( 2 );
commit;
select t.*, versions_operation, versions_startscn, versions_endscn
from t
versions between scn 19279246 and maxvalue
order by versions_startscn nulls first;
C1 VERSIONS_OPERATION VERSIONS_STARTSCN VERSIONS_ENDSCN
1 <null> <null> 19279250
2 U 19279250 19279253
2 D 19279253 <null>
The default window for this is small (900s). So it's highly unlikely you'll be able to go back and view changes from weeks or even hours ago. Enable Flashback Data Archive to define how long you want to keep the history for.

SQL Error: ORA-14006: invalid partition name

I am trying to partition an existing table in Oracle 12C R1 using below SQL statement.
ALTER TABLE TABLE_NAME MODIFY
PARTITION BY RANGE (DATE_COLUMN_NAME)
INTERVAL (NUMTOYMINTERVAL(1,'MONTH'))
(
PARTITION part_01 VALUES LESS THAN (TO_DATE('01-SEP-2017', 'DD-MON-RRRR'))
) ONLINE;
Getting error:
Error report -
SQL Error: ORA-14006: invalid partition name
14006. 00000 - "invalid partition name"
*Cause: a partition name of the form <identifier> is
expected but not present.
*Action: enter an appropriate partition name.
Partition needs to be done on the basis of data datatype column with the interval of one month.
Min value of Date time column in the Table is 01-SEP-2017.
You can't partition an existing table like that. That statement is modifying the partition that hasn't been created yet. I don't know the automatic way to do this operation and I am not sure that you can do it.
Although I have done this thing many times but with manual steps. Do the following if you can't find an automated solution:
Create a partitioned table named table_name_part with your clauses and all your preferences.
Insert into this partitioned table all rows from original table. Pay attention to compression. If you have some compression on table (Basic or HCC) you have to use + APPEND hint.
Create on partitioned table your constrains and indexes from the original table.
Rename the tables and drop the original table. Do not drop it until you make some counts on them.
I saw that your table has the option to auto-create partition if it does not exists. (NUMTOYMINTERVAL(1,'MONTH')) So you have to create your table with first partition only. I assume that you have here a lot of read-only data, so you won't have any problem with consistency instead of last month. Probably there is some read-write data so there you have to be more careful with the moment when you want to insert data in new table and switch tables.
Hope to help you. As far as I know there might be a package named DBMS_REDEFINITION that can help you with an automated version of my steps. If you need more details or need some help on my method, please don't hesitate.
UPDATE:
From Oracle 12c R2 you can convert a table from an unpartitioned to a partitioned one with your method. Find a link below. Now this is a challenge for me and I am trying to convert, but I think there is no way to make this conversion online in 12c R1.
In previous releases you could partition a non-partitioned table using
EXCHANGE PARTITION or DBMS_REDEFINITION in an "almost online" manner,
but both methods require multiple steps. Oracle Database 12c Release 2
makes it easier than ever to convert a non-partitioned table to a
partitioned table, requiring only a single command and no downtime.
https://oracle-base.com/articles/12c/online-conversion-of-a-non-partitioned-table-to-a-partitioned-table-12cr2
Solution
I found a solution for you. Here you will have all of my steps that I run to convert table online. :)
1. Create regular table and populate it.
CREATE TABLE SCOTT.tab_unpartitioned
(
id NUMBER,
description VARCHAR2 ( 50 ),
created_date DATE
);
INSERT INTO tab_unpartitioned
SELECT LEVEL,
'Description for ' || LEVEL,
ADD_MONTHS ( TO_DATE ( '01-JAN-2017', 'DD-MON-YYYY' ),
-TRUNC ( DBMS_RANDOM.VALUE ( 1, 4 ) - 1 ) * 12 )
FROM DUAL
CONNECT BY LEVEL <= 10000;
COMMIT;
2. Create partitioned table with same structure.
--If you are on 11g create table with CREATE TABLE command but with different name. ex: tab_partitioned
CREATE TABLE SCOTT.tab_partitioned
(
id NUMBER,
description VARCHAR2 ( 50 ),
created_date DATE
)
PARTITION BY RANGE (created_date)
INTERVAL( NUMTOYMINTERVAL(1,'YEAR'))
(PARTITION part_2015 VALUES LESS THAN (TO_DATE ( '01-JAN-2016', 'DD-MON-YYYY' )),
PARTITION part_2016 VALUES LESS THAN (TO_DATE ( '01-JAN-2017', 'DD-MON-YYYY' )),
PARTITION part_2017 VALUES LESS THAN (TO_DATE ( '01-JAN-2018', 'DD-MON-YYYY' )));
--this is an alter command that works only in 12c.
ALTER TABLE tab_partitioned
MODIFY
PARTITION BY RANGE (created_date)
(PARTITION part_2015 VALUES LESS THAN (TO_DATE ( '01-JAN-2016', 'DD-MON-YYYY' )),
PARTITION part_2016 VALUES LESS THAN (TO_DATE ( '01-JAN-2017', 'DD-MON-YYYY' )),
PARTITION part_2017 VALUES LESS THAN (TO_DATE ( '01-JAN-2018', 'DD-MON-YYYY' )));
3. Check if the table can be converted. This procedure should run without any error.
Prerequisites: table should have an UNIQUE INDEX and a Primary Key constraint.
EXEC DBMS_REDEFINITION.CAN_REDEF_TABLE('SCOTT','TAB_UNPARTITIONED');
4. Run the following steps like I have done.
EXEC DBMS_REDEFINITION.START_REDEF_TABLE('SCOTT','TAB_UNPARTITIONED','TAB_PARTITIONED');
var num_errors varchar2(2000);
EXEC DBMS_REDEFINITION.COPY_TABLE_DEPENDENTS('SCOTT','TAB_UNPARTITIONED','TAB_PARTITIONED', 1,TRUE,TRUE,TRUE,FALSE,:NUM_ERRORS,FALSE);
SQL> PRINT NUM_ERRORS -- Should return 0
EXEC DBMS_REDEFINITION.SYNC_INTERIM_TABLE('SCOTT','TAB_UNPARTITIONED','TAB_PARTITIONED');
EXEC DBMS_REDEFINITION.FINISH_REDEF_TABLE('SCOTT','TAB_UNPARTITIONED','TAB_PARTITIONED');
At the end of the script you will see that the original table is partitioned.
Try Oracle Live SQL I used to use Oracle 11g EE and got the same error message. So I tried Oracle live SQL and it perfectly worked. It has very simple and easy to understand interface,
For example, I'm creating a sales table and inserting some dummy data and partition it using range partitioning method,
CREATE TABLE sales
(product VARCHAR(300),
country VARCHAR(100),
sales_year DATE);
INSERT INTO sales (product, country, sales_year )
VALUES ('Computer','Kazakhstan',TO_DATE('01/02/2018','DD/MM/YYYY'));
INSERT INTO sales (product, country, sales_year )
VALUES ('Mobile Phone','China',TO_DATE('23/12/2019','DD/MM/YYYY'));
INSERT INTO sales (product, country, sales_year )
VALUES ('Camara','USA',TO_DATE('20/11/2020','DD/MM/YYYY'));
INSERT INTO sales (product, country, sales_year )
VALUES ('Watch','Bangladesh',TO_DATE('19/03/2020','DD/MM/YYYY'));
INSERT INTO sales (product, country, sales_year )
VALUES ('Cake','Sri Lanka',TO_DATE('13/04/2021','DD/MM/YYYY'));
ALTER TABLE sales MODIFY
PARTITION BY RANGE(sales_year)
INTERVAL(INTERVAL '1' YEAR)
(
PARTITION sales_2018 VALUES LESS THAN(TO_DATE('01/01/2019','DD/MM/YYYY')),
PARTITION sales_2019 VALUES LESS THAN(TO_DATE('01/01/2020','DD/MM/YYYY')),
PARTITION sales_2020 VALUES LESS THAN(TO_DATE('01/01/2021','DD/MM/YYYY')),
PARTITION sales_2021 VALUES LESS THAN(TO_DATE('01/01/2022','DD/MM/YYYY'))
)ONLINE;
Finally, I can write SELECT query for partitions to confirm that the partitions are created successfully.
SELECT *
FROM sales PARTITION (sales_2020);
And it gives the expected output,

Oracle DB daily partitioning

I have the following table
CREATE TABLE "METRIC_VALUE_RAW"
(
"SUBELEMENT_ID" INTEGER NOT NULL ,
"METRIC_METADATA_ID" INTEGER NOT NULL ,
"METRIC_VALUE_INT" INTEGER,
"METRIC_VALUE_FLOAT" FLOAT(126),
"TIME_STAMP" TIMESTAMP NOT NULL
) ;
Every hour data will be loaded into the table using sql loader.
I want to create partitions so that data for every day go into a partition.
In table I want to store data for 30 days. So when it crosses 30 days, the oldest partition should get deleted.
Can you share your ideas on how I can design the partitions.
here is an example how to do it on Oracle 11g and it works very well. I haven't tried it on Oracle 10g, you can try it.
This is the way, how to create a table with daily partitions:
CREATE TABLE XXX (
partition_date DATE,
...,
...,
)
PARTITION BY RANGE (partition_date)
INTERVAL (NUMTODSINTERVAL(1, 'day'))
(
PARTITION part_01 values LESS THAN (TO_DATE('2000-01-01','YYYY-MM-DD'))
)
TABLESPACE MY_TABLESPACE
NOLOGGING;
As you see above, Oracle will automaticaly create separate partitions for each distinct partition_day after 1st January 2000. The records, whose partition_date is older than this date, will be stored in partition called 'part_01'.
You can monitore your table partitions using this statement:
SELECT * FROM user_tab_partitions WHERE table_name = 'XXX';
Afterwards, when you would like to delete some partitions, use following command:
ALTER TABLE XXX DROP PARTITION AAAAAA UPDATE GLOBAL INDEXES
where 'AAAAAA' is parition name.
I hope it will help you!
As i said , There are big differences in partition automation between 10g and 11g.
In 10G you will have to manually manage the partitions during your ETL process (I'm sure every 10g DBA has a utility package he wrote to manage partitions ... ).
For steps 1 & 2 , you have several options
load data directly into the daily partition.
load data into a new partition and merge it into the daily one.
load data into a new partition every hour, and during a maintenance
window merge all hourly partitions into a daily partition.
The right way for you depends on your needs. Is the newly added data is queried immediately ? In what manner ? Would you query for data across several hours (or loads...) ? Are you showing aggregations ? are you performing DML operations on the data (DDL operations on partitions cause massive locking).
about 3, again - manually. drop old partitions.
In 11G, you have the new interval partition feature with automates some of the tasks mentioned above.
Following is a sample create table sql to parititon data:
CREATE TABLE quarterly_report_status (
report_id INT NOT NULL,
report_status VARCHAR(20) NOT NULL,
report_updated TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
)
PARTITION BY RANGE ( UNIX_TIMESTAMP(report_updated) ) (
PARTITION p0 VALUES LESS THAN ( UNIX_TIMESTAMP('2008-01-01 00:00:00') ),
PARTITION p1 VALUES LESS THAN ( UNIX_TIMESTAMP('2008-01-02 00:00:00') ),
PARTITION p2 VALUES LESS THAN ( UNIX_TIMESTAMP('2008-01-03 00:00:00') ),
PARTITION p3 VALUES LESS THAN ( UNIX_TIMESTAMP('2008-01-04 00:00:00') ),
PARTITION p4 VALUES LESS THAN ( UNIX_TIMESTAMP('2008-01-05 00:00:00') ),
PARTITION p5 VALUES LESS THAN ( UNIX_TIMESTAMP('2008-01-06 00:00:00') ),
PARTITION p6 VALUES LESS THAN ( UNIX_TIMESTAMP('2008-01-07 00:00:00') ),
PARTITION p7 VALUES LESS THAN ( UNIX_TIMESTAMP('2008-01-08 00:00:00') ),
PARTITION p8 VALUES LESS THAN ( UNIX_TIMESTAMP('2008-01-09 00:00:00') ),
PARTITION p9 VALUES LESS THAN (MAXVALUE)
);
Paritions will be created by DBa and rest will be taken care of by oracle.
If you want to delete partition then you will have to write separate jobs for it.

Resources