Oracle 12C, non partitioned, no ASM.
This is the background. I have a table with multiple columns, 3 of them being -
TRAN_DATE DATE
TRAN_TIME TIMESTAMP(6)
FINAL_DATETIME NOT NULL TIMESTAMP(6)
The table has around 70 million records. What I want to do is concatenate the tran_date and the tran_time field and update the final_datetime field with that output, for all 70 million records.
This is the query I have -
update MYSCHEMA.MYTAB set FINAL_DATETIME = (to_timestamp( (to_char(tran_date, 'YYYY/MM/DD') || ' ' ||to_char(TRAN_TIME,'HH24MISS') ), 'YYYY-MM-DD HH24:MI:SS.FF'))
Eg:
At present (for one record)
TRAN_DATE=01-DEC-16
TRAN_TIME=01-JAN-70 12.35.58.000000 AM /*I need only the time part from this*/
FINAL_DATETIME=22-OCT-18 04.37.18.870000 PM
Post the query - the FINAL_DATETIME needs to be
01-DEC-16 12.35.58.000000 AM
The to_timestamp does require 2 character strings and I fear this will slow down the update a lot. Any suggestions?
What more can I do to increase performance? No one else will be using the table at this point, so, I do have the option to
Drop indices
Turn off logging
and anything more anyone can suggest.
Any help is appreciated.
I would prefer CTAS method and your job would be simpler if you didn't have indexes, triggers and constraints on your table.
Create a new table for the column to be modified.
CREATE TABLE mytab_new
NOLOGGING
AS
SELECT /*+ FULL(mytab) PARALLEL(mytab, 10) */ --hint to speed up the select.
CAST(tran_date AS TIMESTAMP) + ( tran_time - trunc(tran_time) ) AS final_datetime
FROM mytab;
I have included only one(the final) column in your new table because storing the other two in the new table is waste of resources. You may include other columns in select apart from the two now redundant ones.
Read logging/nologging to know about NOLOGGING option in the select.
Next step is to rebuild indexes, triggers and constraints for the new table new_mytab using the definition from mytab for other columns if they exist.
Then rename the tables
rename mytab to mytab_bkp;
rename mytab_new to mytab;
You may drop the table mytab_bkp after validating the new table or later when you feel you no longer need it.
Demo
Related
I am trying to partition an existing table in Oracle 12C R1 using below SQL statement.
ALTER TABLE TABLE_NAME MODIFY
PARTITION BY RANGE (DATE_COLUMN_NAME)
INTERVAL (NUMTOYMINTERVAL(1,'MONTH'))
(
PARTITION part_01 VALUES LESS THAN (TO_DATE('01-SEP-2017', 'DD-MON-RRRR'))
) ONLINE;
Getting error:
Error report -
SQL Error: ORA-14006: invalid partition name
14006. 00000 - "invalid partition name"
*Cause: a partition name of the form <identifier> is
expected but not present.
*Action: enter an appropriate partition name.
Partition needs to be done on the basis of data datatype column with the interval of one month.
Min value of Date time column in the Table is 01-SEP-2017.
You can't partition an existing table like that. That statement is modifying the partition that hasn't been created yet. I don't know the automatic way to do this operation and I am not sure that you can do it.
Although I have done this thing many times but with manual steps. Do the following if you can't find an automated solution:
Create a partitioned table named table_name_part with your clauses and all your preferences.
Insert into this partitioned table all rows from original table. Pay attention to compression. If you have some compression on table (Basic or HCC) you have to use + APPEND hint.
Create on partitioned table your constrains and indexes from the original table.
Rename the tables and drop the original table. Do not drop it until you make some counts on them.
I saw that your table has the option to auto-create partition if it does not exists. (NUMTOYMINTERVAL(1,'MONTH')) So you have to create your table with first partition only. I assume that you have here a lot of read-only data, so you won't have any problem with consistency instead of last month. Probably there is some read-write data so there you have to be more careful with the moment when you want to insert data in new table and switch tables.
Hope to help you. As far as I know there might be a package named DBMS_REDEFINITION that can help you with an automated version of my steps. If you need more details or need some help on my method, please don't hesitate.
UPDATE:
From Oracle 12c R2 you can convert a table from an unpartitioned to a partitioned one with your method. Find a link below. Now this is a challenge for me and I am trying to convert, but I think there is no way to make this conversion online in 12c R1.
In previous releases you could partition a non-partitioned table using
EXCHANGE PARTITION or DBMS_REDEFINITION in an "almost online" manner,
but both methods require multiple steps. Oracle Database 12c Release 2
makes it easier than ever to convert a non-partitioned table to a
partitioned table, requiring only a single command and no downtime.
https://oracle-base.com/articles/12c/online-conversion-of-a-non-partitioned-table-to-a-partitioned-table-12cr2
Solution
I found a solution for you. Here you will have all of my steps that I run to convert table online. :)
1. Create regular table and populate it.
CREATE TABLE SCOTT.tab_unpartitioned
(
id NUMBER,
description VARCHAR2 ( 50 ),
created_date DATE
);
INSERT INTO tab_unpartitioned
SELECT LEVEL,
'Description for ' || LEVEL,
ADD_MONTHS ( TO_DATE ( '01-JAN-2017', 'DD-MON-YYYY' ),
-TRUNC ( DBMS_RANDOM.VALUE ( 1, 4 ) - 1 ) * 12 )
FROM DUAL
CONNECT BY LEVEL <= 10000;
COMMIT;
2. Create partitioned table with same structure.
--If you are on 11g create table with CREATE TABLE command but with different name. ex: tab_partitioned
CREATE TABLE SCOTT.tab_partitioned
(
id NUMBER,
description VARCHAR2 ( 50 ),
created_date DATE
)
PARTITION BY RANGE (created_date)
INTERVAL( NUMTOYMINTERVAL(1,'YEAR'))
(PARTITION part_2015 VALUES LESS THAN (TO_DATE ( '01-JAN-2016', 'DD-MON-YYYY' )),
PARTITION part_2016 VALUES LESS THAN (TO_DATE ( '01-JAN-2017', 'DD-MON-YYYY' )),
PARTITION part_2017 VALUES LESS THAN (TO_DATE ( '01-JAN-2018', 'DD-MON-YYYY' )));
--this is an alter command that works only in 12c.
ALTER TABLE tab_partitioned
MODIFY
PARTITION BY RANGE (created_date)
(PARTITION part_2015 VALUES LESS THAN (TO_DATE ( '01-JAN-2016', 'DD-MON-YYYY' )),
PARTITION part_2016 VALUES LESS THAN (TO_DATE ( '01-JAN-2017', 'DD-MON-YYYY' )),
PARTITION part_2017 VALUES LESS THAN (TO_DATE ( '01-JAN-2018', 'DD-MON-YYYY' )));
3. Check if the table can be converted. This procedure should run without any error.
Prerequisites: table should have an UNIQUE INDEX and a Primary Key constraint.
EXEC DBMS_REDEFINITION.CAN_REDEF_TABLE('SCOTT','TAB_UNPARTITIONED');
4. Run the following steps like I have done.
EXEC DBMS_REDEFINITION.START_REDEF_TABLE('SCOTT','TAB_UNPARTITIONED','TAB_PARTITIONED');
var num_errors varchar2(2000);
EXEC DBMS_REDEFINITION.COPY_TABLE_DEPENDENTS('SCOTT','TAB_UNPARTITIONED','TAB_PARTITIONED', 1,TRUE,TRUE,TRUE,FALSE,:NUM_ERRORS,FALSE);
SQL> PRINT NUM_ERRORS -- Should return 0
EXEC DBMS_REDEFINITION.SYNC_INTERIM_TABLE('SCOTT','TAB_UNPARTITIONED','TAB_PARTITIONED');
EXEC DBMS_REDEFINITION.FINISH_REDEF_TABLE('SCOTT','TAB_UNPARTITIONED','TAB_PARTITIONED');
At the end of the script you will see that the original table is partitioned.
Try Oracle Live SQL I used to use Oracle 11g EE and got the same error message. So I tried Oracle live SQL and it perfectly worked. It has very simple and easy to understand interface,
For example, I'm creating a sales table and inserting some dummy data and partition it using range partitioning method,
CREATE TABLE sales
(product VARCHAR(300),
country VARCHAR(100),
sales_year DATE);
INSERT INTO sales (product, country, sales_year )
VALUES ('Computer','Kazakhstan',TO_DATE('01/02/2018','DD/MM/YYYY'));
INSERT INTO sales (product, country, sales_year )
VALUES ('Mobile Phone','China',TO_DATE('23/12/2019','DD/MM/YYYY'));
INSERT INTO sales (product, country, sales_year )
VALUES ('Camara','USA',TO_DATE('20/11/2020','DD/MM/YYYY'));
INSERT INTO sales (product, country, sales_year )
VALUES ('Watch','Bangladesh',TO_DATE('19/03/2020','DD/MM/YYYY'));
INSERT INTO sales (product, country, sales_year )
VALUES ('Cake','Sri Lanka',TO_DATE('13/04/2021','DD/MM/YYYY'));
ALTER TABLE sales MODIFY
PARTITION BY RANGE(sales_year)
INTERVAL(INTERVAL '1' YEAR)
(
PARTITION sales_2018 VALUES LESS THAN(TO_DATE('01/01/2019','DD/MM/YYYY')),
PARTITION sales_2019 VALUES LESS THAN(TO_DATE('01/01/2020','DD/MM/YYYY')),
PARTITION sales_2020 VALUES LESS THAN(TO_DATE('01/01/2021','DD/MM/YYYY')),
PARTITION sales_2021 VALUES LESS THAN(TO_DATE('01/01/2022','DD/MM/YYYY'))
)ONLINE;
Finally, I can write SELECT query for partitions to confirm that the partitions are created successfully.
SELECT *
FROM sales PARTITION (sales_2020);
And it gives the expected output,
Right now I run the following Hive query
CREATE TABLE dwo_analysis.exp_shown AS
SELECT
MIN(sc.date_time) as first_shown_time,
SUBSTR(sc.post_evar12,1,24) as guid,
sc.post_evar238 as experiment_name,
sc.post_evar239 as variant_name
FROM test
WHERE report_suite='adbemmarvelweb.prod'
AND date >= DATE_SUB(CURRENT_DATE,90) AND date < DATE_SUB(CURRENT_DATE, 2)
AND post_prop5 = 'experiment:standard:authenticated:shown'
AND post_evar238 NOT LIKE 'control%'
AND post_evar238 <> ''
AND post_evar239 <> ''
The table test is large. I would like to optimize this query by running it once, and every other time updating the table by getting the last 2 days of data and adding it to the table.
so basically run the above query once and every time run it again but with the condition
WHERE click_date >= DATE_SUB(CURRENT_DATE, 2) AND click_date < DATE_SUB(CURRENT_DATE)
How do I update the table using hive to populate the the rows as mentioned in the condition above?
First, your queries would be quicker if the Hive table were partitioned based on date. Your create table statement isn't inserting into any partitions, therefore I suspect your table is not partitioned. It would also be quicker if the source data were Parquet/ORC
In any case, you can overwrite the table for a date range like so
INSERT OVERWRITE TABLE dwo_analysis.exp_shown
SELECT * FROM test
WHERE click_date
BETWEEN DATE_SUB(CURRENT_DATE, 2) AND CURRENT_DATE;
In my Oracle SQL Developer, i have a table with a column with DATE format. When i insert a new row into this table, and insert a new value in this column, it automatically suggestes me the current date with the current hour.
I would like that it automatically suggestes me current date, but with 00:00:00 hour . Is there some setting or parameter that i can set in my SQL Developer to have this result?
We can't able to insert 00:00:00 hours ... the hour value should be between 1 to 12...
we can use below query to insert 00:00:00 hours but the value will be changed to 12:00:00
INSERT INTO TABLE (DATE_COL) VALUES
( TO_DATE ('11/16/2017 00:00:00 ', 'MM/DD/YYYY HH24:MI:SS '));
It seems to me that your DATE column is set with a DEFAULT of SYSDATE. This means, for any INSERT operations which do not specify a value in your DATE column, the current date and time will populate for that row. However, if INSERT operations do specify a value in your DATE column, then the specified date value will supersede the DEFAULT of SYSDATE.
If an application is controlling INSERT operations on that table, then one solution is to ensure the application utilizes the TRUNC() function to obtain your desired results. For example:
INSERT INTO tbl_target
(
col_date,
col_value
)
VALUES
(
TRUNC(SYSDATE, 'DDD'),
5000
)
;
However, if there are multiple applications or interfaces where users could be inserting new rows into the table, (e.g. using Microsoft Access or users running INSERT statements via SQL Developer) and you can't force all of those interfaces to utilize the TRUNC() function on that column during insertion, then you need to look into other options.
If you can ensure via applications that INSERT operations will not actually reference the DATE, then you can simply ALTER the table so that the DATE column will have a DEFAULT of TRUNC(SYSDATE). A CHECK CONSTRAINT can be added for further integrity:
ALTER TABLE tbl_target
MODIFY
(
col_date DATE DEFAULT TRUNC(SYSDATE, 'DDD') NOT NULL
)
ADD
(
CONSTRAINT tbl_target_CHK_dt CHECK(col_date = TRUNC(col_date, 'DDD'))
)
;
However, if users still have the freedom to specify the DATE when inserting new rows, you will want to use a TRIGGER:
CREATE OR REPLACE TRIGGER tbl_target_biu_row
BEFORE INSERT OR UPDATE OF col_val
ON tbl_target
FOR EACH ROW
BEGIN
:NEW.col_date := TRUNC(SYSDATE, 'DDD');
END tbl_target_biu_row
;
This will take of needing to manage the application code of all external INSERT operations on the table. Keep in mind, the above trigger is also modifying the DATE column if a user updates the specified value column.
Table myfirst3 have 4 columns and 1.2 million records.
Table mtl_object_genealogy has over 10 million records.
Running the below code takes very long time. How to tune this code using with options?
WITH level1 as (
SELECT mln_parent.lot_number,
mln_parent.inventory_item_id,
gen.lot_num ,--fg_lot,
gen.segment1,
gen.rcv_date.
FROM mtl_lot_numbers mln_parent,
(SELECT MOG1.parent_object_id,
p.segment1,
p.lot_num,
p.rcv_date
FROM mtl_object_genealogy MOG1 ,
myfirst3 p
START WITH MOG1.object_id = p.gen_object_id
AND (MOG1.end_date_active IS NULL OR MOG1.end_date_active > SYSDATE)
CONNECT BY nocycle PRIOR MOG1.parent_object_id = MOG1.object_id
AND (MOG1.end_date_active IS NULL OR MOG1.end_date_active > SYSDATE)
UNION all
SELECT p1.gen_object_id,
p1.segment1,
p1.lot_num,
p1.rcv_date
FROM myfirst3 p1 ) gen
WHERE mln_parent.gen_object_id = gen.parent_object_id )
select /*+ NO_CPU_COSTING */ *
from level1;
execution plan
CREATE TABLE APPS.MYFIRST3
(
TO_ORGANIZATION_ID NUMBER,
LOT_NUM VARCHAR2(80 BYTE),
ITEM_ID NUMBER,
FROM_ORGANIZATION_ID NUMBER,
GEN_OBJECT_ID NUMBER,
SEGMENT1 VARCHAR2(40 BYTE),
RCV_DATE DATE
);
CREATE TABLE INV.MTL_OBJECT_GENEALOGY
(
OBJECT_ID NUMBER NOT NULL,
OBJECT_TYPE NUMBER NOT NULL,
PARENT_OBJECT_ID NUMBER NOT NULL,
START_DATE_ACTIVE DATE NOT NULL,
END_DATE_ACTIVE DATE,
GENEALOGY_ORIGIN NUMBER,
ORIGIN_TXN_ID NUMBER,
GENEALOGY_TYPE NUMBER,
);
CREATE INDEX INV.MTL_OBJECT_GENEALOGY_N1 ON INV.MTL_OBJECT_GENEALOGY(OBJECT_ID);
CREATE INDEX INV.MTL_OBJECT_GENEALOGY_N2 ON INV.MTL_OBJECT_GENEALOGY(PARENT_OBJECT_ID);
Your explain plan shows some very big numbers. The optimizer reckons the final result set will be about 3227,000,000,000 rows. Just returning that many rows will take some time.
All table accesses are Full Table Scans. As you have big tables that will eat time too.
As for improvements, it's pretty hard to for us understand the logic of your query. This is your data model, you business rules, your data. You haven't explained anything so all we can do is guess.
Why are you using the WITH clause? You only use the level result set once, so just have a regular FROM clause.
Why are you using UNION ALL? That operation just duplicates the records retrieved from myfirst3 ( all those values are already included as rows where MOG1.object_id = p.gen_object_id.
The MERGE JOIN CARTESIAN operation is interesting. Oracle uses it to implement transitive closure. It is an expensive operation but that's because treewalking a hierarchy is an expensive thing to do. It is unfortunate for you that you are generating all the parent-child relationships for a table with 27 million records. That's bad.
The full table scans aren't the problem. There are no filters on myfirst3 so obviously the database has to get all the records. If there is one parent for each myfirst3 record that's 10% of the contents mtl_object_genealogy so a full table scan would be efficient; but you're rolling up the entire hierarchy so it's like you're looking at a much greater chunk of the table.
Your indexes are irrelevant in the face of such numbers. What might help is a composite index on mtl_object_genealogy(OBJECT_ID, PARENT_OBJECT_ID, END_DATE_ACTIVE).
You want all the levels of PARENT_OBJECT_ID for the records in myfirst3. If you run this query often and mtl_object_genealogy is a slowly changing table you should consider materializing the transitive closure into a table which just has records for all the permutations of leaf records and parents.
To sum up:
Ditch the WITH clause
Drop the UNION ALL
Tune the tree-walk with a composite index (or materializing it)
I have to create a procedure which searches any recently added records and if there are then move them to ARCHIVE table.
This is my statement which filters recently added records
SELECT
CL_ID,
CL_NAME,
CL_SURNAME,
CL_PHONE,
VEH_ID,
VEH_REG_NO,
VEH_MODEL,
VEH_MAKE_YEAR,
WD_ID,
WORK_DESC,
INV_ID,
INV_SERIES,
INV_NUM,
INV_DATE,
INV_PRICE
FROM
CLIENT,
INVOICE,
VEHICLE,
WORKS,
WORKS_DONE
WHERE
Client.CL_ID=Invoice.INV_CL_ID and
Invoice.INV_CL_ID = Client.CL_ID and
Client.CL_ID = Vehicle.VEH_CL_ID and
Vehicle.VEH_ID = Works_Done.WD_VEH_ID and
Works_done.WD_INV_ID = Invoice.INV_ID and
WORKS_DONE.WD_WORK_ID = Works.WORK_ID and
Works_done. Timestamp >= sysdate -1;
You may need something like this (pseudo-code):
create or replace procedure moveRecords is
vLimitDate timestamp := systimestamp -1;
begin
insert into table2
select *
from table1
where your_date >= vLimitDate;
--
delete table1
where your_date >= vLimitDate;
end;
Here are the steps I've used for this sort of task in the past.
Create a global temporary table (GTT) to hold a set of ROWIDs
Perform a multitable direct path insert, which selects the rows to be archived from the source table and inserts their ROWIDs into the GTT and the rest of the data into the archive table.
Perform a delete from the source table, where the source table ROWID is in the GTT of rowids
Issue a commit.
The business with the GTT and the ROWIDs ensures that you have 100% guaranteed stability in the set of rows that you are selecting and then deleting from the source table, regardless of any changes that might occur between the start of your select and the start of your delete (other than someone causing a partitioned table row migration or shrinking the table).
You could alternatively achieve that through changing the transaction isolation level.
O.K. may be something like this...
The downside is - it can be slow for large tables.
The upside is that there is no dependence on date and time - so you can run it anytime and synchronize your archives with live data...
create or replace procedure archive is
begin
insert into archive_table
(
select * from main_table
minus
select * from archive_table
);
end;