I have declared MV logs on 4 source tables (database A ) and then I have created 4 Materialized Views on this tables (Database B using dblinks ) with FAST Refresh which works fine. But I need to use this Materialized view which would be refreshed every 5 minutes and I want to extract recent changes i.e Delta only and then apply to 4-5 different target tables. This are big tables and target tables needs to refreshed instantly how can I get just delta ?
May seem obvious to some but I am not able to design reliable process/mechanism. Any help/suggestions ?
Add column to your MV with default sysdate, for example "date_delta". And then use this case: WHERE date_delta >= sysdate - 5/1440 (5/1440 - it's 5 minutes)
Related
We created a materialized view with fast refresh on commit. We have seven base tables for MV and corresponding logs created too. We have Golden Gate configured for replication on base tables. Due to load on MV Golden Gate job on base tables getting abended.
create materialized view MV_SPON_PERMIT_RES
refresh fast on commit
as
SELECT
p.rowid p_rowid,
pd.rowid pd_rowid,
ep.rowid ep_rowid,
tl.rowid tl_rowid,
la.rowid la_rowid,
ip.rowid ip_rowid,
pp.rowid pp_rowid,
on tableA,tableB,TableC
union all
p.rowid p_rowid,
pd.rowid pd_rowid,
ep.rowid ep_rowid,
tl.rowid tl_rowid,
la.rowid la_rowid,
ip.rowid ip_rowid,
pp.rowid pp_rowid
on tableX,tableY,tableZ
Problem: fast refresh on commit is making Golden Gate job abend:
DELETE FROM "VISION_INTG"."MV_SPON_PERMIT_RES" SNA$
WHERE "PP_ROWID" IN (SELECT/*+ NO_MERGE HASH_SJ */ *
FROM (SELECT CHARTOROWID ("MAS$"."M_ROW$$") RID$
FROM "VISION_INTG"."MLOG$test" "MAS$"
WHERE "MAS$".SNAPTIME$$ > :B_ST0) MAS$)
AND "FILE_TYPE" = 'Permit'
Please help me in this.
If we are using Golden gate on replicating the data between layers then we cant use MV with refresh fast commit. we can go MV on demand which should refresh data in incremental way. If incremental way refresh is taking time then we need to check the system parameter.
_mv_refresh_use_stats
As per oracle documentation it should be set to false
alter system set "_mv_refresh_use_stats"=FALSE;
I have read quite a lot for past few hours about refreshing MV in Oracle, but I cannot still find an answer to my question. Imagine I have a MV view on top of a table with change logs. So that there are three records in this MV:
COL_ID, COL1
1, "OLD"
2, "OLD"
3, "OLD"
Now let's say the value for COL1 has changed to "EDITED" for record 1 in the table used to create MV. I want to perform fast, in-place refresh to update MV as fast as possible. In real life example with around 50M records, this would take around 3 minutes to refresh.
Imagine situation.
Refresh process is still running (there are still records that have not been amended in the MV).
Neverthelss, record with ID = 1 has been already processed so that it has value of "EDITED" in MV.
In another session, a query to MV is executed to get the value of record with ID = 1.
As the result will record with value "OLD" or "EDITED" be given?
I understand that because this is an in-place refresh after this record is processed by refresh mechanism the value in the materialized view will reflect the value in origin table ("EDITED"). But is there any mechanism (like undo logs) that would make the whole refresh atomic? And by this I mean that unless all amendments are done to the materialized views (unless the refresh process is done) if user queries for rows that have been modified in the process of undergoing refresh, he/she will be presented to an old, cached value - before the change.
I presume that this bevahiour is true for out-of-place refresh, but since the former seems to be working way more efficient in terms of time consumption I was curious whether this is also true for the in-place transformation. If not by default, is there any way to force this atomic behaviour?
----- [EDIT]
I run the test following test to see if during the refresh process the results I get from the materialized view will change gradually.
-- create table
create table MV_REFRESH_ATOMICITY_TEST
(
id NUMBER,
value NUMBER
)
-- populate initial data
-- delete from MV_REFRESH_ATOMICITY_TEST
declare
begin
for i in 1..10000000 loop
insert into MV_REFRESH_ATOMICITY_TEST values(i, 0);
end loop;
end;
-- check if equal zero and 1M
select sum(value) from MV_REFRESH_ATOMICITY_TEST
select to_char(count(*),'999,999,999') as COUNT from MV_REFRESH_ATOMICITY_TEST
-- create mv logs on the table
-- drop materialized view log on MV_REFRESH_ATOMICITY_TEST;
create materialized view log on MV_REFRESH_ATOMICITY_TEST with rowid;
-- create mv on top
-- drop materialized view MV_REFRESH_ATOMICITY_TEST_MV
create materialized view MV_REFRESH_ATOMICITY_TEST_MV
refresh fast on demand with rowid
as
select
fact.*,
fact.ROWID "FACT_ROWID"
from
MV_REFRESH_ATOMICITY_TEST fact
-- check if equals zero and 10M
select sum(value) from MV_REFRESH_ATOMICITY_TEST_MV
select to_char(count(*),'999,999,999') as COUNT from MV_REFRESH_ATOMICITY_TEST_MV
-- change value for first million records, 1 milion records in the middle, last milion of records
update MV_REFRESH_ATOMICITY_TEST set value = 1 where id between 1 and 1000000
update MV_REFRESH_ATOMICITY_TEST set value = 1 where id between 5000001 and 6000000
update MV_REFRESH_ATOMICITY_TEST set value = 1 where id between 9000001 and 10000000
-- check if equals 3.000.000
select to_char(sum(value),'999,999,999') as "SUM" from MV_REFRESH_ATOMICITY_TEST
-- check if equals 3.000.000
select to_char(count(*),'999,999,999') from MLOG$_MV_REFRESH_ATOMICITY;
--select * from MLOG$_MV_REFRESH_ATOMICITY;
-- while refreshing mv
-- exec dbms_mview.refresh('MV_REFRESH_ATOMICITY_TEST_MV', 'F');
-- below sum should be equal 0
select
( select sum(value) from MV_REFRESH_ATOMICITY_TEST_MV ) "SUM",
( select count(*) from MV_REFRESH_ATOMICITY_TEST_MV ) "NUMBER OF RECORDS"
from dual
So what I discovered by constantly executing the last select statement was that the only time that the SUM value changed, it already changed by 3M, meaning all the records have been changed in one go - atomically.
Nevertheless, I am not 100% percent sure I can trust this experiment as at some point it took around 40s to execute these select queries. The whole refresh statement took 911s to execute.
[EDIT]
This question has been marked as a possible duplicate of this thread. The other thread does respond to a similar problem, but for a complete-refresh which to my understanding is performed in very different way than fast-refresh which is the case here. Therefore I am not sure whether the same explanation can be applied here.
As far I can see from Oracle documentation (http://docs.oracle.com/cd/B19306_01/server.102/b14226/repmview.htm#i31171) - all refresh is done in atomic way:
A materialized view's data does not necessarily match the current data of its master table or master materialized view at all times. A materialized view is a transactionally (read) consistent reflection of its master as the data existed at a specific point in time (that is, at creation or when a refresh occurs).
Oracle provides even greater read consistency with materialized view groups:
To preserve referential integrity and transactional (read) consistency among multiple materialized views, Oracle has the ability to refresh individual materialized views as part of a refresh group. After refreshing all of the materialized views in a refresh group, the data of all materialized views in the group correspond to the same transactionally consistent point in time.
Hi I have created 5 MViews for tables in Remote DB 2 days back in production environment. Out of that one Mview, at the time of creation it worked fine. But 2 days after(today) the MView's last refresh date is still showing the day before yesterday's date.
I checked on the DBA_JOBS:
LAST_DATE,NEXT_DATE,BROKEN,INTERVAL,FAILURES,WHAT
NULL,"23-JUL-13 01:00:00", N,"trunc(SYSDATE+1)+1/24",12,"dbms_refresh.refresh('"OCCSS_ENTMT_HK"."MV_SCI_STD_CODE_VALUE"');"
MView create statement:
CREATE MATERIALIZED VIEW MV_SCI_STD_CODE_VALUE BUILD IMMEDIATE USING INDEX REFRESH FORCE ON DEMAND START WITH SYSDATE+0 NEXT TRUNC(SYSDATE+1)+1/24 WITH PRIMARY KEY USING DEFAULT LOCAL ROLLBACK SEGMENT USING ENFORCED CONSTRAINTS DISABLE QUERY REWRITE AS SELECT STV_STD_CODE_NUM STANDARD_CODE,
STV_STD_CODE_VALUE STANDARD_CODE_VALUE,
STV_STD_CODE_VALUE_DESC DESCRIPTION,
stv_prnt_std_code_value parent_code FROM SCIADMIN.P03_STD_CODE_VALUE#SCI_LINK02.HK.BT.COM P03 WHERE STV_STD_CODE_NUM IN ('19','31','54','5','6','300') AND p03.update_status_ind <> 'D';
COMMENT ON MATERIALIZED VIEW MV_SCI_STD_CODE_VALUE IS 'snapshot table for snapshot MV_SCI_STD_CODE_VALUE';
The Mview is scheduled to run once in one day. But the failures shown is 12.
I couldn't find any jobs for the above in DBMS_JOBS_RUNNING.
I need to investigate this issue. What would be the possible issues which could have caused this failure?
I have a Materialized view in Oracle that contains a LEFT JOIN which takes a very long time to update. When I update the underlying table it takes 63914.765 s to run (yes that is almost 17 hours).
I am using a LEFT JOIN on the same table, because I want to pivot the data from rows to columns. The pivot command is not available in this Oracle version, and using a GROUP BY + CASE is not allowed on a FAST REFRESH Materialized View.
The Materialized View Log looks like this:
CREATE MATERIALIZED VIEW LOG ON Programmes_Titles
WITH PRIMARY KEY, rowid
INCLUDING NEW Values;
The Materialized View itself looks like this (it contains 700000 rows, the Programmes_Titles table contains 900000 rows):
CREATE MATERIALIZED VIEW Mv_Web_Programmes
REFRESH FAST ON COMMIT
AS
SELECT
t1.ProgrammeId,
t1.Title as MainTitle,
t2.Title as SecondaryTitle,
--Primary key
t1.Title_Id as t1_titleId,
t2.Title_Id as t2_titleId,
t1.rowid as t1_rowid,
t2.rowid as t2_rowid
FROM
Programmes_Titles t1,
Programmes_Titles t2
WHERE
t1.Titles_Group_Type = 'mainTitle'
AND t1.Programme_Id = t2.Programme_Id(+) AND t2.Titles_Group_Type(+) = 'secondaryTitle'
The UPDATE statement I use is this:
UPDATE Programmes_Titles
SET Title = 'New title'
WHERE rowid = 'AAAL4cAAEAAAftTABB'
This UPDATE statement takes 17 hours.
When using an INNER JOIN (remove the (+)'s) it takes milliseconds.
I also tried adding INDEXES on the Mv_Web_Programmes Materialized View, but that did not seem to help either. (It still runs for more than a minute, which is way to slow, I am not waiting 17 hours after every change, so it might improved the UPDATE)
So my question is: Why does is take such a long time to UPDATE the underlying table? How can I improve this?
I've managed to reproduce your problem on a 10.2.0.3 instance. The self- and outer-join seems to be the major problem (although with indexes on every column of the MV it finally did update in under a minute).
At first I thought you could use an aggregate MV:
SQL> CREATE MATERIALIZED VIEW LOG ON Programmes_Titles
2 WITH PRIMARY KEY, ROWID (programmeId, Titles_Group_Type, title)
3 INCLUDING NEW Values;
Materialized view log created
SQL> CREATE MATERIALIZED VIEW Mv_Web_Programmes
2 REFRESH FAST ON COMMIT
3 AS
4 SELECT ProgrammeId,
5 MAX(decode(t1.Titles_Group_Type, 'mainTitle', t1.Title)) MainTl,
6 MAX(decode(t1.Titles_Group_Type, 'secondaryTitle', t1.Title)) SecTl
7 FROM Programmes_Titles t1
8 GROUP BY ProgrammeId;
Materialized view created
Unfortunately, as you have noticed, as of 10g a MV that contains MIN or MAX can only be fast-refreshed on commit after insert (so called insert-only MV). The above solution would not work for update/delete (the MV would have to be refreshed manually).
You could trace your session and open the trace file to see what SQL query gets executed so that you can find if you can optimize it via indexes.
We too faced this issue recently on Oracle 11.2.0.3
In our case, it was unavoidable to to remove an 'OUTER JOIN' due to functional impact.
On investigation, it was found that Oracle was adding a nasty HASH_SH (Hash Semi Join) hint with MV refresh DML.
Nothing worked including things mentioned in following blog-
http://www.adellera.it/blog/2010/03/11/fast-refresh-of-join-only-mvs-_mv_refresh_use_stats-and-locking-log-stats/#comment-2975
In the end, a hidden hint worked...(though in general, it should be avoided by making change in application if possible)
Oracle Doc ID 1949537.1 suggests that setting the hidden _mv_refresh_use_hash_sj parameter to FALSE should prevent it using that hint.
alter session set "_mv_refresh_use_hash_sj"=FALSE;
That stopped CBO using the HASH_SJ hint.
Posting it here in the interests of others.
I have a self referencing table in Oracle 9i, and a view that gets data from it:
CREATE OR REPLACE VIEW config AS
SELECT c.node_id,
c.parent_node_id,
c.config_key,
c.config_value,
(SELECT c2.config_key
FROM vera.config_tab c2
WHERE c2.node_id = c.parent_node_id) AS parent_config_key,
sys_connect_by_path(config_key, '.') path,
sys_connect_by_path(config_key, '->') php_notation
FROM config_tab c
CONNECT BY c.parent_node_id = PRIOR c.node_id
START WITH c.parent_node_id IS NULL
ORDER BY LEVEL DESC
The table stores configuration for PHP application. Now I need to use same config in oracle view.
I would like to select some values from the view by path, but unfortunately this takes 0,15s so it's unacceptable cost.
SELECT * FROM some_table
WHERE some_column IN (
SELECT config_value FROM config_tab WHERE path = 'a.path.to.config'
)
At first I thought of a function index on sys_connect_by_path, but it is impossible, as it needs also CONNECT BY clause.
Any suggestions how can I emulate an index on the path column from the 'config' view?
If your data doesn't change frequently in the config_tab, you could use a materialized view with the same query as your view. You could then index the path column of your materialized view.
CREATE MATERIALIZED VIEW config
REFRESH COMPLETE ON DEMAND
AS <your_query>;
CREATE INDEX ix_config_path ON config (path);
Since this is a complex query, you would need to do a full refresh of your materialized view every time the base table is updated so that the data in the MV doesn't become stale.
Update
Your column path will be defined as a VARCHAR2(4000). You could limit the size of this column in order to index it. In your query, replace sys_connect_by_path(...) by SUBSTR(sys_connect_by_path(..., 1, 1000) for example.
You won't be able to use REFRESH ON COMMIT on a complex MV. A simple trigger won't work. You will have to modify the code that updates your base table to include a refresh somehow, I don't know if this is practical in your environment.
You could also use a trigger that submits a job that will refresh the MV. The job will execute once you commit (this is a feature of dbms_job). This is more complex since you will have to check that you only trigger the job once per transaction (using a package variable for example). Again, this is only practical if you don't update the base table frequently.