I have a table on master site which only allow insert operation. I want to replicate the rows which are inserted recently so, need something which can track the last record replicated into the local site and perform the replication afterwards.
I have tried Oracle materialized view but still confuse that whether I use the Fast Refresh or Complete Refresh. I need all the newly inserted rows replicated in one transaction.
Are there any better approach to do that? Any help would be highly appreciated.
Thanks.
A fast refresh would copy incremental changes over the network but requires that a materialized view log be created on the master site on the source table. That adds some overhead to the inserts happening on the master table but would generally make the refresh more efficient.
A complete refresh would copy every row over the network every time the materialized view is refreshed. That is likely to be less efficient from a refresh perspective but there will be no overhead to inserts on the source table and the master site does not need to create a materialized view log.
Oracle provides a host of data replication technologies-- materialized views are the oldest and probably the least efficient but are relatively trivial to set up. Streams is a newer technology that has much lower overhead but is quite a bit more complex to set up. Golden Gate is the preferred replication technology today but that has extra licensing costs.
Related
I have an Oracle materialized view refreshed on demand. Since the view is pretty large, it takes a while to update. I am wondering: when do the updates really become visible to the queries that try read the view? Does RDBMS update the view in a buffer and then makes it visible when update is complete? Or will the changes gradually become visible to the reader? Is this possible to control this?
Materialized views can be refreshed on commit or on demand.
Oracle's documentation is quite good
https://docs.oracle.com/cd/B19306_01/server.102/b14200/statements_6002.htm#i2116410
To have materialized views refresh incrementally you must use the FAST option and you must have materialized view logs to capture changes on the source tables.
If you schedule the demanded refreshes frequently there will be less work to be done for each refresh. If you schedule them less frequently there will be more work to be done. You need to investigate how changes on the source tables are distributed over time to choose well.
Changes will be visible to users when the refresh (whatever size) commits.
I am new to Teradata & fortunately got a chance to work on both DDL-DML statements.
One thing I observed is Teradata is very slow when time comes to UPDATE the data in a table having large number of records.
The simplest way I found on the Google to perform this update is to write an INSERT-SELECT statement with a CASE on column holding values to be update with new values.
But what when this situation arrives in Data Warehouse environment, when we need to update multiple columns from a table holding millions of rows ?
Which would be the best approach to follow ?
INSERT-SELECT only OR MERGE-UPDATE OR MLOAD ?
Not sure if any of the above approach is not used for this UPDATE operation.
Thank you in advance!
At enterprise level, we expect volumes to be huge and updates are often part of some scheduled jobs/scripts.
With huge volume of data, Updates comes as a costly operation that involve risk of blocking table for some time in case the update fails (due to fallback journal). Although scripts are tested well, and failures seldom happen in production environments, it's always better to have data that needs to be updated loaded to a temporary table in required form and inserted back to same table after deleting matching records to maintain SCD-1 (Where we don't maintain history).
I have materialized view in my own schema and the materialized view source is the other schema table but , this master table has 900 rows and i did some dml operations on this master table after that i will refresh this materialized view as you know , and i did some resarch " how can i refresh my own materialized view " and it says " you can do complete refresh or fast refresh " but i didnt understand these solutions meaning so my question is simple ;
What is the difference between complete refresh and fast refresh in MV ?
P.S:If my master table has 1 million or more rows , which one i should choose? (fast or complete)
Thank you for your reply.
"Complete Refresh" means you truncate entire materialized view and insert new data.
"Fast Refresh" means you update (or insert/delete) only the rows which have been changed on master tables.
And just as information "Force Refresh" mean, Oracle tries to make a Fast Refresh and if this is not possible then do "Complete Refresh"
Usually Fast Refresh is much faster than Complete Refresh but it has restrictions. You have to define MATERIALIZED VIEW LOG on master tables.
Here is a full list of restrictions General Restrictions on Fast Refresh, there are quite many.
As always it depends, so if possible try both and measure for your application. As a general rule the fast refresh is likely to be much faster provided that only a small part of the data has changed. If all the data has changed a complete refresh is better.
With the fast refresh Oracle keeps track of the changes for the base tables and applies the changes to the materialized view when it is refreshed. A complete refresh on the other hand rebuilds the materialized view from scratch. With millions of rows that will be expensive, but again it is impossible to pick the best option without knowing more about your application.
We have to create a materialised view in our database from a remote database which is currently in production. The view has about 5 crore records and is taking a long time. In between , the connection has snapped once and not even a single record persisted in our database. Since the remote database is production server, we get a very limited window to create the view.
My question is, can we have something like auto commit /auto start from where left at last time while the view is created so that we don't have to do the entire operation in one go ?
We are tying to make the query such that records can be fetched in smaller numbers as a alternate. But the data is read only for us and doesn't really have a where clause at this point which we can use.
Due to sensitive nature of data, I cannot post the view structure or the query.
No, you can not commit during the process of creating the view. But why don't you import the data to a table instead of a view? This would provide the possibility to commit in between. Furthermore this might provide the possibility, to load only the delta of changes maybe on daily basis - this would reduce the required time dramatically.
I have been researching using materialized views for data aggregation and reporting purposes for a company that is largely centered around transactions (using an Oracle db). The current reporting system is dependent upon a series of views that obscure a lot of the complex data logic of the application. These views place a heavy burden on the system when they are called.
We are interested in using the "fast refresh" for incremental updates to perform some of the complex query logic prior to use in reporting; however, there is a concern within the organization that the materialized view logs (which are required for this fast refresh) will have an impact on our current transaction performance in the database. This performance is very essential to our organization therefore there is a great fear of any change.
Here is an example of the type of materialized view log we would need to implement:
create materialized view log on transaction
with rowid, sequence(transaction_id,account_id,order_id,currency_id,price,transaction_date,payment_processor_id)
including new values;
We would not be using the "on commit" clause for updates but rather the "on demand" clause in creation of the view, as we understand this would have a performance impact.
Will implementing this type of logging affect database transaction performance? I imagine that it must slightly affect performance as there is an additional write procedure (to the log) that is wrapped in the commit, but I cannot find any reference to this in the Oracle documentation. Any literature or advice on this subject would be greatly appreciated.
Thanks for your help!
Yes, there will be an impact. The materialized log needs to be maintained synchronously so the transactions will need to insert a new row into the materialized view log for every row that is modified in the base table. How great an impact depends heavily on the system. If your system is I/O bound and you've optimized it so that physically writing the changes to the base table is a significant fraction of the wait time, the impact will be much greater than if your system is CPU bound and most of your wait time is spent reading data or performing computations.
If you are really concerned about the performance of the OLTP system, it would make sense to offload reporting to a different database on a different server. You can replicate the data to the reporting server using Streams (or GoldenGate if you can afford the additional licensing) which will have less of an impact on the source than materialized views because the redo information can be read asynchronously (and can be read on the reporting server rather than putting that workload on the production server). You could then define materialized views on the reporting server where they won't have any impact on the OLTP server. Or you could create a logical standby database as your reporting server and create the materialized views there. Either way, moving the reporting workload off the production server and reading the redo data asynchronously will protect the performance of the production server.