We are using Oracle Materialized view to extract data for a reporting system. The setting of the MV is to refresh on every commit.
After migrating to Oracle 19 form Oracle 12 system was very slow for few days and eventually became unresponsive after few days due to various bugs in Oracle 19 related to Materialized View.
We did a workaround of changing MV setting from on commit to OnDemand every 5 min to solve this issue.
The parent table on which this view is created gets updated every second and has 2 million records.
Having a MV on a table which is updated so frequently is fundamentally not a good design.
We don't want to depend on Oracle patches to fix this and planning to redesign this table.
Can I change this view to simple view and possibly apply all possible ways to improve query response time and expect that I will have view query returned in few seconds given I have 2 mil records in Parent table.
Assume that table has basic datatypes only and view is reading about 5-6 fields.
We tried oncommit to ondemand as a workaround but as per the business need, we need committed data for the reporting system and cannot be delayed more than one min.
Related
In our oracle query, I used materialized view to improve the performance and saw a drastic change in performance time of query, response time is in .17 seconds now rather 30 seconds so its working in case we are working on historical data set and need to have one time refresh of materialized view but now requirement is if base table of materialized view would be updating frequently in that case it would be working as the refresh activity of materialized view would be consuming time as well, currently the base table has around 170000 records but in future definitely going to increase so is there any solution or option to resolve this. As i have tried to apply Indexes as well on the table but not able to achieve response time within .17 sec or around the last response time based on index creation was 18 sec.
Looking into getting near-realtime analytics data from bigquery, and considering costs vs. accuracy, it seems like using materialized view might be a great win.
Considering "near real time" will change to a minimum of 1 minute refresh_interval_minutes, my main concern is that while from documentation it seems that the query will be only on the delta data, the billing will be on the "standard" minimum of 10MB per table.
As I see it, if this minimum is being forced, it dismiss using materialized view as a valid solution for near-realtime.
I would have used "standard" caching over the queries, but caching does not work when querying table with "buffered data", as far as I understand it.
Please advise,
Shushu
The following are key characteristics of BigQuery Materialized Views:
Zero maintenance: A materialized view is recomputed in background once the base table has changed. All incremental data changes from the base tables are automatically added to the materialized views. No user inputs are required.
In other words it means that incremental changes like streaming data into, is automatically added to the materialized view. That means you don't need to set 1 minute refresh period. The maximum refresh frequency cap is 7 days. You could set it to 7 days, or even disable manual refresh as you don't have deletes.
Here is current scenario - We have 3 tables in Oracle DB (with millions of records) which are being used to generate SSRS reports.
These reports are displaying complex data calculation such as deviations, median etc.
SSRS fetch data using stored procs in oracle (joining all the 3 tables) based on date parameters
Calculations are performed in SSRS and data is displayed in tables and charts
Now, for small date duration, report is getting generated quite fast, so no issues there.
When date range is big like a week or 2-3 months, report takes lot of time to process and most of the time it gets timed out as well.
To resolve this issue, I am thinking to remove calculations from SSRS and move them to DB level. Where we can have pre-calculated data
which will be served to SSRS reports for faster report generation.
In order to do this, I can see 2 options -
Oracle Materialized Views
SSAS Cube
I have never used Materialized Views before, so I am a bit skeptical about its performance specially FAST REFRESH issues.
What way would you prefer? MV or SSAS or mix of both?
Data models (SSAS) are great for organizing data, consolidating business logic, and defining how calculations behave in different scopes. They are generally faster to query than the raw data which is what you currently have. There is some caching involved, but you still have to query the data and wait for it to be processed. Models are also most appropriate when you have multiple reports that will be using a common set of data.
With a materialized view, you can shift the heavy lifting of calculation time to the scheduled refresh. Think of it as essentially the same as creating a new table that is refreshed by a procedure. This will greatly improve query times for the report especially if the date column you're filtering on is indexed. Also, the development and maintenance requirements are much lower for this than a model.
So, based on your specifications I would suggest the materialized view.
I would concur with the Materialized View (MV) approach. Depending on the amount and type (insert vs update vs delete) would determine if a fast refresh is possible or practical.
Counter intuitively, a FULL refresh is often a better approach, since you can better take advantage of set based SQL processing, together with parallelism to build the MV.
We have to create a materialised view in our database from a remote database which is currently in production. The view has about 5 crore records and is taking a long time. In between , the connection has snapped once and not even a single record persisted in our database. Since the remote database is production server, we get a very limited window to create the view.
My question is, can we have something like auto commit /auto start from where left at last time while the view is created so that we don't have to do the entire operation in one go ?
We are tying to make the query such that records can be fetched in smaller numbers as a alternate. But the data is read only for us and doesn't really have a where clause at this point which we can use.
Due to sensitive nature of data, I cannot post the view structure or the query.
No, you can not commit during the process of creating the view. But why don't you import the data to a table instead of a view? This would provide the possibility to commit in between. Furthermore this might provide the possibility, to load only the delta of changes maybe on daily basis - this would reduce the required time dramatically.
I am creating an oracle view basically as a read-only summary of data from other tables. It is not necessary to update the view every time if there is any updates to associated tables otherwise the workload will be huge and a waste. It is better to refresh the view every 5 minutes and then the front end can receive the latest data that will be all.
However, how to setup the oracle view refreshing interval? Do we have to settle this constraints in PL/SQL?