If I have table T with 10 columns and create materialized view log with only 3 of them as well as materialized view with those 3, why when I updated ANY column in the table (besides those in the log) the record is inserted into MLOG$_T ? Is there a way to avoid?
Thanks
No, you can't avoid it. The primary purpose of the log is to identify the rows that have changed, not the changed data itself (although you can include that in the log as well). The process for maintaining the log doesn't know anything about the view(s) that might be created that will use the logs, so it has to maintain the log for any change in the table.
This of it this way: suppose you later create another materialized view on the same table that uses other columns. If the log is to be useful, it must maintain information about changes for those rows as well. After all, you can only create one log per table.
Related
Having a structure where there is a base table, then a materialized view base_mv that aggregates sending the result TO an AggregatedMergeTree table base_agg_by_id. Then we have a view over this final table. base_unique. Similarly as in this blog post](https://www.altinity.com/blog/clickhouse-continues-to-crush-time-series).
However, if I delete from base, I would expect the base_mv would trigger the mutation and then act on it, and reflected on the base_agg_by_id, but it doesn't.
Is this the expected behaviour? How to DELETE in such a schema?
I've seen here that in MVs that keep data you can act on .inner tables. However in this case, since the table is from an AggregatedMergeTree and its fields are defined as functions (e.g. AggregateFunction(argMax, String, DateTime) ), I cannot apply a deletion via a value such as ALTER base_agg_by_id DELETE WHERE field = 'myval'.
Note. For the record, we have these tables in a replicated environment using Replicated* engine: base_d, base_agg_by_id_d, base_unique_d
Mutations are not propagated to materialized views.
The reason is very simple: it not possible in common case. And even in cases when it is theoretically possible it can be very expensive operation.
For example, let's say you're deleting one record from the table which references some userid. And your materialized view contains uniqState( userid ). Data structures used for calculating uniqState don't support 'remove' operation; but even if they would - the is no way to decide if that userid should be removed or not without rereading whole data for the partition again because that userid could be seen in other records too.
So in general case, you need to refill the whole partition for your AggregatedMergeTree.
I.e. something like (daily partitioning case):
ALTER amt_table DROP PARTITION '2019-03-01';
-- use same select as in your materialized view
INSERT INTO amt_table SELECT ... WHERE date = '2019-03-01';
We have a table TB_1 which has monthly and weekly data partitioned by monthly and weekly columns. We also have materialized view MV_1 inherited from the table TB_1.
We would like to refresh the materialized view by certain weekly or monthly basis.
Not sure how we can filter out weekly or monthly changes from all the changes captured in the materialized log for partial refresh.
Right now we are thinking to have a flag column in TB_1. By clearing the materialized log and updating the flag, We think we can achieve this.
Is there anyway efficient way than the process for partial refresh on specific criteria?
We have a similar case here: the solution we found was to partition the materialized view month by month (using PCT ). If you have this as an option in your licensing, this could be a solution. Then you must partition the "details tables", TB_1, and probably the "details tables" of MV_1.
execute dbms_mview.refresh(
list => 'your_partitioned_mview'
, method => 'P' -- this is where PCT is specified
, atomic_refresh => false
);
There are also other solution on Is it possible to partially refresh a materialized view in Oracle? .
edit:
I'd say the solution of a fast refresh using a "to be refreshed" flag is worth being tried. Not sure you need to clear the Mview log beforehand. Just changing the value of the flag for the record to be updated should work. Here is a nice howto I found.
If you have Oracle 12.2, they introduced real-time Mviews, which might be what you are looking for...
Oracle 12.2 introduced the concept of real-time materialized views, which allow a statement-level wind-forward of a stale materialised view, making the data appear fresh to the statement. This wind-forward is based on changes computed using materialized view logs, similar to a conventional fast refresh, but the operation only affect the current statement. The changes are not persisted in the materialized view, so a conventional refresh is still required at some point.
#use416, please keep us posted for what actually worked for your case.
Something failed while we were swapping disks for our ClickHouse database. When ClickHouse started, I had to attach all the tables as they were not there via ATTACH TABLE IF NOT EXISTS ....
Is there a way to do the same for materialized views? I couldn't find a way how to do that and when I try to create it from scratch (CREATE MATERIALIZED VIEW IF NOT EXISTS ..., ClickHouse says:
Data directory for table already containing data parts - probably it
was unclean DROP table or manual intervention. You must either clear
directory by hand or use ATTACH TABLE instead of CREATE TABLE if you
need to use that parts.
So the files are still there but dunno how to attach the view.
You need to attach the ".inner." table first.
Materialized views do not store data, they create a special table with the engine that you choose when you create the view. The name of that table is ".inner.the_name_of_the_view".
So you need to attach that table first, and then attach the materialized view.
When attaching other tables and restarting the ClickHouse server, the views got attached automatically. I was trying to attach .inner tables too but it didn't let me.
Good day,
Is it technically possible to create a materialized view against a another view in the master database (as opposed to a solid table)?
My DBA advises that oracle does not allow creation of materialized view logs against a view. IMO a view is pretty much the same as a table, so it ought to be possible.
Has anyone ever done this successfully (Oracle 11g).
The documentation is pretty clear about this:
Restrictions on Master Tables of Materialized View Logs
The following restrictions apply to master tables of materialized view logs:
You cannot create a materialized view log for a temporary table or for a view.
You cannot create a materialized view log for a master table with a virtual column.
A view is not pretty much the same as a table. It's a stored query, so it has no storage and DML is only possible through instead-of triggers. It doesn't have the basis for noticing an underlying data change.
Just thinking about it, in order to support a view log - even for a simple single-table view - an update to the view's base table would have to check if there were any views; check if any of those had materialized view logs; and then work out if the change needed to be logged - which would mean executing each view's query and looking at the entire result set. Imagine doing that for every change to the underlying table. And then imagine a more complicated view query, with aggregates or multiple tables, etc.
You'd effectively be materialising the view query to decide whether was a change that needed to be logged, to update an actual materialized view. This is partly what actual materialized views do - stop you having to repeatedly re-execute an expensive view query by tracking changes on the master table(s).
You have to create the materialized view logs on the (normal) view's base tables.
I have a very complex Oracle view based on other materialized views, regular views as well as some tables (I can't "fast refresh" it). Most of the time, existing records in this view are based on a date and are "stable", with new record sets having new dates.
Occasionally, I receive back-dates. I know what those are and how to deal with them if I were maintaining a table, but I would like to keep this a "view". A complete refresh would take around 30 minutes, but it only takes 25 seconds for any given date.
Can I specify that only a portion of a materialized view should be updated (i.e. the affected dates)?
Do I have to scrap the view and use a table and a procedure to populate or refresh a given date in that table?
Partition by date as in answer 3 (skaffman).
You could just do the refresh of a normal mv(table_refreshed below) and than use the exchange keyword i.e.
ALTER TABLE all_partitions
EXCHANGE PARTITION to_calculate
WITH TABLE table_refreshed
WITHOUT VALIDATION
UPDATE GLOBAL INDEXES;
After more reading and judging by the lack of answers to this question, I come come to the conclusion that it is not possible to refresh a single partition of a materialized view.
If you can give a syntax example that proves otherwise, I will happily mark your answer the accepted one.
To others who might find this questions useful in the future: you might also want to know that in Oracle 10g, refreshing a partition (or any mview) will cause Oracle to issue DELETE, followed by INSERT.
If this is giving you performance problems (like me), there is an option to use atomic_refresh => false, which will TRUNCATE, then INSERT /*+APPEND*/.
I have been able to refresh a single partition of a materialized view with partition change tracking.
It seems to require that the view is created with REFRESH FAST WITH ROWID option and DBMS_MVIEW.REFRESH is called with 'P' method.
You can partition materialized views just as you can with normal tables. Partition your mview by date, and then you can refresh only the required partition.