I have one big table A who has PK (C1, C2, C3) and many other columns, to make the select faster, a smaller table B was created with PK (C1, C2). So we can do a select by joining the two tables to find a row in A.
But the problem now is that it can happen that if data is corrupted in B which results in a joint select returns nothing but we still have a row in A.
Am I doing something wrong with this design and how can I ensure the data in those two tables are consistent?
Thanks a lot.
Standard way - if those tables are in a master-detail relationship - is to create a foreign key constraint which will prevent deleting master if details exist.
If you can fix it now, do it - then create the constraint.
If you can't, then create foreign key constraint using INITIALLY DEFERRED DEFERRABLE option so that current values aren't checked, but future DML will be.
Finally, to fetch data although certain rows don't exist any more, use outer join.
"Am I doing something wrong with this design"
Well it's hard to be sure without more details about your scenario but probably you just needed a non-unique index on A(C1, C2).
Although I would like to see some benchmarking which proves an index-range scan on your primary key index was not up to the job. Especially as it seems likely the join on table B is using that access path.
Performance tuning an Oracle database is a matter of understanding and juggling many variables. It's not just a case of "bung on another index". We need to understand what the database is actually doing and why the optimiser made that choice. So, please read this post on asking Oracle tuning questions which will give you some insight into how to approach query optimisation.
Related
Do you know if there's a way to join two tables by, for
example, using a foreign key constraint like in MySQL (I don't seem to
find anything about this) ? If not, is there a replacement ?
Thanks!
I interpret your question as below -
Is there a way to limit the values that can be used on the tableX to only the IDs that exist on the tableY? For example via using a foreign key constraint like in MySQL!
BigQuery does not provide any direct mechanism for this to happen.
But you can easily achieve this by yourself.
For example, assume you need to insert some data to tableX, but you want to make sure that only those rows will be inserted where id in that new data is in tableY
So, you can "enforce" this via below query
#standardSQL
SELECT n.*
FROM newData AS n
JOIN tableY AS y
ON n.id = y.id
... you can run this query with tableX as destination and ONLY needed rows will be inserted
Hope you got an idea
Also you can check existing related feature requests -
https://issuetracker.google.com/issues/35906045
https://issuetracker.google.com/issues/35906043
Since you asked two questions (Stack Overflow suggests asking 1 question per question), I'll answer one:
Also, do you know if there's a way to join two tables by, for example,
using a foreign key
In BigQuery you can join tables by any key - even by keys defined on the fly (this is pretty useful when you need to join two tables from different datasets that choose to encode identical values in different ways).
Why would you need a foreign key to do these joins?
In our one of production database, we have 4 column table and there are no PK,UK constraints on it. only one notnull constraint on one column. The inserts are slow on this table and when I checked the indexes , there is one index which is built on all columns.
It is a normal table and not IOT. I really don't see a need of all column index, but wondering why the developers has created it?
Appreciate your thoughts?
It might be usefull, i.e. if you (mainly) query all columns oracle doesn't have to access the table at all, but can get all the data from the index. Though inserts take longer because a larger index has to be maintained by the dbms everytime.
One case where it could be useful is,
Say for example, you are trying to check the existence of records in this table and for that you have to have joins on all four columns. So in such a case if you have written a correlated query like below,
SELECT <something>
FROM table_1 t1
WHERE EXISTS
(SELECT 1 FROM table_t2 t2 where t1.c1=t2.c1 and t1.c2=t2.c2 and t1.c3=t2.c3 and t1.c4=t2.c4)
Apart from above case, it looks an error to me from developer's side.
Indexes are good to better query optimization but causes slow updates/inserts because the indexes needs to be updated at each modification.
If these tables first use is querying and inserts happens only in a specific periods like a batch at the beginning or the end of the day only, then you can remove the indexes before updating tables and then restore them.
In addition, all the queries all these tables need to be analysed to see which indexes are useful and which are not?
Anyway, You need to ask developers before removing these indexes.
I have a situation like this:
create table a(
a_id number(38) not null,
constraint pk_a primary key (id)
);
create table b(
a_id number(38) not null
);
create index b_a_id_index on b(a_id);
Now b.a_id is in fact meant to be a foreign key referencing a.a_id, but it isn't formally declared as such. Obviously, it should be for integrity reasons. But does a foreign key constraint also improve join performance in general or in specific cases? If yes, for what types of query transformations?
Is there any relevant documentation about this topic?
I'm using Oracle 11g (11.2.0.2.0)
Yes, having foreign key constraints in place can improve query performance. There are various transforms that are open to the optimizer when appropriate foreign key constraints exist that are not generally available. For example, if you were to join A and B but only select data from B, the optimizer could eliminate A from the query plan entirely if there was a foreign key constraint in place (this sort of thing comes in very handy when you've got useful views that join in more tables than your current query strictly needs because you don't have to trade the performance costs of the extra joins against the code reuse from using an existing view). They also come in handy when you're doing things like using things like query rewrite to rewrite a query to use a materialized view in a data warehouse/ DSS type system.
Tom Kyte has a presentation Metadata Matters that talks about how various types of constraints, along with other pieces of metadata, can influence the optimizer.
As Justin already pointed out, JOIN elimination is an essential non-cost based SQL transformation, which can be applied based on the presence of meta data only. I have blogged about this more recently:
JOIN Elimination: An Essential Optimiser Feature for Advanced SQL Usage
10 Cool SQL Optimisations That do not Depend on the Cost Model
As I originally assumed, there are a lot of SQL transformations that depend on meta data, so adding foreign key constraints (and other constraints) definitely can impact performance in a positive way.
I have two databases with identical table layouts. There are a dozen or so tables of interest. They are a number of FK between them.
I have been asked to write a stored procedure to copy data from database A to database B based on the PK of the parent table at the top of the hierarchy. I may receive just one value, or a list of values. I'm supposed to select all records from database A that match the value(s) and insert/update them into database B. This includes all the records in the child tables too.
My questions is whats the best(most efficent/ best practice) way to do this?
Should I write a dozen select from... insert into... statements?
Should I join the tables together an try to insert into all the tables at the same time?
Thanks!
Additional info:
The record should be inserted if it is not already there. (based on the PK of the respective table). Otherwise it should be updated.
Obviously I need to traverse down to all child tables, so There would only be one record to copy in the parent table, but the child table might have 10, and the child's child table might have 500. I would of course need to update the record if it already existed, insert if it does not for the child tables too...
UPDATE:
I think it would make the solution simpler if I just deleted all records related to the top level key, and then insert all the new records rather than trying to do updates.
So I guess the questions is it best to just do a dozen:
delete from ... where ... in ...
select from ... where ... in ...
insert into...
or is it better to do some kinda of fancy joins to do all the inserts in one sql statement?
I would do this by disabling all the foreign key constraints, then doing a set of MERGE statements to deal with the updates and inserts, then enable all the constraints.
Think about logging. How much redo do you want to generate?
You might find that it's quicker and better to truncate all the target tables and then do inserts of everything with nolog. Could be simpler than the merges.
One major main alternative would be to drop all the target tables and use export and import. Might be a lot faster.
A second alternative would be to use materialized views, particularly if you don't need to do updates on the target tables. That way, Oracle does all the heavy lifting for you. You can force integrity by choosing refresh groups carefully.
There are several ways to deal with this business problem. A PL/SQL program may not be the best.
So we've been told that one source of TM Enq contention can be unindexed FK's. My question is which one.
I have an INSERT INTO Table_B that is recording TM Enq Wait.
It contains a PK that is the parent to other tables and it has columns that are FK constrained to other PKs.
So which FKs need indexed: that table's columns or its children?
NB: I know that this isn't the only cause of TM Contention. Can you explain why it couldn't possibly be this if that's the case.
Not sure about Oracle TM Contention, but I'd say normally both sides of a foreign key relation are indexed. Otherwise, the database will have to do table scans.
The index on the parent record is used whenever you insert a new child record, to verify that the parent exists. Often this is a primary key as well, so of course has an index.
The index on the child record is used whenever you change or delete a parent record, to perform cascades (including refusing the update/delete).
The indices on both sides also give the database a good chance of doing fast (indexed) joins, no matter which side its optimizer prefers to come from.
EDIT: Having Googled TM contention, it sounds like you're probably missing the keys on the child records. But make sure to have them on both sides, really.
EDIT 2: Answering the comment,
If you have a OLTP table that has 13 FKs to lookup tables, I'm not
keen on 13 index updates in addition to the table, pk and any other
indexes. An index is important but for specific reasons. If you never
update the parent PK nor delete from the parent, the child index is
not so useful. Or is it?
Depends on the joins and queries you're running, then. E.g., if you run
a query like:
SELECT o.something
FROM oltp_tab o JOIN lookup l ON (o.lookup_no = l.lookup_no)
WHERE l.lookup_name = ?
then the query optimizer would probably like the index on the child
records.
Also, according to http://ashmasters.com/waits/enq-tm-contention/ you
pretty much need to have the indices if you change the parent tables at
all. Apparently you get them from having concurrent changes to the
parent and child tables, unless you have the index. So this is probably
what you're seeing (assuming you're not doing the obvious things, like
updating the referred to columns or deleting rows)
The parent (referenced) column of an enabled foreign key relationship has to be indexed because it has to have an enabled unique or primary key constraint on it.
What mode of TM Enqueue are you seeing?