I have three oracle environments. If I make changes to the data in my DEV environment, I want to be able to take those changes and move them to my other environments.
The challenge is that the data in the other environments will not have the same primary key. So the compare would have to look at the other columns in the table. If the table referenced another table, it would have to include columns from that table as well.
So, for example, table A might have 4 columns that were all IDs to other tables, and so manipulating table A would require referring to those other tables. Can anyone suggest a product for this?
Updated Requirements: Nothing about the systems as they currently stand can be changed. IDs will never be in synch. The synchronization must only happen when desired (after our bi-monthly updates). Synchronization can not be done over DB links (though the DEV system can read data across the DB links). I can definitely write the SQL to do all this, its just the kind of thing that is prone to error (and typos) and if there is a 3rd party application out there that can take care of this for me, I'd rather spend the money on that.
Updates as per request: All systems are 11g Enterprise. The amount of data is very small, bandwidth isn't an issue. The synchronization happens every couple of months. Basically, we have rules that tell our UI how to behave. Those rules are stored in various tables. From time to time, we change those rules. This can be thousands of records, but not tens of thousands. We don't want the kind of synchronization that one might want from 'live' data. What I need to be able to do is specify the two database instances, and the tables in question, and then have a SQL script generated that, effectively, moves the changes from database A to database B. Given the nature of the data, I've been considering simply truncating the target tables and sqlloading the data into the other environments. That isn't quite as simple as it sounds as there is one 'live' table that refers to these records and that would require some specific updates potentially (I'm actively researching this possibility as well, it just isn't as simple as it sounds).
Here's code that will provide examples of what I mean. You'll notice IDs, and those are NOT consistent across environments, which is what makes this tricky.
drop table mydata;
drop table MYTOWN;
drop table MYJOB;
drop table MYEMPLOYER;
drop table mystate;
create table MYJOB
(MYJOBID varchar2(1000) NOT NULL PRIMARY KEY,
MYJOB varchar2(1000));
create table MYEMPLOYER
(MYEMPLOYERID varchar2(1000) NOT NULL PRIMARY KEY,
MYEMPLOYER varchar2(1000));
create table MYSTATE
(MYSTATEID varchar2(1000) NOT NULL PRIMARY KEY,
MYSTATE varchar2(1000));
create table MYTOWN
(MYTOWNID varchar2(1000) NOT NULL PRIMARY KEY,
MYTOWN varchar2(1000),
MYSTATEID varchar2(1000),
CONSTRAINT MYSTATE_FK FOREIGN KEY (MYSTATEID) REFERENCES MYSTATE (MYSTATEID) ENABLE);
create table MYDATA
(MYDATAID varchar2(1000) NOT NULL PRIMARY KEY,
MYTOWNID varchar2(1000),
MYJOBID varchar2(1000),
MYEMPLOYERID varchar2(1000),
CONSTRAINT MYTOWN_FK FOREIGN KEY (MYTOWNID) REFERENCES MYTOWN (MYTOWNID) ENABLE,
CONSTRAINT MYJOB_FK FOREIGN KEY (MYJOBID) REFERENCES MYJOB (MYJOBID) ENABLE,
CONSTRAINT MYEMPLOYER_FK FOREIGN KEY (MYEMPLOYERID) REFERENCES MYEMPLOYER (MYEMPLOYERID) ENABLE
);
create sequence mydataid_seq;
insert into myemployer values ('937436', 'Bank Of America');
insert into myemployer values ('43', 'Google');
insert into myemployer values ('2', 'Toms Taxi');
insert into myjob values ('8','Programmer');
insert into myjob values ('10','Cook');
insert into myjob values ('5','Driver');
insert into mystate values ('7643','MA');
insert into mystate values ('23','CA');
insert into mystate values ('54','NM');
insert into mytown values ('4743','BOSTON','7643');
insert into mytown values ('321','SANDIEGO','23');
insert into mytown values ('92037','SANTA FE','54');
insert into mydata values ('78','4743','8','937436');
insert into mydata values ('23455','321','10','43');
insert into mydata values ('901','92037','5','2');
--to select a unique row
select mt.mytown, ms.mystate, mj.myjob, me.myemployer
from mydata md, mytown mt, mystate ms, myemployer me, myjob mj
where md.mytownid=mt.mytownid
and mt.mystateid=ms.mystateid
and md.myjobid=mj.myjobid
and md.myemployerid=me.myemployerid;
--to delete a row
delete from mydata md where md.mydataid =
(select md.mydataid
from mydata md, mytown mt, mystate ms, myemployer me, myjob mj
where md.mytownid=mt.mytownid
and mt.mystateid=ms.mystateid
and md.myjobid=mj.myjobid
and md.myemployerid=me.myemployerid
and mt.mytown='SANDIEGO'
and ms.mystate='CA'
and mj.myjob='Cook'
and me.myemployer='Google');
--to insert a row
insert into mydata (mydataid,mytownid, myjobid, myemployerid)
(select mydataid_seq.nextval, mt.mytownid, mj.myjobid, me.myemployerid
from mytown mt, mystate ms, myemployer me, myjob mj
where mt.mytown='SANTA FE'
and mj.myjob='Programmer'
and me.myemployer='Toms Taxi'
and ms.mystate='NM'
);
Look at Oracle GoldenGate.
http://www.oracle.com/technetwork/middleware/goldengate/overview/index.html
May be it would be like a heavy gun solution.
The best solution depends on your concrete needs (how much data? how often? how much transformation must done? bandwidth between servers? their load? and so on...).
Oracle has many replication options.
Update:
Advise about GoldenGate is still actual, but it isn't cheap solution.
But, if you have some coding skills and interest, then it maybe easier to:
use db-link and write PLSQL packages with a syncing logic. Then, you can call it when you need, or run it regularly as Job/Task. Central database must see others and must have an ability to establish a direct connection.
or you can write an external application and place a syncing logic there. Databases could be in separate DMZs.
Of course, you can use Oracle Streams (11g enterprise includes it), but i can't guarantee that it would work without additional codding in your case. So, this approach can turn into first one with complications.
Sometimes, using serious replication solutions take more time for setting up them and maintain. In your case, you need just a syncing.
And you can find some kind of ready and free solutions. But, in most cases they suffer from poor quality. So, I don't recommend them.
PS:
You can solve inconsistence of data, by using two approaches:
Easiest: use oracle sequences with different start values for key columns for each database (don't forget to set up max value too, to prevent overlapping)
Normal solution: redesign your data model to include information about office/server
Related
I'm trying to find a way to perform a migration from two tables on the same database. This migration should be as fast as possible in order to minimize the downtime.
To put it on an example lets say I have a person table like so:
person_table -> (id, name, address)
So a person as an Id, a name and an address. My system will contain millions of person registries and it was decided that the person table should be partitioned. To do so, I've created a new table:
partitioned_person_table->(id,name,address,partition_time)
Now this table will contain an extra column called partition_time. This is the partition key for this table since this is a range partition (one partition every hour).
Finally, I need to find a way to move all the information from the person_table to the partitioned_person_table with the best performance.
The first thing I could try is to simply create a statement like:
INSERT INTO partitioned_person_table (id, name, address, partition_time)
SELECT id, name, address, CURRENT_TIMESTAMP FROM person_table;
The problem is that when it comes to millions of registries this might become very slow (also the temporary tablespace might not be able to handle all this information)
My second approach was to use the EXCHANGE PARTITION method. Unfortunetly, I cannot do this because the tables contain diffrent column numbers.
Is there any other way that I can perfom this with the best performance (less downtime) ?
Thank you.
If you can live with the state, that all the current records would be located in one partition (and your INSERT approach suggest that), you may only
1) add a new column partition_time either as NULL or possible with metadata default only - required 12c
2) switch the table to a partitioned table either with online redefinition (if you have no maintainace window, where the table is offline) or with exchange partition otherwise.
I have two tables named as patient and pharmacy. Each patient is uniquely associated with one pharmacy. I want to create the foreign key constraint at run-time between these tables.
Create table patient
(patient_Id varchar2(5) primary key,
patient_name varchar2(20));
Create table pharmacy
(pharmacy_Id varchar2(5) primary key,
pharmacy_name varchar2(20);
Create table patient_pharmacy_mapper
(patient_Id varchar2(5) references patient(patient_Id),
pharmacy_Id varchar2(5) references pharmacy(pharmacy_Id));
Instead of writing the "references" at design time, Can I create/delete these foreign key constraints at run-time (when any DML statement fires)?
I know little about creating a trigger where we have to call a procedure with the "Alter table statement".
DDL statements automatically commit the transaction. As you are not allowed to commit (or rollback) in a trigger, you can not run DDL statements in a trigger (neither with static SQL nor with dynamic SQL)
The whole idea does not make sense. The only sane way to do this, is to create the FK constraints when creating the tables. You gain no security from delaying this, absolutely none.
Let me just add another vote to the others about this being a Very Bad Idea (tm). FK relationships enforce fundamental business rules. They are part of the design, to be implemented when at the same time tables are created. Any time (and I emphasize ANY time) you find yourself wanting to execute DDL at run time, you need to step back, get a cup of coffee, and reconsider.
Like the other statements I also say it is a very bad idea.
Consider, you can enable/disable constraints or you can set them deferred: SET CONSTRAINT[S]
Perhaps this is a solution for you problem.
Lets say i have a query which is fetching col1 after joining multiple tables. I want to insert values of that col1 in a table which is on remote db i.e. i would be using dblink to do that.
Now that col1 would be fetched from 4-5 different db's. There is chances that a value1 fetch from db1 would b in db2 as well. How can i avoid duplicates ?
In my remote db, I have created col1 a primary key. so when inserting, an error would be thrown if there is a duplicate key, end result failing rest of the process. Which i don't want to. I was thiking about 2 approaches
Write a PLSQL script, For each value, determine if value already exists or not. If it doesn't then insert.
Write a PLSQL script and insert and catch the duplicate key exception. The exception would be ignore and it will keep inserting (it doesn't sound that good).
Which approach would you prefer? Is there anything else i can do ?
I would use the MERGE statement and WHEN NOT MATCHED THEN INSERT.
The same merger could also update but it doesn't have to, just leave the update part out.
The different databases can have duplicate primary keys but that doesn't mean the records are duplicates. The actual data may be different in each case. Or the records may represent the same real world thing but at different statuses, Don't know, you haven't provided enough explanation.
The point is, you need much more analysis of why duplicate records can exist and probably a more sophisticated approach to handling collisions. Do you need to take all records (in which case you need a synthetic key)? Or do you take only one instance (so how do you decide precedence)? Other scenarios may exist.
In any case, MERGE or PL/SQL loops are likely to be too crude a solution.
First off, I would suggest that your target database drive all of these inserts because inserting/updating across a database link can create some locking issues and further complicate things especially with multiple databases attempting to access and perform DML on the same table. However if that isn't possible the solutions below will work.
I would fix your primary key problem by including a table look-up on the target table for each row.
INSERT INTO customer#dblink.oracle.com cust
(emp_name,
emp_id)
VALUES
(SELECT
cust.employee_name,
cust.employee_id --primary_key
FROM
source_table st
WHERE NOT EXISTS
(SELECT 1
FROM customer#dblink.oracle.com cust
WHERE cust.employee_id = st.emp_id));
Again, I would not recommend DML transactions across database links unless absolutely necessary as you can sometimes have weird locking behavior.
A PL/SQL procedure or anonymous PL/SQL block could be used to create a bulk processing solution as follows:
CREATE OR REPLACE PROCEDURE send_unique_data
AS
TYPE tab_cust IS TABLE OF customer#dblink.oracle.com%ROWTYPE
INDEX BY PLS_INTEGER;
t_records tab_cust;
BEGIN
SELECT
cust.employee_name,
cust.employee_id --primary_key
BULK COLLECT
INTO t_records
FROM source_table;
FORALL i IN t_records.FIRST...t_records.LAST SAVE EXCEPTIONS
INSERT INTO customer#dblink.oracle.com
VALUES t_records(i);
END send_unique_data;
You can also call the system SQL%BULKEXCEPTIONS collection in case you want to do anything with the records that threw exceptions (such as unique_constraint violations). Be warned that this solution will cause the target table to suffers from performance issues if there are lots of duplicate data attempting to be inserted.
what is use-case of IOT (Index Organized Table) ?
Let say I have table like
id
Name
surname
i know the IOT but bit confuse about the use case of IOT
Your three columns don't make a good use case.
IOT are most useful when you often access many consecutive rows from a table. Then you define a primary key such that the required order is represented.
A good example could be time series data such as historical stock prices. In order to draw a chart of the stock price of a share, many rows are read with consecutive dates.
So the primary key would be stock ticker (or security ID) and the date. The additional columns could be the last price and the volume.
A regular table - even with an index on ticker and date - would be much slower because the actual rows would be distributed over the whole disk. This is because you cannot influence the order of the rows and because data is inserted day by day (and not ticker by ticker).
In an index-organized table, the data for the same ticker ends up on a few disk pages, and the required disk pages can be easily found.
Setup of the table:
CREATE TABLE MARKET_DATA
(
TICKER VARCHAR2(20 BYTE) NOT NULL ENABLE,
P_DATE DATE NOT NULL ENABLE,
LAST_PRICE NUMBER,
VOLUME NUMBER,
CONSTRAINT MARKET_DATA_PK PRIMARY KEY (TICKER, P_DATE) ENABLE
)
ORGANIZATION INDEX;
Typical query:
SELECT TICKER, P_DATE, LAST_PRICE, VOLUME
FROM MARKET_DATA
WHERE TICKER = 'MSFT'
AND P_DATE BETWEEN SYSDATE - 1825 AND SYSDATE
ORDER BY P_DATE;
Think of index organized tables as indexes. We all know the point of an index: to improve access speeds to particular rows of data. This is a performance optimisation of trick of building compound indexes on sub-sets of columns which can be used to satisfy commonly-run queries. If an index can completely satisy the columns in a query's projection the optimizer knows it doesn't have to read from the table at all.
IOTs are just this approach taken to its logical confusion: buidl the index and throw away the underlying table.
There are two criteria for deciding whether to implement a table as an IOT:
It should consists of a primary key (one or more columns) and at most one other column. (okay, perhaps two other columns at a stretch, but it's an warning flag).
The only access route for the table is the primary key (or its leading columns).
That second point is the one which catches most people out, and is the main reason why the use cases for IOT are pretty rare. Oracle don't recommend building other indexes on an IOT, so that means any access which doesn't drive from the primary key will be a Full Table Scan. That might not matter if the table is small and we don't need to access it through some other path very often, but it's a killer for most application tables.
It is also likely that a candidate table will have a relatively small number of rows, and is likely to be fairly static. But this is not a hard'n'fast rule; certainly a huge, volatile table which matched the two criteria listed above could still be considered for implementations as an IOT.
So what makes a good candidate dor index organization? Reference data. Most code lookup tables are like something this:
code number not null primary key
description not null varchar2(30)
Almost always we're only interested in getting the description for a given code. So building it as an IOT will save space and reduce the access time to get the description.
I have come into a bump at my current company where they have an account and a member. For some reason or another both are stored in separate tables.
Right now a member and an account can be registered. That's fine, except the users of both member and an account can have the same username. This is of course as you all know just wrong. Especially since they use the username to login to the same system except with different functionality levels.
Right now we are doing a check at the application level, and we're just wondering if it's possible to get the database to enforce two columns to be unique, say like a union of the two tables.
Can't set them up as primary or foreign key at the moment but that's for future anyway. Right now looking for a quick fix. In the future I will probably merge databases and get all members added on as new rows in the account table and add a boolean for IsMember.
In general, I agree with the consensus opinion that it's better to fix the design than to kluge a fix using triggers. However, a properly implemented trigger-based solution is still probably better than your current situation.
If you're going to use triggers, the right way to do it is to:
Create a new table that will contain nothing but usernames, with a primary key enforcing uniqueness (this may, in fact, be a good candidate for an indes-organized table).
Create before-insert triggers on both existing tables that add the new username to the new table. If the new username already exists, an error will be thrown, preventing the insert of both rows. Of course, the application will need to be able to handle this error gracefully (presumably it already can, for scenarios in which the new username already exists in the table it's being added to).
The wrong way to do this would be to make the trigger select from the other table, in order to verify uniqueness.
You can add a trigger that enforces your requirement.
The recommended triggers tend to be really brittle with concurrent transaction.
What you can do (AFAIK) is to create a materialized view containing the union of the column in question and put a unique constraint on that column.
Make sure you do some performance tests though.
As you use a soft delete pattern.
A trigger could be used (on each table) as a temporary measure.
By inserting a disabled record in the the other table, you will get a failure if the other record already exists
Remember this will not enforce the rule on existing data, only records that are inserted will be checked
Something like this:
-- Insert into the accounts table too
CREATE OR REPLACE TRIGGER tr_member_chk
BEFORE INSERT ON members
FOR EACH ROW
BEGIN
INSERT INTO account (name, id, etc, isenabled) VALUES(:new.name, :new.id, :new.etc, 0);
END;
-- Insert into the members table too
CREATE OR REPLACE TRIGGER tr_account_chk
BEFORE INSERT ON accounts
FOR EACH ROW
BEGIN
INSERT INTO members (name, id, etc,isenabled) VALUES(:new.name, :new.id, :new.etc,0);
END;