I have a table design doubt.
First table called products_mast contains
CONTRACTS_ID (PK),
AGREEMENT_NO,
SUPPLIER_CODE,
START_DATE,
END_DATE
PROD_EXTENSION_NO
Second table called products_det contains
HIRING_ID (PK),
CONTRACTS_ID(FK),
PRODUCT_CODE,
RATE
The above tables are linked with CONTRACTS_ID. Start Dates and End Dates are in one table and product rates in second table.
In my third table TRANSACTIONS contains,
TRANS_ID,
TRANS_REF_NO,
HIRING_ID FK (products_det),
REMARKS
One issue is normally the agreement_no can get extended for a period with new product rates for the extended agreement. So I add those details in
products_mast and products_det with same agreement no with extension 002 (for new extension). In TRANSACTIONS how can I refer to correct
rates if I am linking HIRING_ID with products_det because HIRING_ID gets incremented with new extension. How can I resolve this issue?
I'm not quite sure I follow exactly what you are suggesting regards agreement no value of extension 002 (new extension) however if it's a case of uniquely identifying the transaction per agreement no, what about adding a FK to the table Transactions linking it back to the AGREEMENT_NO field in the products_mast table?
Related
I'm trying to find a way to perform a migration from two tables on the same database. This migration should be as fast as possible in order to minimize the downtime.
To put it on an example lets say I have a person table like so:
person_table -> (id, name, address)
So a person as an Id, a name and an address. My system will contain millions of person registries and it was decided that the person table should be partitioned. To do so, I've created a new table:
partitioned_person_table->(id,name,address,partition_time)
Now this table will contain an extra column called partition_time. This is the partition key for this table since this is a range partition (one partition every hour).
Finally, I need to find a way to move all the information from the person_table to the partitioned_person_table with the best performance.
The first thing I could try is to simply create a statement like:
INSERT INTO partitioned_person_table (id, name, address, partition_time)
SELECT id, name, address, CURRENT_TIMESTAMP FROM person_table;
The problem is that when it comes to millions of registries this might become very slow (also the temporary tablespace might not be able to handle all this information)
My second approach was to use the EXCHANGE PARTITION method. Unfortunetly, I cannot do this because the tables contain diffrent column numbers.
Is there any other way that I can perfom this with the best performance (less downtime) ?
Thank you.
If you can live with the state, that all the current records would be located in one partition (and your INSERT approach suggest that), you may only
1) add a new column partition_time either as NULL or possible with metadata default only - required 12c
2) switch the table to a partitioned table either with online redefinition (if you have no maintainace window, where the table is offline) or with exchange partition otherwise.
I need to update the some tables in my application from some other warehouse tables which would be updating weekly or biweekly. I should update my tables based on those. And these are having foreign keys in another tables. So I cannot just truncate the table and reinsert the whole data every time. So I have to take the delta and update accordingly based on few primary key columns which doesn't change. Need some inputs on how to implement this approach.
My approach:
Check the last updated time of those tables, views.
If it is most recent then compare each row based on the primary key in my table and warehouse table.
update each column if it is different.
Do nothing if there is no change in columns.
insert if there is a new record.
My Question:
How do I implement this? Writing a PL/SQL code is it a good and efficient way? as the expected number of records are around 800K.
Please provide any sample code or links.
I would go for Pl/Sql and bulk collect forall method. You can use minus in your cursor in order to reduce data size and calculating difference.
You can check this site for more information about bulk collect, forall and engines: http://www.oracle.com/technetwork/issue-archive/2012/12-sep/o52plsql-1709862.html
There are many parts to your question above and I will answer as best I can:
While it is possible to disable referencing foreign keys, truncate the table, repopulate the table with the updated data then reenable the foreign keys, given your requirements described above I don't believe truncating the table each time to be optimal
Yes, in principle PL/SQL is a good way to achieve what you are wanting to
achieve as this is too complex to deal with in native SQL and PL/SQL is an efficient alternative
Conceptually, the approach I would take is something like as follows:
Initial set up:
create a sequence called activity_seq
Add an "activity_id" column of type number to your source tables with a unique constraint
Add a trigger to the source table/s setting activity_id = activity_seq.nextval for each insert / update of a table row
create some kind of master table to hold the "last processed activity id" value
Then bi/weekly:
retrieve the value of "last processed activity id" from the master
table
select all rows in the source table/s having activity_id value > "last processed activity id" value
iterate through the selected source rows and update the target if a match is found based on whatever your match criterion is, or if
no match is found then insert a new row into the target (I assume
there is no delete as you do not mention it)
on completion, update the master table "last processed activity id" to the greatest value of activity_id for the source rows
processed in step 3 above.
(please note that, depending on your environment and the number of rows processed, the above process may need to be split and repeated over a number of transactions)
I hope this proves helpful
I have a question about how to change relationships between tales in an Oracle database while preserving existing data.
Let's say I want to represent People and Employers such that each person works for a single employer. I do this with a PERSON table and an EMPLOYER table with a 1:M relationship of EMPLOYER to PERSON. The PERSON table had columns ID, NAME, and EMPLOYER_ID, and the EMPLOYER table had columns ID, NAME, AND LOCATION.
If I wanted to update this schema so a PERSON can work for more than one EMPLOYER, I could add a PERSON_EMPLOYER table with columns for each ID.
Could anyone give some pointers on the most sensible way to do this and move my existing data in? I think I can add a join table, but I'm not sure how to populate it with existing employer:employer data. After that I guess I remove EMPLOYER_ID column from PERSON.
Should I just be backing up the database and doing this operation in a script?
Thank you so much.
Having a backup is always a good idea.
But in my opinion transferring data from one table to another it is quite reliable operation. You do not necessary make a script, just do it step by step and check changes.
Create a new PERSON_EMPLOYER table
Copy existing data to PERSON_EMPLOYER table.
COMMIT data changes.
Check data in PERSON_EMPLOYER table
Drop EMPLOYER_ID column from PERSON table. (No need to remove the column immediately, it can be done later when you will be sure that everything is fine with the your data.)
For transferring data from PERSON table to PERSON_EMPLOYER table you can use simple INSERT
INSERT INTO employer_person (employer_id, person_id)
SELECT employer_id, person_id FROM person;
Do not forget COMMIT this operation!
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 9 years ago.
Improve this question
I have a situation i can't find an explanation for, here it is. (I'll use hypothetical info since the original are really big.)
I have a table, let's say:
table_a
-------------
name
last name
dept
status
notes
And this table has a trigger on insert, which does a lot of validation to the info to change the status field of the new record according to the results of the validation, some of the validations are:
- check for the name existing in a dictionary
- check for the last name existing in a dictionary
- check that fields (name,last name,dept) aren't already inserted in table_b
- ... and so on
The thing is, if I do an insert on the table via query, like
insert into table_a
(name,last_name,dept,status,notes)
values
('john','smith',1,0,'new');
it takes only 173 ms to do all the validation process, update the status field and insert the record in the table. (the validation process does all the searches via indexes)
But if I try this via SQLloader, reading a file with 5000 records, it takes like 40 minutes to validate and insert 149 records (of course i killed it...)
So I tried loading the data disabling the trigger (to check speed)
and I got that it loads like all the records in less than 10 seconds.
So my question is, what can I do to improve this process? My only theory is that I could be saturating the database because it loads so fast and launches many instances of the trigger, but I really don't know.
My objective is to load around 60 files with info and validate them through the process in the trigger (willing to try other options though).
I would really appreciatte any help you can provide!!
COMPLEMENT---------------------------------------------------------------------------------
Thanks for the answer, now i'll read all about this, now hope you can help me with this part. let me explain some of the functionality i need (and i used a trigger cause i couldn't think of anything else)
so the table data comes with this (important) fields:
pid name lastname birthdate dept cicle notes
the data comes like this
name lastname birthdate dept
now, the trigger does this to the data:
Calls a function to calculate the pid (is calculated based on the name, lastname and birthdate with an algorithm)
Calls a function to check for the names on the dictionary (thats because in my dictionary i have single names, meaning if a person is named john aaron smith jones the function splits john aaron in two, and searches for john and aaron in the dictionary in separate querys, thats why i didn't use a foreign key [to avoid having lots of combinations john aaron,john alan,john pierce..etc]) --->kinda stuck on how to implement this one with keys without changing the dictionary...maybe with a CHECK?, the lastname foreign key would be good idea.
Gets the cicle from another table according to the dept and the current date (because a person can appear twice in the table in the same dept but in different cicle) --->how could i get this cicle value in a more efficient way to do the correct search?
And finally, after all this validation is done, i need to know exactly which validation wasn't met (thus the field notes) so the trigger concatenates all the strings of failed validations, like this:
lastname not in dictionary, cannot calculate pid (invalid date), name not in dictionary
i know that if the constraint check isn't met all i could do is insert the record in another table with the constraint-failed error message, but that only leaves me with one validation, am i right? but i need to validate all of them and send the report to other department so they can review the data and make all the necessary adjustments to it.
Anyway, this is my situation right now, i'll explore possibilities and hope you can share some light on the overall process, Thank you very much for your time.
You're halfway to the solution already:
"So I tried loading the data disabling the trigger (to check speed) ... it loads like all the records in less than 10 seconds."
This is not a surprise. Your current implementation executes a lot of single row SELECT statements for each row you insert into table B. That will inevitably give you a poor performance profile. SQL is a set-based language and performs better with multi-row operations.
So, what you need to do is find a way to replace all the SELECT statements which more efficient alternatives. Then you'll be able to drop the triggers permanently. For instance, replace the look-ups on the dictionary with foreign keys between the table A columns and the reference table. Relational integrity constraints, being internal Oracle code, perform much better than any code we can write (and work in multi-user environments too).
The rule about not inserting into table A if a combination of columns already exists in table B is more problematic. Not because it's hard to do but because it sounds like poor relational design. If you don't want to load records in table A when they already exits in table B why aren't you loading into table B directly? Or perhaps you have a sub-set of columns which should be extracted from table A and table B and formed into table C (which would have foreign key relationships with A and B)?
Anyway, leaving that to one side, you can do this with set-based SQL by replacing SQL*Loader with an external table. An external table allows us to present a CSV file to the database as if it were a regular table. This means we can use it in normal SQL statements. Find out more.
So, with foreign key constraints on dictionary and an external table you can replace teh SQL Loader code with this statement (subject to whatever other rules are subsumed into "...and so on"):
insert into table_a
select ext.*
from external_table ext
left outer join table_b b
on (ext.name = b.name and ext.last_name = b.last_name and ext.dept=b.dept)
where b.name is null
log errors into err_table_a ('load_fail') ;
This employs the DML error logging syntax to capture constraint errors for all rows in a set-based fashion. Find out more. It won't raise exceptions for rows which already exist in table B. You could either use the multi-table INSERT ALL to route rows into an overflow table or use a MINUS set operation after the event to find rows in the external table which aren't in table A. Depends on your end goal and how you need to report things.
Perhaps a more complex answer than you were expecting. Oracle SQL is a very extensive SQL implementation, with a lot of functionality for improving the efficient of bulk operations. It really pays us to read the Concepts Guide and the SQL Reference to find out just how much we can do with Oracle.
I have little problem with programming trigger for my dtb. I need to control 2 values in one 1 table. I have table called Concert and it has 2 foreign keys: 1 is the id of table Place. Second is not important for this I think.
Concert: id_concert, id_place<fk>, id_organizer<fk>, date, name, sponsor
Place: id_place, name, capacity, adress, town
What I want to eliminate is, that 2 concerts organized at same day cannot be on one place. So, I need to somehow control that user cannot insert the same date and same place for concert if there already concert with this values exists.
Thank you very much for your suggestions and sorry for bad english.
You need to add a unique constraint on your Concert table that consists of the (id_place, date) pair. This would instruct the database engine to not allow more than one Concert in the same place at the same time.
For Oracle, information can be found here: http://www.techonthenet.com/oracle/unique.php
CREATE TABLE Concert
(
... (filled in with your existing table definition)
CONSTRAINT concert_place_unique UNIQUE (id_place, date)
);
or to alter an existing table:
ALTER TABLE Concert
add CONSTRAINT concert_place_unique UNIQUE (id_place, date);
Constraints are the proper way to handle this condition, not triggers. Constraints are database intrinsic and have no race conditions and prevent the data from being added in the first place.