I was reading about bad file in Informatica and it was very clear concept. However, didn't understand following row indicators :-
"Rolled-back insert" = Writer
"Rolled-back update" = Writer
"Rolled-back delete" = Writer
"Committed insert" = Writer
"Committed update" = Writer
"Committed delete" = Writer
I'm trying to understand when above scenario will happen? I mean when and how "Rolled-back insert" and other indicators will happen?
Source
https://dwbi.org/etl/informatica/143-informatica-reject-or-bad-files
Very good and straight forward question.
Basically data base table may have some constraints or indexes associated for one or more columns. For example, if the source data is null and it is a not null column in a target table in this case following operation is occurred,
Rolled-back insert
Rolled-back update
Rolled-back delete
Like wise for primary key column insertion of duplicate values. These or some reasons.
After successful data loading commit process occurs based on the operation you perform in the mapping such as insert, update or delete.
Committed insert
Committed update
Committed delete
Even though commit operation is triggered, data will not be reflected in the target for the above mentioned reasons.
Hope you got what you want. If not kindly raise your queries.
Related
I have a web application using a mariaDB10.4.10 INNO_DB table which is updated every 5 minutes from a script.
The script is working like:
Create a temp table from a table XY and writing data to the temp table from a received csv file. When the data is written, the script starts a transaction, drop the XY table and rename the temp table to the XY, and commits the transaction.
Nevertheless some times a user gets an "XY table does not exists" error working with the application.
I already tried to LOCK the XY table in the transaction but it doesn't change a thing.
How can I solve this? Is there any kind of locking (I thought locking is no longer possible with INNO_DB?)
Do this another way.
Create the temporary table (not as temporary table, as a real table). Fill it as needed. Nobody else knows it's there, you have all the time.
SET autocommit = 0; // OR: LOCK TABLE xy WRITE, tempxy READ;
DELETE FROM xy;
INSERT INTO xy SELECT * FROM tempxy;
COMMIT WORK; // OR: UNLOCK TABLES;
DROP TABLE tempxy;
This way, other customers will see the old table until point 5, then they'll start seeing the new table.
If you use LOCK, customers will stall from point 2 to point 5, which, depending on time needed, might be bad.
At point #3, in some scenarios you might be able to optimize things by deleting only rows that are not in tempxy, and running an INSERT ON DUPLICATE KEY UPDATE at point 4.
Funnily enough, I answered recently another question that was somewhat like yours.
autoincrement
To prevent autoincrement column from overflowing, you can replace COMMIT WORK with ALTER TABLE xy AUTO_INCREMENT=. This is a dirty hack and relies on the fact that this DDL command in MySQL/MariaDB will execute an implicit COMMIT immediately followed by the DDL command itself. If nobody else inserts in that table, it is completely safe. If somebody else inserts in that table at the exact same time your script is running, it should be safe in MySQL 5.7 and derived releases; it might not be in other releases and flavours, e.g. MySQL 8.0 or Percona.
In practice, you fill up tempxy using a new autoincrement from 1 (since tempxy has been just created), then perform the DELETE/INSERT, and update the autoincrement counter to the count of rows you've just inserted.
To be completely sure, you can use a cooperative lock around the DDL command, on the one hand, and anyone else wanting to perform an INSERT, on the other:
script thread other thread
SELECT GET_LOCK('xy-ddl', 30);
SELECT GET_LOCK('xy-ddl', 30);
ALTER TABLE `xy` AUTO_INCREMENT=12345; # thread waits while
# script thread commits
# and runs DDL
SELECT RELEASE_LOCK('xy-ddl'); # thread can acquire lock
INSERT INTO ...
DROP TABLE tempxy; # Gets id = 12346
SELECT RELEASE_LOCK('xy-ddl');
I recently received a case which my client came across the ORA-00001: unique constraint violated error. This happened when a program tried to truncate two tables and then insert data into them.
From the error-log file, the truncate step was completed,
delete from INTERNET_GROUP
delete from INTERNET_ITEM
BUT right after this, the insertion to the Internet_group table triggered the ORA-00001 error. I am wondering if there is any database settings related to this error? I never used Oracle and am wondering if Oracle puts a lock on a row with SELECT statement, in which case the row is locked and not deleted somehow? Any help is appreciated.
Please know that there is a difference between truncate and delete. You say you truncated the table, but you mention "delete from" . That is entirely different.
If you're sure you want to empty the tables, try replacing with
truncate table internet_group reuse storage;
Mind you that a commit is not necessary with the truncate statement as this is considered a DDL (data definition language) statement and not a DML (Data modification language) statement like updates and deletes.
Also, there is no row locking on selects. But changes are only applied and visible for other sessions in the database when commit-ed.
I guess that is wat happened; you deleted the records but did not execute a commit (yet) and subsequently inserted new records.
edit:
I now realize you're probably inserting multiple records....
The other option might be, that the data itself causes a violation. Can you please provide the constraints on the table? There must be a primary key or unique constraint. You might want to hold that against your dataset.
I need to update the some tables in my application from some other warehouse tables which would be updating weekly or biweekly. I should update my tables based on those. And these are having foreign keys in another tables. So I cannot just truncate the table and reinsert the whole data every time. So I have to take the delta and update accordingly based on few primary key columns which doesn't change. Need some inputs on how to implement this approach.
My approach:
Check the last updated time of those tables, views.
If it is most recent then compare each row based on the primary key in my table and warehouse table.
update each column if it is different.
Do nothing if there is no change in columns.
insert if there is a new record.
My Question:
How do I implement this? Writing a PL/SQL code is it a good and efficient way? as the expected number of records are around 800K.
Please provide any sample code or links.
I would go for Pl/Sql and bulk collect forall method. You can use minus in your cursor in order to reduce data size and calculating difference.
You can check this site for more information about bulk collect, forall and engines: http://www.oracle.com/technetwork/issue-archive/2012/12-sep/o52plsql-1709862.html
There are many parts to your question above and I will answer as best I can:
While it is possible to disable referencing foreign keys, truncate the table, repopulate the table with the updated data then reenable the foreign keys, given your requirements described above I don't believe truncating the table each time to be optimal
Yes, in principle PL/SQL is a good way to achieve what you are wanting to
achieve as this is too complex to deal with in native SQL and PL/SQL is an efficient alternative
Conceptually, the approach I would take is something like as follows:
Initial set up:
create a sequence called activity_seq
Add an "activity_id" column of type number to your source tables with a unique constraint
Add a trigger to the source table/s setting activity_id = activity_seq.nextval for each insert / update of a table row
create some kind of master table to hold the "last processed activity id" value
Then bi/weekly:
retrieve the value of "last processed activity id" from the master
table
select all rows in the source table/s having activity_id value > "last processed activity id" value
iterate through the selected source rows and update the target if a match is found based on whatever your match criterion is, or if
no match is found then insert a new row into the target (I assume
there is no delete as you do not mention it)
on completion, update the master table "last processed activity id" to the greatest value of activity_id for the source rows
processed in step 3 above.
(please note that, depending on your environment and the number of rows processed, the above process may need to be split and repeated over a number of transactions)
I hope this proves helpful
I have two datablocks A and B.Block A has a foreign key corresponding to Block B.I want insertion and updation to happen in both the blocks.
I have a situation where when the details of the value changed in block B during updation is not present in the DB,then the values has to be inserted in the db and other changes in Block A has to be updated.
So I am moving from Insertion to updation here
In my case , Insertion is happening but my Block A changes are not getting updated .Instead I am getting."Record Already Inserted" error.
Any ideas would be of much help..
Check out the POST built-in. You can call it when navigating to detail block, to make sure the master record is already in the DB. It will trigger any insert or update validation, but the record won't be commited yet.
About the "Record Already Inserted" error, from form builder help:
FRM-40600: Record has already been inserted.
Cause: You attempted to insert or update a record, but uniqueness is enforced on the block's primary key items. The record, as inserted or updated, is not unique.
Action: Change the values in one or more primary key fields of the current record, making them unique. If the requirement of unique primary key fields creates difficulties, consider eliminating the constraint.
With respect to Oracle RDBMS, which rollback is faster?
Rollback1 : Insert 1000000 records and then rollback
or
Rollback2 : Delete 1000000 records and then rollback
You can find out the % of completion of query using this:
SELECT SESSION_ID, percent_complete, estimated_completion_time
FROM SYS.DM_EXEC_REUQESTS
To answer this question we should know how Oracle handles insert ande delete operations internally. I know that when you insert something, it inserts on memory, then when you commit. Oracle writes to disk.
For delete operation I found this : http://www.dba-oracle.com/t_oracle_soft_logical_deletes.htm So it logically deletes usually and when it is possible it deletes real-time.
Now we should talk about rollback, when you rollback an insert, you clear rows from memory and clear your redo logs. It sounds simple. When you rollback a delete (if Oracle deleted the rows real-time) it should go to redo log, read, then insert the deleted rows back to database.
So if I am right and logical, rollbacking a delete operation should take more time than rollbacking insert.
Also if you are deleting with condition, delete process should take more time than inserting by itself.
P.S. Thanks for the question by the way, it's interesting and made me do some research on Oracle internals.