I have to develop an informatica process that loads data from a flatfile into the target (simple truncate & load), but the catch is that :
If the number of rejected rows is greater than 100, the process should stop, i.e. the session should fail & the data in the target must be rolled back to what it was originally before load.
I think the TC Transformation might be useful here , but am not sure of how to use this. It would be great if I could get some help on this.
Thanks !
You can't use truncate in such scenario - it's irreversible. Try loading the data into a temporary table first (with Truncate table option enabled). Create a second session that will execute a set of sql commands like
`truncate table YourTable
insert into YourTable select * from YourTempTable`
Link the two with a condition like $yourTempTableSession.TgtFailedRows>100.
To meet the second requirement (i.e. to fail the workflow) add a Control task and set it to Abort top level workflow. Add a link from the temp table session load with a condition like $yourTempTableSession.TgtFailedRows>100.
Related
A VB 6 program is processing records and inserting in a temporary table, then these records are moved from this temporary table to actual table as
connection.Execute "INSERT INTO MAIN_TABLE SELECT * FROM TEMP_TABLE"
The temporary table is then truncated when records are moved
connection.Execute "TRUNCATE TABLE TEMP_TABLE"
This is working fine untill I use PARALLEL hint for INSERT query. I receive this error on TRUNCATE
ORA-00054: resource busy and acquire with NOWAIT specified or timeout
expired
It looks to me that parallel query returns before completing the job and TRUNCATE command is issued causing the lock.
I checked the number of records inserted as below and found that it is far less than the number of records in temporary table
connection.Execute "INSERT /*+ PARALLEL */ INTO MAIN_TABLE SELECT * FROM TEMP_TABLE", recordsAffected
Is there any way to wait for INSERT to complete?
Delete may be slower but Truncate is DDL which you can't run at the same time as DML. In fact, Truncate requires exclusive access to the table. DML on tables will request a share mode lock on the table which means you can't do DDL against the table at the same time.
A possible alternate solution would be to use synonyms. You have your table A
and a synonym S pointing to A
create table B as select * from A where 1=0;
create or replace synonym S for B
Your app now uses B instead of A so you can do what you want with A.
Do this every time you want to "truncate"
This assumes you're using ADO - though I now notice you don't have
that tag in your question.
Can you monitor the connection state with a loop waiting for executing to finish?
Something like
EDIT - Fix Boolean Add to use + instead of "AND"
While Conn.State = (adStateOpen + adStateExecuting)
DoEvents
Sleep 500 ' uses Sleep API command to delay 1/2 second
Wend
Sleep API declare
Edit - Add Asynch Hint/Option
Also - it might help the ADO connection to give it a hint that its running asynchronously, by adding the adAsyncExecute to end of your execute command
ie. Change the execute sql command to look like
conn.execute sqlString, recordsaffected, adAsyncExecute
We have simple case, We have a table with column emailId i.e. unique.....using oracle DB
Question#1
Multiple concurrent user can check if some email id is available or not. Like 2 user that same time check availability of: abc#test.com
session1: select emailid from user_table;
//If not present allow user to complete rest of the process & insert info
session2: select emailid from user_table;
Now both session will get that this email id (abc#test.com) is available & both try to insert, I know one of them will get error upon insertion BUT how we can make sure only 1 user get availability & other get not available upon select ??
Question#2
Also in case both sessions inserted the same value, then first will succeed, is there ways that 2nd session update that row instead of throwing error. Like we have another column for timestamp & want that 2nd session instead of throwing error simple update the timestamp column ?
As this is a rather abstract question, here are only some general guidelines:
To deal with concurrent insert in a table, you need an unique index, and be prepared in your code to deal with ORA-00001 error unique constraint violated. Never rely only on check before insert(unless you have somehow exclusive access to your table -- and even if so ... as of myself, I would add an unique index: doesn't cost much and make me sleep better)
Oracle has a MERGE statement that allows you update or insert based on a condition. This operation is sometimes called an upsert. By using that keywork you should be able to find more informationsSee Oracle: how to UPSERT (update or insert into a table?) for example.
Now for, some thoughts about you specific case (maybe):
The only way for the system to work as you suggested, would be to make some kind of reservation when you check for availability (i.e.: immediately insert the row, instead of just select). And then update the row when the user confirm. But that means: (1) you will have to somehow deal with never-confirmed reservations (2) that doesn't dispense you to have an unique index, and to deal with ORA-00001.
Can I find out when the last INSERT, UPDATE or DELETE statement was performed on a table in an Oracle database and if so, how?
A little background: The Oracle version is 10g. I have a batch application that runs regularly, reads data from a single Oracle table and writes it into a file. I would like to skip this if the data hasn't changed since the last time the job ran.
The application is written in C++ and communicates with Oracle via OCI. It logs into Oracle with a "normal" user, so I can't use any special admin stuff.
Edit: Okay, "Special Admin Stuff" wasn't exactly a good description. What I mean is: I can't do anything besides SELECTing from tables and calling stored procedures. Changing anything about the database itself (like adding triggers), is sadly not an option if want to get it done before 2010.
I'm really late to this party but here's how I did it:
SELECT SCN_TO_TIMESTAMP(MAX(ora_rowscn)) from myTable;
It's close enough for my purposes.
Since you are on 10g, you could potentially use the ORA_ROWSCN pseudocolumn. That gives you an upper bound of the last SCN (system change number) that caused a change in the row. Since this is an increasing sequence, you could store off the maximum ORA_ROWSCN that you've seen and then look only for data with an SCN greater than that.
By default, ORA_ROWSCN is actually maintained at the block level, so a change to any row in a block will change the ORA_ROWSCN for all rows in the block. This is probably quite sufficient if the intention is to minimize the number of rows you process multiple times with no changes if we're talking about "normal" data access patterns. You can rebuild the table with ROWDEPENDENCIES which will cause the ORA_ROWSCN to be tracked at the row level, which gives you more granular information but requires a one-time effort to rebuild the table.
Another option would be to configure something like Change Data Capture (CDC) and to make your OCI application a subscriber to changes to the table, but that also requires a one-time effort to configure CDC.
Ask your DBA about auditing. He can start an audit with a simple command like :
AUDIT INSERT ON user.table
Then you can query the table USER_AUDIT_OBJECT to determine if there has been an insert on your table since the last export.
google for Oracle auditing for more info...
SELECT * FROM all_tab_modifications;
Could you run a checksum of some sort on the result and store that locally? Then when your application queries the database, you can compare its checksum and determine if you should import it?
It looks like you may be able to use the ORA_HASH function to accomplish this.
Update: Another good resource: 10g’s ORA_HASH function to determine if two Oracle tables’ data are equal
Oracle can watch tables for changes and when a change occurs can execute a callback function in PL/SQL or OCI. The callback gets an object that's a collection of tables which changed, and that has a collection of rowid which changed, and the type of action, Ins, upd, del.
So you don't even go to the table, you sit and wait to be called. You'll only go if there are changes to write.
It's called Database Change Notification. It's much simpler than CDC as Justin mentioned, but both require some fancy admin stuff. The good part is that neither of these require changes to the APPLICATION.
The caveat is that CDC is fine for high volume tables, DCN is not.
If the auditing is enabled on the server, just simply use
SELECT *
FROM ALL_TAB_MODIFICATIONS
WHERE TABLE_NAME IN ()
You would need to add a trigger on insert, update, delete that sets a value in another table to sysdate.
When you run application, it would read the value and save it somewhere so that the next time it is run it has a reference to compare.
Would you consider that "Special Admin Stuff"?
It would be better to describe what you're actually doing so you get clearer answers.
How long does the batch process take to write the file? It may be easiest to let it go ahead and then compare the file against a copy of the file from the previous run to see if they are identical.
If any one is still looking for an answer they can use Oracle Database Change Notification feature coming with Oracle 10g. It requires CHANGE NOTIFICATION system privilege. You can register listeners when to trigger a notification back to the application.
Please use the below statement
select * from all_objects ao where ao.OBJECT_TYPE = 'TABLE' and ao.OWNER = 'YOUR_SCHEMA_NAME'
I have a number of tables that use the trigger/sequence column to simulate auto_increment on their primary keys which has worked great for some time.
In order to speed the time necessary to perform regression testing against software that uses the db, I create control files using some sample data, and added running of these to the build process.
This change is causing most of the tests to crash though as the testing process installs the schema from scratch, and the sequences are returning values that already exist in the tables. Is there any way to programtically say "Update sequences to max value in column" or do I need to write out a whole script by hand that updates all these sequences, or can I/should I change the trigger that substitutes the null value for the sequence to some how check this (though I think this might cause the mutating table problem)?
You can generate a script to create the sequences with the start values you need (based on their existing values)....
SELECT 'CREATE SEQUENCE '||sequence_name||' START WITH '||last_number||';'
FROM ALL_SEQUENCES
WHERE OWNER = your_schema
(If I understand the question correctly)
Here's a simple way to update a sequence value - in this case setting the sequence to 1000 if it is currently 50:
alter sequence MYSEQUENCE increment by 950 nocache;
select MYSEQUENCE_S.nextval from dual;
alter sequence MYSEQUENCE increment by 1;
Kudos to the creators of PL/SQL Developer for including this technique in their tool.
As part of your schema rebuild, why not drop and recreate the sequence?
I tried to put it in a sentence but it is better to give an example:
SELECT * FROM someTable WHERE id = someID;
returns no rows
...
some time passes (no inserts are done to the table and no ID updates)
...
SELECT * FROM someTable WHERE id = someID;
returns one row!
Is it possible that some DB mechanism prevents first SELECT to return row?
Oracle log has no errors.
No transactions are rolled back when two selects are executed.
You can't see uncommitted data in another session. When did the commit happen?
EDIT1: Are you the only one using this database? Or did/do you have multiple sessions?
I think in another session you or someone else has inserted this row, you do your select and you don't see this row. After that a commit happens in the other session (maybe implicit because a session is closed) and then you see this row when you select again.
I can think of other explanations, but I first want to know are you only one using this database.
With read consistency as provided by Oracle, you should not see a row appear like that. If you are running in some mode with automatic commits, so that each statement is a self-contained transaction, then read consistency is not being violated. Which program are you using to access the database? I agree with the other observations; the row should not appear if your session is not inserting it and no other session is active at the same time. I don't know of a DBMS that indulges in spontaneous data generation.
Don't you have scheduled jobs in that Oracle?