I have a quite complicated PL/SQL procedure selecting data dynamically and multiple times. Eventually, the data is aggregated and returned.
However, due to multiple selects, the returned data might be inconsistant, because data changes might happen between the different selects. A process might look like this:
Select data 1
Change from a different session
Select data 2
--> Data 2 might now be inconsistant with data 1 due to the changes.
Unfortunately, I cannot select all data at once with a single select. This would be too complicated.
Is there a way in Oracle to somehow "freeze" the data state (not respecting any incoming changes) until the procedure is finished?
You are looking for Oracle's Serializable Isolation Level:
In the serializable isolation level, a transaction sees only changes committed at the time the transaction—not the query—began and changes made by the transaction itself.
It is started like this:
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
Related
Is there a way to retrieve output from PL/SQL continuously rather than wait until the SP completes its execution. Continuously mean as when it executes the execute immediate.
Any other mechanism to retrieve pl/sql output?
As per Oracle docs
Output that you create using PUT or PUT_LINE is buffered in the SGA. The output cannot be retrieved until the PL/SQL program unit from which it was buffered returns to its caller. So, for example, Enterprise Manager or SQL*Plus do not display DBMS_OUTPUT messages until the PL/SQL program completes.
As far as I know, there is a way, but not with DBMS_OUTPUT.PUT_LINE. Technique I use is:
create a log table which will accept values you'd normally display using DBMS_OUTPUT.PUT_LINE. Columns I use are
ID (a sequence, to be able to sort data)
Date (to know what happened when; might not be enough for sorting purposes because operations that take very short time to finish might have the same timestamp)
Message (a VARCHAR2 column, large enough to accept the whole information)
create a logging procedure which will be inserting values into that table. It should be an autonomous transaction so that you could COMMIT within (and be able to access data from other sessions), without affecting the main transaction
Doing so, you'd
start your PL/SQL procedure
call the logging procedure whenever appropriate (basically, where you'd put the DBMS_OUTPUT.PUT_LINE call)
in another session, periodically query the log table as select * from log_table order by ID desc
Additionally, you could write a simple Apex application with one report page which selects from the logging table and refreshes periodically (for example, every 10 seconds or so) and view the main PL/SQL procedure's execution.
The approach that Littlefoot has provided is what I normally use as well.
However, there is another approach that you can try for a specific use case. Let's say you have a long-running batch job (like a payroll process for example). You do not wish to be tied down in front of the screen monitoring the progress. But you want to know as soon as the processing of any of the rows of data hits an error so that you can take action or inform a relevant team. In this case, you could add code to send out emails with all the information from the database as soon as the processing of a row hits an error (or meets any condition you specify).
You can do this using the functions and procedures provided in the 'UTL_MAIL' package. UTL_MAIL Documentation from Oracle
For monitoring progress without the overhead of logging to tables and autonomous transactions. I use:
DBMS_APPLICATION.SET_CLIENT_INFO( TO_CHAR(SYSDATE, 'HH24:MI:SS') || ' On step A' );
and then monitor in v$session.client_infofor your session. It's all in memory and won't persist of course but it is a quick and easy ~zero cost way of posting progress.
Another option (Linux/UNIX) for centralised logging that is persistent and again avoids logging in the database more generally viewable that I like is interfacing to syslog and having Splunk or similar pick these up. If you have Splunk or similar then this makes the monitoring viewable without having to connect to the database query directly. See this post here for how to do this.
https://community.oracle.com/thread/2343125
I don't use Stored procedures very often and was wondering if it made sense to wrap my select queries in a transaction.
My procedure has three simple select queries, two of which use the returned value of the first.
In a highly concurrent application it could (theoretically) happen that data you've read in the first select is modified before the other selects are executed.
If that is a situation that could occur in your application you should use a transaction to wrap your selects. Make sure you pick the correct isolation level though, not all transaction types guarantee consistent reads.
Update :
You may also find this article on concurrent update/insert solutions (aka upsert) interesting. It puts several common methods of upsert to the test to see what method actually guarantees data is not modified between a select and the next statement. The results are, well, shocking I'd say.
Transactions are usually used when you have CREATE, UPDATE or DELETE statements and you want to have the atomic behavior, that is, Either commit everything or commit nothing.
However, you could use a transaction for READ select statements to:
Make sure nobody else could update the table of interest while the bunch of your select query is executing.
Have a look at this msdn post.
Most databases run every single query in a transaction even if not specified it is implicitly wrapped. This includes select statements.
PostgreSQL actually treats every SQL statement as being executed within a transaction. If you do not issue a BEGIN command, then each individual statement has an implicit BEGIN and (if successful) COMMIT wrapped around it. A group of statements surrounded by BEGIN and COMMIT is sometimes called a transaction block.
https://www.postgresql.org/docs/current/tutorial-transactions.html
I have a use case where I need to do the following things in one transaction:
start the transaction
INSERT an item into a table
SELECT all the items in the table
dump the selected items into a file (this file is versioned and another program always uses the latest version)
If all the above things succeed, commit the transaction, if not, rollback.
If two transactions begin almost simultaneously, it is possible that before the first transaction A commits what it has inserted into the table (step 4), the second transaction B has already performed the SELECT operation(step 2) whose result doesn't contain yet the inserted item by the first transaction(as it is not yet committed by A, so not visible to B). In this case, when A finishes, it will have correctly dumped a file File1 containing its inserted item. Later, B finishes, it will have dumped another file File2 containing only its inserted item but not the one inserted by A. Since File2 is more recent, we will use File2. The problem is that File2 doesn't contain the item inserted by A even though this item is well in the DB.
I would like to know if it is feasible to solve this problem by locking the read(SELECT) of the table when a transaction inserts something into the table until its commit or rollback and if yes, how this locking can be implemented in Spring with Oracle as DB.
You need some sort of synchronization between the transactions:
start the transaction
Obtain a lock to prevent the transaction in another session to proceed or wait until the transaction in the other session finishes
INSERT an item into a table
SELECT ......
......
Commit and release the lock
The easiest way is to use LOCK TABLE command, at least in SHARE mode (SHARE ROW EXCLUSIVE or EXCLUSIVE modes can also be used, but they are too restrictve for this case).
The advantage of this approach is that the lock is automatically released at commit or rollback.
The disadvantage is a fact, that this lock can interfere with other transactions in the system that update this table at the same time, and could reduce an overall performance.
Another approach is to use DBMS_LOCK package. This lock doesn't affect other transactions that don't explicitely use that lock. The drawaback is that this package is difficult to use, the lock is not released on commit nor rollback, you must explicitelly release the lock at the end of the transaction, and thus all exceptions must be carefully handled, othervise a deadlock easily could occur.
One more solution is to create a "dummy" table with a single row in it, for example:
CREATE TABLE my_special_lock_table(
int x
);
INSERT INTO my_special_lock_table VALUES(1);
COMMIT:
and then use SELECT x FROM my_special_lock_table FOR UPDATE
or - even easier - simple UPDATE my_special_lock_table SET x=x in your transaction.
This will place an exclusive lock on a row in this table and synchronize only this one transaction.
A drawback is that another "dummy" table must be created.
But this solution doesn't affect the other transactions in the system, the lock is automatically released upon commit or rollback, and it is portable - it should work in all other databases, not only in Oracle.
Use spring's REPEATABLE_READ or SERIALIZABLE isolation levels:
REPEATABLE_READ A constant indicating that dirty reads and
non-repeatable reads are prevented; phantom reads can occur. This
level prohibits a transaction from reading a row with uncommitted
changes in it, and it also prohibits the situation where one
transaction reads a row, a second transaction alters the row, and the
first transaction rereads the row, getting different values the second
time (a "non-repeatable read").
SERIALIZABLE A constant indicating that dirty reads, non-repeatable
reads and phantom reads are prevented. This level includes the
prohibitions in ISOLATION_REPEATABLE_READ and further prohibits the
situation where one transaction reads all rows that satisfy a WHERE
condition, a second transaction inserts a row that satisfies that
WHERE condition, and the first transaction rereads for the same
condition, retrieving the additional "phantom" row in the second read.
with serializable or repeatable read, the group will be protected from non-repeatable reads:
connection 1: connection 2:
set transaction isolation level
repeatable read
begin transaction
select name from users where id = 1
update user set name = 'Bill' where id = 1
select name from users where id = 1 |
commit transaction |
|--> executed here
In this scenario, the update will block until the first transaction is complete.
Higher isolation levels are rarely used because they lower the number of people that can work in the database at the same time. At the highest level, serializable, a reporting query halts any update activity.
I think you need to serialize the whole transaction. While a SELECT ... FOR UPDATE could work, it does not really buy you anything, since you would be selecting all rows. You may as well just take and release a lock, using DBMS_LOCK()
When doing concurrent MERGEs while every session uses a different value (shown as *** in the snippet below) for the primary key column id, everything is fine if I do it manually in 2 terminal sessions.
MERGE
INTO x
USING (SELECT *** as id FROM DUAL) MERGE_SRC
ON (x.id = MERGE_SRC.id)
WHEN MATCHED THEN UPDATE SET val = val + 1 WHERE id = ***
WHEN NOT MATCHED THEN INSERT VALUES (***, 99);
COMMIT;
However, running a multi-threaded load test with 3 or more threads, I will relatively quickly run into ORA-08177 with locked table. Why is that? (And why is it non-deterministic in that it does not always happen when transactions overlap?)
The table was created using
create table x (id int primary key, val int);
SQL Server btw never throws exceptions with an equivalent MERGE statement, running the same experiment. That is even true when working on the same row simultaneously.
Is it because perhaps MERGE is not atomic, and the serializable mode runs optimistically, so that the race might only show with sufficient contention? Still, why does it happen even when not working on the same row concurrently?
Btw, my attempts to fix this using the strictest lock available were unsuccessful. So any ideas on how to make this atomic are very much appreciated. It looks like relaxing the isolation level would rid me of the exception, but risk inconsistencies in case there turn out to be 2 updates on the same row (otherwise why would it balk in serializable mode in the first place).
The exception you're seeing is a direct consequence of using strict serialization. If you have more than one transaction active simultaneously, each started with SET TRANSACTION ISOLATION LEVEL SERIALIZABLE, when any one of them commits the others will get an ORA-08177. That's how strict serialization is enforced - the database throws an ORA-08177 in any session started with ISOLATION LEVEL SERIALIZABLE if another transaction commits into a table which the serializable session needs. So, basically, if you really need strict serialization you have to handle the ORA-08177's intelligently, as in the following:
DECLARE
bSerializable_trans_complete BOOLEAN := FALSE;
excpSerializable EXCEPTION;
PRAGMA EXCEPTION_INIT(excpSerializable, -08177);
BEGIN
<<SERIALIZABLE_LOOP>>
WHILE NOT bSerializable_trans_complete
LOOP
BEGIN
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
MERGE ...; -- or whatever
COMMIT;
bSerializable_trans_complete := TRUE; -- allow SERIALIZABLE_LOOP to exit
EXCEPTION
WHEN excpSerializable THEN
ROLLBACK;
CONTINUE SERIALIZABLE_LOOP;
END;
END LOOP; -- SERIALIZABLE_LOOP
END;
Serialization is not magic, and it's not "free" (where "free" means "I as the developer don't have to do anything to make it work properly"). It requires more planning and work on the part of the developer to have it function properly, not less. Share and enjoy.
In Oracle, the SERIALIZABLE mode works optimistically, in contrast to e.g. SQL Server, which does pessimistic locking in that mode. Which means that in the latter case you can even concurrently change the same row without running into exceptions.
Despite the docs:
Oracle Database permits a serializable transaction to modify a row only if changes to the row made by other transactions were already committed when the serializable transaction began. The database generates an error when a serializable transaction tries to update or delete data changed by a different transaction that committed after the serializable transaction began:
load testing showed that exceptions may also get thrown when not working on the same row simultaneously, although that is not guaranteed, in contrast to when working on the same row, which will always result in an ORA-08177.
In oracle 10g, I have one stored procedure per table per operation (insert, select, update, delete). Actually there can be multiple procedures per table per operation, for example in case of select it can be SelectList, SelectOneRecord, Search (with dynamic query).
None of these procedures have transactions.
Sometimes I have to combine multiple operations in a transaction. For example insert in one table and update in another table, all in one transaction. For this I make a separate procedure which has transaction. This procedure then call the two procedures.
To enable the above combination-of-procedure-calls-in-single-transaction, I do not put any transaction behavior in the procedures, as discussed above.
Most of the time I have to perform only one operation, such as insert in one table. Since the insert procedure do not have transaction behavior therefore I have to make a separate procedure that have transaction behavior and that procedure call the insert procedure.
I end up with lots of basic procedures (one table, one operation) and lots of transaction procedures that are basically wrappers around basic procedures.
My question is, is there some way to have conditional transactional behavior in the basic procedures. By this I mean that some if-condition where I can put the transaction logic, so that the transaction behavior can be on or off based on some parameter that I pass. Then when I want to do only one operation, such as insert in a table, I call the basic procedure with transaction behavior; and when I want to call two procedures in a transaction, such as insert in one table and update in another table all in one transaction, then I make a separate transaction procedure and call the two basic procedure without transaction behavior.
The following is a transaction procedure that call another procedure and wrap it in a transaction:
BEGIN
SAVEPOINT the_start;
BasicProcedure(<list of parameters>);
COMMIT;
EXCEPTION
WHEN OTHERS THEN
BEGIN
ROLLBACK TO the_start;
RAISE;
END;
END;
I can very well put the savepoint line and the commit line in if-statements, but can I also put the exception block in a if-statement. Do I have to put the exception block in if-statement? What if I catch exception in procedure, would it automatically rollback when exception comes?
What you have written is called a Table API. Some people swear by them, others anathematize them. The case for Table APIs is basically: modularity and code reuse. The case against is primarily performance: they encourage row by row or table by table processing, when a bespoke SQL join would be more efficient.
Personally I have stood on both sides of the fence. Currently I favour tailored code: it makes transactions easier to work with.
Which brings me to your situation. Table APIs are supposed to be generic and usable in all situations. That means they cannot control the management of transactions: that properly belongs to the programs which call the Table API methods. These programs are the code which implements the business logic: a business transaction consists of a number of activities which constitute a Unit Of Work. All of these have to succeed in order for the transaction to succeed, otherwise the business transaction needs to be rolled back. If the Table API commands issue their own commits, a subsequent failure would leave the business transaction in an inconsistent state. ACID applies at this level as well as the individual SL level.
This is actually no different from writing stored procedures with bespoke SQL in them.
"it's unclear to me if you're advocating making the business logic
transactions stored procedures."
This is a large area to cover, and there's more to design PL/SQL applications than business logic. (If you have access to a time machine you should travel back to Open World 2009 to catch my presentation on "Designing PL/SQL with Intent")
But broadly, yes, the outward-facing aspect of the PL/SQL layer should consist of business logic APIs, organised around Units of Work i.e. business transactions.
"such procedures be autonomous transactions? "
Absolutely not. Autonomous transactions are fit for only one purpose: logging and auditing of activity, where we need a permanent record of what occurred without affecting the wider transaction. Any other use of that pragma is a kludge or a data corruption bug waiting to happen.
Business transactions are transactions pure and simple. Either the stored procedures should own the commit or else they should defer it to the calling program.