I see below details for one of the deadlock detected in oracle 12g trace files but i am not getting why deadlock is happening here ?
Deadlock happens when thread 1 acquires lock on table1 or table rows but wait for table 2 rows and at the same time thread 2 acquires lock on table 2 rows byt wait for table1 rows
But i do not see the details which session is acquired the lock on which table and waiting for which resource . Any help what are the object which got locked here and
cause of it ?
Deadlock graph:
---------Blocker(s)-------- ---------Waiter(s)---------
Resource Name process session holds waits process session holds waits
TX-00290010-00015F75-00000000-00000000 295 1200 X 288 10 X
TX-00570012-00005D9B-00000000-00000000 288 10 X 295 1200 X
session 1200: DID 0001-0127-00014421 session 10: DID 0001-0120-00016BD1
session 10: DID 0001-0120-00016BD1 session 1200: DID 0001-0127-00014421
Rows waited on:
Session 1200: obj - rowid = 00051348 - BABRNIAARAAKfNLAAl
...
Session 10: obj - rowid = 000514F2 - BABRTyAAJAAKWbIAAY
....
----- Information for the OTHER waiting sessions -----
....
current SQL:
update employee set name=:1
----- End of information for the OTHER waiting sessions -----
Information for THIS session:
----- Current SQL Statement for this session (sql_id=5dfr2prw60rh1) -----
update department set address =:1 where id=:1
===================================================
Your output says the current session is trying to update a locked record in the department table (the "information for THIS session" output). The other session is trying to update every employee record (the "information for the OTHER waiting sessions" output). The current session must have updated a record in the employee table, blocking the other session, while the other session updated the record the current session is trying to update.
I assume this is some sort of exercise to cause a deadlock, since you're setting every employee record to the same name.
Related
I have got a java app which after x number of requests just hangs. Looking at the app, it hangs on the execution of a query against an oracle database. Checked the blocking session using this sql and getting this:
Clearly SID 68 is blocking the SID 10. But why isn't it showing the blocking sql? The output from v$lock shows the following locks:
How do I find the actual blocking sqls?
The reason it's not showing the blocking SQL statement is because this cannot be known.
Consider this scenario:
Session 1 - 'SELECT FOR UPDATE ... WHERE EMP_ID = '123" (thus locking that row)
SESSION 1 - .... some other SQL (still hasn't committed)
Session 1 - .... some other SQL (still hasn't committed)
Session 1 - .... some other SQL (still hasn't committed)
Session 1 - .... some other SQL (still hasn't committed)
Session 2 - 'SELECT FOR UPDATE ... WHERE EMP_ID = '123" (is blocked by session 1, which _still has not committed and so is still holding it's lock)
So, we can know that session 1 is the blocking session, but there is no view that will indicate that it was the sql 3 previous to it's current sql that actuall placed the lock.
In your case, the blocking session is inactive, you must look at PREV_SQL_ID on V$SESSION in order to identify the last sql executed by the session that remains inactive.
V$LOCK lists the locks currently held by the Oracle Database and outstanding requests for a lock or latch. There are many types of locks in Oracle that is why you have so many rows on that view.
Regards
I want to make a script where the oracle sessions which go into a deadlock are killed automatically.Is it possible to find out the session id for the sessions which go into a deadlock.As of now I have to bounce the database to remove the deadlock.Is any solution to this problem possible?
I want to make a script where the oracle sessions which go into a deadlock are killed automatically
EDIT Explained in a better way, corrected few sentences, and added a test case to demonstrate deadlock scenario.
Why do you want to re-invent the wheel? Oracle detects a deadlock automatically, throws ORA-00060: deadlock detected while waiting for resource, and rolls back one of the transactions involved in the deadlock which Oracle decided as the victim. The previous successful transactions are not rolled back. Even after the deadlock error, if a commit is issued, the previous successful transaction will be committed. At this time, the other session's transaction will also succeed and you could issue a commit. There is nothing that you need to explicitly do here. Deadlocks are automatically cleared -- you never need to clear them.
Usually, Oracle takes a second or two to detect a deadlock and throws the error.
You can try with a simple test case as demonstrated here : Understanding Oracle Deadlock
Let's look at a test case -
SQL> CREATE TABLE t_test(col_1 NUMBER, col_2 NUMBER);
Table created
SQL> INSERT INTO t_test VALUES(1,2);
1 row inserted
SQL> INSERT INTO t_test VALUES(3,4);
1 row inserted
SQL> COMMIT;
Commit complete
SQL> SELECT * FROM t_test;
COL_1 COL_2
---------- ----------
1 2
3 4
Note the time of each transaction, I have set time on timing on for a better understanding.
SESSION : 1
12:16:06 SQL> UPDATE t_test SET col_1 = 5 WHERE col_2=2;
1 row updated.
Elapsed: 00:00:00.00
SESSION : 2
12:16:04 SQL> UPDATE t_test SET col_1 = 6 WHERE col_2=4;
1 row updated.
Elapsed: 00:00:00.00
12:16:31 SQL> UPDATE t_test SET col_1 = 7 WHERE col_2=2;
At this point, SESSION 2 keeps waiting.
SESSION : 1
12:16:15 SQL> UPDATE t_test SET col_1 = 8 WHERE col_2=4;
At this point, SESSION 2 is the victim of deadlock, SESSION 1 is still waiting.
Let's look at the session details from SESSION 2 -
12:22:15 SQL> select sid,status,program,sql_id, state, wait_class, blocking_session_status, event from v$session where schemaname='LALIT' and program='sqlplus.exe';
SID STATUS PROGRAM SQL_ID STATE WAIT_CLASS BLOCKING_SE EVENT
---------- -------- --------------- ------------- ------------------- --------------- ----------- ----------------------------------------------------------------
14 ACTIVE sqlplus.exe 60qmqpmbmyhxn WAITED SHORT TIME Network NOT IN WAIT SQL*Net message to client
134 ACTIVE sqlplus.exe 5x0zg4qwus29v WAITING Application VALID enq: TX - row lock contention
Elapsed: 00:00:00.00
12:22:18 SQL>
So, v$session details when viewed in SESSION 2, i.e. SID 14, says the status is ACTIVE.
Let's look at the session details from another session, lets call it SESSION 3 for the sake. Remember, SESSION 1 is still waiting.
SQL> set time on timing on
12:24:41 SQL> select sid,status,program,sql_id, state, wait_class, blocking_session_status, event from v$session where schemaname='LALIT' and program='sqlplus.exe'
SID STATUS PROGRAM SQL_ID STATE WAIT_CLASS BLOCKING_SE EVENT
---------- -------- --------------- ------------- ------------------- ---------- ----------- ------------------------------
13 ACTIVE sqlplus.exe 60qmqpmbmyhxn WAITED SHORT TIME Network NOT IN WAIT SQL*Net message to client
14 INACTIVE sqlplus.exe WAITING Idle NO HOLDER SQL*Net message from client
134 ACTIVE sqlplus.exe 5x0zg4qwus29v WAITING Applicatio VALID enq: TX - row lock contention
n
Elapsed: 00:00:00.01
12:24:44 SQL>
So, for other sessions, SESSION 2, i.e. SID 14, is INACTIVE. SESSION 1 is still WAITING with event enq: TX - row lock contention.
Let's commit SESSION 2 -
12:22:18 SQL> commit;
Commit complete.
Elapsed: 00:00:00.01
12:25:43 SQL>
At this point, the lock is released for SESSION 1, let's commit session 1 as well -
12:16:15 SQL> UPDATE t_test SET col_1 = 8 WHERE col_2=4;
1 row updated.
Elapsed: 00:08:27.29
12:25:43 SQL> commit;
Commit complete.
Elapsed: 00:00:00.00
12:26:26 SQL>
Elapsed: 00:08:27.29 shows SESSION 1 was waiting that long till SESSION 2 was committed.
To summarize, here is the entire story of session 1 -
12:16:06 SQL> UPDATE t_test SET col_1 = 5 WHERE col_2=2;
1 row updated.
Elapsed: 00:00:00.00
12:16:15 SQL> UPDATE t_test SET col_1 = 8 WHERE col_2=4;
1 row updated.
Elapsed: 00:08:27.29
12:25:43 SQL> commit;
Commit complete.
Elapsed: 00:00:00.00
12:26:26 SQL>
To summarize, here is the entire story of session 2 -
12:16:04 SQL> UPDATE t_test SET col_1 = 6 WHERE col_2=4;
1 row updated.
Elapsed: 00:00:00.00
12:16:31 SQL> UPDATE t_test SET col_1 = 7 WHERE col_2=2;
UPDATE t_test SET col_1 = 7 WHERE col_2=2
*
ERROR at line 1:
ORA-00060: deadlock detected while waiting for resource
Elapsed: 00:00:24.47
12:22:15 SQL> select sid,status,program,sql_id, state, wait_class, blocking_session_status, event from v$session where schemaname='LALIT' and program='sqlplus.exe';
SID STATUS PROGRAM SQL_ID STATE WAIT_CLASS BLOCKING_SE EVENT
---------- -------- --------------- ------------- ------------------- --------------- ----------- ----------------------------------------------------------------
14 ACTIVE sqlplus.exe 60qmqpmbmyhxn WAITED SHORT TIME Network NOT IN WAIT SQL*Net message to client
134 ACTIVE sqlplus.exe 5x0zg4qwus29v WAITING Application VALID enq: TX - row lock contention
Elapsed: 00:00:00.00
12:22:18 SQL> commit;
Commit complete.
Elapsed: 00:00:00.01
12:25:43 SQL>
Now, let's see which transaction actually got rolled back and which got committed -
12:25:43 SQL> select * from t_test;
COL_1 COL_2
---------- ----------
5 2
8 4
Elapsed: 00:00:00.00
12:30:36 SQL>
Conclusion
In my opinion, the best way to know the session details of a deadlock is to log the details as verbose as possible. Else, it is a nightmare for a DBA to investigate without proper information logged. For that matter, even a Developer would find it to be an herculean task to rectify and fix the actual design flaw if the deadlock error details are not logged verbosely. And to conclude with a one liner statement, A deadlock is due to design flaw, Oracle is just the victim and the application being the culprit. Deadlocks are scary, but they point out the design flaws that must be rectified sooner or later.
user 1
update table_c set id = 200 where id = 13;
BEGIN
DBMS_LOCK.sleep(14);
END;
/
update table_c set id = 200 where id = 15;
user 2
update table_c set id = 2000 where id = 15;
BEGIN
DBMS_LOCK.sleep(14);
END;
/
update table_c set id = 1000 where id = 13;
What is a Deadlock?
A deadlock occurs when a session (A) wants a resource held by another session (B) , but that session also wants a resource held by the first session (A). There can be more than 2 sessions involved but the idea is the same.
Diagnostic information produced by an ORA-60
ORA-60 error normally writes the error message in the alert.log together with the name of the trace file created. The exact format of this varies between Oracle releases. The trace
file will be written to the directory indicated by the USER_DUMP_DEST or BACKGROUND_DUMP_DEST, depending on the type of process that creates the trace file.
Resolving Deadlock
Oracle is smart and it is able to find deadlock situation within 3 seconds. Below are the options to avoid deadlocks inside the database
Ask the session getting deadlock error ORA-00060 to issue either COMMIT or ROLLBACK
Ask the waiting session to kill the SQL / transaction
Look inside alert log / trace file for the sessions involved in deadlock and inform application team to improve the code
As a DBA, whenever you get a Deadlock alert, immediately contact application team and inform them.
I'm having greate problems with Magento last friday we upgraded Magento from 1.7 to 1.8..
The issue is that we're having a lot of deadlocks in the MySQL database.
Our server setup is
1 Load Balancer
4 Webservers (Apache, PHP5, APC)
2 MySQL Servers (64 GB Ram, 30 cores SSD HDD) - 1 Master (Has Memcache for sessions) - 1 Slave (Has Redis for caching)
The deadlock's is less on Magento 1.8 than 1.7 but the still appear from time to time ..
Any one has some good ideas on how to get pass this problem.
Heres some data from SHOW ENGINE INNODB STATUS;
LATEST DETECTED DEADLOCK
130930 12:03:35
* (1) TRANSACTION:
TRANSACTION 918EEC3B, ACTIVE 37 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 41 lock struct(s), heap size 6960, 50 row lock(s), undo log entries 6
MySQL thread id 51899, OS thread handle 0x7f9774169700, query id 2583719 xxx.xx.xxx.47 dbxxx Updating
UPDATE m17_index_process SET started_at = '2013-09-30 10:03:36' WHERE (process_id='8')
* (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 594 page no 3 n bits 208 index PRIMARY of table xxx.xx.xxx.47 dbxxx.m17_index_process trx id 918EEC3B lock_mode X locks rec but not gap waiting
* (2) TRANSACTION:
TRANSACTION 918EE3E7, ACTIVE 72 sec starting index read
mysql tables in use 1, locked 1
680 lock struct(s), heap size 80312, 150043 row lock(s), undo log entries 294
MySQL thread id 51642, OS thread handle 0x7f8a336c7700, query id 2586254 xxx.xx.xxx.47 dbxxx Updating
UPDATE m17_index_process SET started_at = '2013-09-30 10:03:40' WHERE (process_id='8')
(2) HOLDS THE LOCK(S):
RECORD LOCKS space id 594 page no 3 n bits 208 index PRIMARY of table dbxxx.m17_index_process trx id 918EE3E7 lock mode S locks rec but not gap
(2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 594 page no 3 n bits 208 index PRIMARY of table dbxxx.m17_index_process trx id 918EE3E7 lock_mode X locks rec but not gap waiting
* WE ROLL BACK TRANSACTION (1)
Best Regards.
Rasmus
Seems deadlocks are due to indexing processes. Try disabling automatic indexes Magento - Programmatically Disable Automatic Indexing
and doing them manually.
Also try disabling cron for some time and check if issues reoccur.
Its possible that many store admins saving products from different stores. In that case product save may be causing deadlock with index processes.
Thanks
I have been getting this "ora-00060 deadlock detected while waiting for resource" error often now in my application when multiple users are using the application. I have got the trace file from the oracle Admin, but need help in reading it. Below is bits of data from the trace file, which i hope would help in locating the cause.
*** 2013-06-25 09:37:35.324
DEADLOCK DETECTED ( ORA-00060 )
[Transaction Deadlock]
The following deadlock is not an ORACLE error. It is a deadlock due
to user error in the design of an application
or from issuing incorrect ad-hoc SQL. The following
information may aid in determining the deadlock:
Deadlock graph:
---------Blocker(s)-------- ---------Waiter(s)---------
Resource Name process session holds waits process session holds waits
TM-000151a2-00000000 210 72 SX SSX 208 24 SX SSX
TM-000151a2-00000000 208 24 SX SSX 210 72 SX SSX
session 72: DID 0001-00D2-000000C6 session 24: DID 0001-00D0-00000043
session 24: DID 0001-00D0-00000043 session 72: DID 0001-00D2-000000C6
Rows waited on:
Session 72: no row
Session 24: no row
----- Information for the OTHER waiting sessions -----
Session 24:
sid: 24 ser: 45245 audsid: 31660323 user: 90/USER
flags: (0x45) USR/- flags_idl: (0x1) BSY/-/-/-/-/-
flags2: (0x40009) -/-/INC
pid: 208 O/S info: user: zgrid, term: UNKNOWN, ospid: 2439
image: oracle#xyz.local
client details:
O/S info: user: , term: , ospid: 1234
machine: xyz.local program:
current SQL:
delete from EMPLOYEE where EMP_ID=:1
----- End of information for the OTHER waiting sessions -----
Information for THIS session:
----- Current SQL Statement for this session (sql_id=dyfg1wd8xa9qt) -----
delete from EMPLOYEE where EMP_ID=:1
===================================================
I would appreciate if some one can tell me what the "Deadlock graph::" is saying. Also the rows waited on section says no rows.
I also read in some blogs that "sqltxt" section from the trace file can suggest the cause. Below is the query i see in that section.
select /*+ all_rows */ count(1) from "USERS"."EMPLOYEE_SALARY" where EMPSAL_EMP_ID=:1
The employee_salary table has foreignkey constraint on EMPSAL_EMP_ID column.
The sql hint says "all_rows", so does it mean that this table gets table level lock when deleting records from employee table? i dont have an index on the foreign key column currently. Would adding an index on this column help?
Kindly post, in case any more information is need.
Thanks
First of all, select statement never lock anything in Oracle, just uses last available consistent version of data. It's not a case for select ... for update which locks data like update since Oracle 9i, but there are no for update clause in the query from question.
Resource Name process session holds waits process session holds waits
TM-000151a2-00000000 210 72 SX SSX 208 24 SX SSX
Session #72 holds table-level lock (TM) with "Row Exclusive" type (SX) and want to acquire "Share Row Exclusive" (SSX) lock on same table. This session blocked by Session #24 which already holds table-level lock of a same type (SX) and waits while SSX lock would be available.
Resource Name process session holds waits process session holds waits
TM-000151a2-00000000 208 24 SX SSX 210 72 SX SSX
This (second row) demonstrates exactly same situation, but in opposite direction: Session #24 waits for SSX lock become available, but blocked by Session #72 which already holds SX lock on same table.
So, Sessions #24 and Session #72 blocks each other: deadlock happens.
Both lock types (SX and SSX) are table-level locks.
To understand the situation I recommend to read this article by Franck Pachot.
Below is citation from this article, which directly relevant to your situation(note that SSX and SRX abbreviations are equivalent):
Referential integrity also acquires TM locks. For example, the common
issue with unindexed foreign keys leads to S locks on child table when
you issue a delete, or update on the key, on the parent table. This is
because without an index, Oracle has no single lower level resource to
lock in order to prevent a concurrent insert that can violate the
referential integrity.
When the foreign key columns are the leading
columns in a regular index, then the first index entry with the parent
value can be used as a single resource and locked with a row level TX
lock.
And what if referential integrity has an on delete cascade? In
addition to the S mode, there is the intention to update rows in the
child table, as with Row X (RX) mode. This is where the share row
exclusive (SRX) occurs: S+RX=SRX.
So, most probable variant is that Session #72 and Session #24 deletes some rows in EMPLOYEE table at same time, and there are on delete cascade constraint for EMPSAL_EMP_ID in conjunction with absence of index on EMPLOYEE_SALARY table in which EMPSAL_EMP_ID column listed first.
In Oracle 10+ versions, can update and delete on the same table cause deadlocks even if they are operating on different rows of same table concurrently?
The table has primary key made-up of two columns, and do not have any FK associated/refereed with any other table. And there is no parent/child relation with other table
What I believe is, it will not create a deadlock, but I'm facing a issue in my application.
adding the oracle trace :
The following deadlock is not an ORACLE error. It is a deadlock due to user error in the design of an application or from issuing incorrect ad-hoc SQL. The following information may aid in determining the deadlock:
Deadlock graph:
---------Blocker(s)-------- ---------Waiter(s)---------
Resource Name process session holds waits process session holds waits
TX-0007003e-0081d6c3 45 790 X 104 20 X
TX-00080043-0085e6be 104 20 X 45 790 X
session 790: DID 0001-002D-000035F9 session 20: DID 0001-0068-000007F6
session 20: DID 0001-0068-000007F6 session 790: DID 0001-002D-000035F9
Rows waited on:
Session 790: obj - rowid = 0000F0C8 - AAAPDIAAMAAAEfIAAA
(dictionary objn - 61640, file - 12, block - 18376, slot - 0)
Session 20: obj - rowid = 0000F0C8 - AAAPDIAAMAAAEfGAAA
(dictionary objn - 61640, file - 12, block - 18374, slot - 0)
----- Information for the OTHER waiting sessions ----- Session 20:
sid: 20 ser: 4225 audsid: 57496371 user: 72/RPT_TABLE
flags: (0x45) USR/- flags_idl: (0x1) BSY/-/-/-/-/-
flags2: (0x40009) -/-/INC
pid: 104 O/S info: user: oracle, term: UNKNOWN, ospid: 20798
image: oracle#caidb10p-node1
client details:
O/S info: user: gtsgen, term: unknown, ospid: 1234
machine: caiapp08p-node0.nam.nsroot.net program: JDBC Thin Client
application name: JDBC Thin Client, hash value=2546894660
current SQL:
delete from RPT_TABLE.TEMP_TABLE_T1 where TEMP_T1_ID=:1
----- End of information for the OTHER waiting sessions -----
Information for THIS session:
----- Current SQL Statement for this session (sql_id=bsaxpc2bdps9q) ----- UPDATE RPT_TABLE.TEMP_TABLE_T1 temp1 SET temp1.CLIENT_ID = (SELECT MIN(INVMAP.CLIENT_ID) FROM LI_REF.REF_CLIENT_MAP INVMAP WHERE INVMAP.F_CODE = :B2 AND INVMAP.AID = temp1.ID AND temp1.R_ID=:B1 )
----- PL/SQL Stack -----
----- PL/SQL Call Stack -----
object line object
handle number name
45887d750 24 procedure RPT_TABLE.T1_UPDATE_StoredProc
6399ba188 1 anonymous block
If you could update your question with the deadlock graph, that would be useful information. (When your application encounters a deadlock, Oracle will raise an ORA-00060, and a tracefile will be written to the user_dump_dest.) If you look in the trace file, you'll find a section called the "Deadlock Graph". If you can post that, and also post the statement that caused the deadlock and other statements involved in the deadlock, then we can begin to draw some conclusions. (All the information I requested is available in the trace file.)
As Alessandro mentioned, it's possible for sessions locking different rows in the same table to deadlock due to unindexed foreign keys on the child table of a parent/child relationship. Also, It's possible that you could have deadlocks on two sessions updating different rows of the same table, even if the table is not part of a parent/child relationship, if, for example, the table has a shortage of ITL entries.
Again, post the information requested above, and I'm confident we can determine the root cause of your deadlock.
Added on 7/30/2012 **
Adding the following, now that the deadlock trace file has been supplied:
Ok, first off, based on the trace file contents, this is a simple deadlock due to sessions overlapping/colliding on the rows they are trying to lock. Despite your previous comments about the deadlock being on different rows, I'm here to tell you that this particular deadlock is due to row-level locking on the same rows.
The fact that the deadlock graph shows mode the lock is held in is 'X' (exclusive) and the mode the lock is waited on is 'X', tells me this is simple row-level locking.
In this case, SID 20 is executing "delete from RPT_TABLE.TEMP_TABLE_T1 where TEMP_T1_ID=:1" and already has a lock on rowid AAAPDIAAMAAAEfIAAA.
Meanwhile, SID 790 is executing "RPT_TABLE.T1_UPDATE_StoredProc", while already holding a lock on rowid AAAPDIAAMAAAEfGAAA.
Note from the "Rows waited on" section of the tracefile, that SID 20 is waiting on the row that SID 790 holds and SID 790 is waiting on the row that SID 20 is holding. This is a classic deadlock.
Some additional information:
Enqueue type is TX (see the deadlock graph), so, this is definitely not locking due to unindexed foreign keys. If it were locking due to unindexed FKs, the enqueue type would be TM, not TX. (There is at least one other case where TM enqueues are involved, and it's not unindexed FKs. So, don't assume that TM enqueue always means unindexed FKs.)
The mode the lock is being waited on is 'X' (exclusive), so this is row-level locking. If the mode waited on was 'S' (shared), then it would not be row-level locking. Rather, it could be ITL shortage or PK or UK enforcement.
Hope that helps!
I don't know if you have foreign keys involved in your application but it could probably be the source of your locks. If so take a look at these links:
http://docs.oracle.com/cd/E11882_01/server.112/e16508/consist.htm#BABCAHDJ
http://docs.oracle.com/cd/E11882_01/server.112/e16508/datainte.htm#CNCPT1657
Oracle Database maximizes the concurrency control of parent keys in relation to dependent foreign keys. Locking behaviour depends on whether foreign key columns are indexed. If foreign keys are not indexed, then the child table will probably be locked more frequently, deadlocks will occur, and concurrency will be decreased. For this reason foreign keys should almost always be indexed. The only exception is when the matching unique or primary key is never updated or deleted.
Locks and Unindexed Foreign Keys
When both of the following conditions are true, the database acquires a full table lock on the child table:
No index exists on the foreign key column of the child table.
A session modifies a primary key in the parent table (for example, deletes a row or modifies primary key attributes) or merges rows into the parent table. Inserts into the parent table do not acquire table locks on the child table.
If this is not your case try to provide more informations about it. Tell us about the Kind of locks Holden/requested by the sessions and take a look at the system tables V$LOCK, V$LOCKED_OBJECT, DBA_DDL_LOCKS, DBA_DML_LOCKS or V$SESSION_WAIT.