Finding cause of deadlock error from oracle trace file - oracle

I have been getting this "ora-00060 deadlock detected while waiting for resource" error often now in my application when multiple users are using the application. I have got the trace file from the oracle Admin, but need help in reading it. Below is bits of data from the trace file, which i hope would help in locating the cause.
*** 2013-06-25 09:37:35.324
DEADLOCK DETECTED ( ORA-00060 )
[Transaction Deadlock]
The following deadlock is not an ORACLE error. It is a deadlock due
to user error in the design of an application
or from issuing incorrect ad-hoc SQL. The following
information may aid in determining the deadlock:
Deadlock graph:
---------Blocker(s)-------- ---------Waiter(s)---------
Resource Name process session holds waits process session holds waits
TM-000151a2-00000000 210 72 SX SSX 208 24 SX SSX
TM-000151a2-00000000 208 24 SX SSX 210 72 SX SSX
session 72: DID 0001-00D2-000000C6 session 24: DID 0001-00D0-00000043
session 24: DID 0001-00D0-00000043 session 72: DID 0001-00D2-000000C6
Rows waited on:
Session 72: no row
Session 24: no row
----- Information for the OTHER waiting sessions -----
Session 24:
sid: 24 ser: 45245 audsid: 31660323 user: 90/USER
flags: (0x45) USR/- flags_idl: (0x1) BSY/-/-/-/-/-
flags2: (0x40009) -/-/INC
pid: 208 O/S info: user: zgrid, term: UNKNOWN, ospid: 2439
image: oracle#xyz.local
client details:
O/S info: user: , term: , ospid: 1234
machine: xyz.local program:
current SQL:
delete from EMPLOYEE where EMP_ID=:1
----- End of information for the OTHER waiting sessions -----
Information for THIS session:
----- Current SQL Statement for this session (sql_id=dyfg1wd8xa9qt) -----
delete from EMPLOYEE where EMP_ID=:1
===================================================
I would appreciate if some one can tell me what the "Deadlock graph::" is saying. Also the rows waited on section says no rows.
I also read in some blogs that "sqltxt" section from the trace file can suggest the cause. Below is the query i see in that section.
select /*+ all_rows */ count(1) from "USERS"."EMPLOYEE_SALARY" where EMPSAL_EMP_ID=:1
The employee_salary table has foreignkey constraint on EMPSAL_EMP_ID column.
The sql hint says "all_rows", so does it mean that this table gets table level lock when deleting records from employee table? i dont have an index on the foreign key column currently. Would adding an index on this column help?
Kindly post, in case any more information is need.
Thanks

First of all, select statement never lock anything in Oracle, just uses last available consistent version of data. It's not a case for select ... for update which locks data like update since Oracle 9i, but there are no for update clause in the query from question.
Resource Name process session holds waits process session holds waits
TM-000151a2-00000000 210 72 SX SSX 208 24 SX SSX
Session #72 holds table-level lock (TM) with "Row Exclusive" type (SX) and want to acquire "Share Row Exclusive" (SSX) lock on same table. This session blocked by Session #24 which already holds table-level lock of a same type (SX) and waits while SSX lock would be available.
Resource Name process session holds waits process session holds waits
TM-000151a2-00000000 208 24 SX SSX 210 72 SX SSX
This (second row) demonstrates exactly same situation, but in opposite direction: Session #24 waits for SSX lock become available, but blocked by Session #72 which already holds SX lock on same table.
So, Sessions #24 and Session #72 blocks each other: deadlock happens.
Both lock types (SX and SSX) are table-level locks.
To understand the situation I recommend to read this article by Franck Pachot.
Below is citation from this article, which directly relevant to your situation(note that SSX and SRX abbreviations are equivalent):
Referential integrity also acquires TM locks. For example, the common
issue with unindexed foreign keys leads to S locks on child table when
you issue a delete, or update on the key, on the parent table. This is
because without an index, Oracle has no single lower level resource to
lock in order to prevent a concurrent insert that can violate the
referential integrity.
When the foreign key columns are the leading
columns in a regular index, then the first index entry with the parent
value can be used as a single resource and locked with a row level TX
lock.
And what if referential integrity has an on delete cascade? In
addition to the S mode, there is the intention to update rows in the
child table, as with Row X (RX) mode. This is where the share row
exclusive (SRX) occurs: S+RX=SRX.
So, most probable variant is that Session #72 and Session #24 deletes some rows in EMPLOYEE table at same time, and there are on delete cascade constraint for EMPSAL_EMP_ID in conjunction with absence of index on EMPLOYEE_SALARY table in which EMPSAL_EMP_ID column listed first.

Related

Root cause of deadlock?

I see below details for one of the deadlock detected in oracle 12g trace files but i am not getting why deadlock is happening here ?
Deadlock happens when thread 1 acquires lock on table1 or table rows but wait for table 2 rows and at the same time thread 2 acquires lock on table 2 rows byt wait for table1 rows
But i do not see the details which session is acquired the lock on which table and waiting for which resource . Any help what are the object which got locked here and
cause of it ?
Deadlock graph:
---------Blocker(s)-------- ---------Waiter(s)---------
Resource Name process session holds waits process session holds waits
TX-00290010-00015F75-00000000-00000000 295 1200 X 288 10 X
TX-00570012-00005D9B-00000000-00000000 288 10 X 295 1200 X
session 1200: DID 0001-0127-00014421 session 10: DID 0001-0120-00016BD1
session 10: DID 0001-0120-00016BD1 session 1200: DID 0001-0127-00014421
Rows waited on:
Session 1200: obj - rowid = 00051348 - BABRNIAARAAKfNLAAl
...
Session 10: obj - rowid = 000514F2 - BABRTyAAJAAKWbIAAY
....
----- Information for the OTHER waiting sessions -----
....
current SQL:
update employee set name=:1
----- End of information for the OTHER waiting sessions -----
Information for THIS session:
----- Current SQL Statement for this session (sql_id=5dfr2prw60rh1) -----
update department set address =:1 where id=:1
===================================================
Your output says the current session is trying to update a locked record in the department table (the "information for THIS session" output). The other session is trying to update every employee record (the "information for the OTHER waiting sessions" output). The current session must have updated a record in the employee table, blocking the other session, while the other session updated the record the current session is trying to update.
I assume this is some sort of exercise to cause a deadlock, since you're setting every employee record to the same name.

Deadlock When Updating different PK rows

I have a problem with my SPring Integration application.
In one point of the code, the apllication throws this error:
ORA-00060 - DEADLOCK DETECTED
I see this in the trace:
Deadlock graph:
---------Blocker(s)-------- ---------Waiter(s)---------
Resource Name process session holds waits process session holds waits
TM-00024912-00000000 58 395 SX SSX 61 441 SX SSX
TM-00024912-00000000 61 441 SX SSX 58 395 SX SSX
session 395: DID 0001-003A-000458EF session 441: DID 0001-003D-0002C56C
session 441: DID 0001-003D-0002C56C session 395: DID 0001-003A-000458EF
Rows waited on:
Session 395: no row
Session 441: no row
I am sure that the query update two differents rows
How is it possible that a deadlock occurs in different PK rows?
Problem solved. Thanks all the people for the answers.
The problem was that tablas are partitioned by foreign key reference , and the FK are not indexed , causing deadlocks by Oracle that blocks all the child table.
I have created all the indexes for the foreign keys and the problem is solved.

Magento 1.7 / 1.8 deadlocks from index_process table

I'm having greate problems with Magento last friday we upgraded Magento from 1.7 to 1.8..
The issue is that we're having a lot of deadlocks in the MySQL database.
Our server setup is
1 Load Balancer
4 Webservers (Apache, PHP5, APC)
2 MySQL Servers (64 GB Ram, 30 cores SSD HDD) - 1 Master (Has Memcache for sessions) - 1 Slave (Has Redis for caching)
The deadlock's is less on Magento 1.8 than 1.7 but the still appear from time to time ..
Any one has some good ideas on how to get pass this problem.
Heres some data from SHOW ENGINE INNODB STATUS;
LATEST DETECTED DEADLOCK
130930 12:03:35
* (1) TRANSACTION:
TRANSACTION 918EEC3B, ACTIVE 37 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 41 lock struct(s), heap size 6960, 50 row lock(s), undo log entries 6
MySQL thread id 51899, OS thread handle 0x7f9774169700, query id 2583719 xxx.xx.xxx.47 dbxxx Updating
UPDATE m17_index_process SET started_at = '2013-09-30 10:03:36' WHERE (process_id='8')
* (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 594 page no 3 n bits 208 index PRIMARY of table xxx.xx.xxx.47 dbxxx.m17_index_process trx id 918EEC3B lock_mode X locks rec but not gap waiting
* (2) TRANSACTION:
TRANSACTION 918EE3E7, ACTIVE 72 sec starting index read
mysql tables in use 1, locked 1
680 lock struct(s), heap size 80312, 150043 row lock(s), undo log entries 294
MySQL thread id 51642, OS thread handle 0x7f8a336c7700, query id 2586254 xxx.xx.xxx.47 dbxxx Updating
UPDATE m17_index_process SET started_at = '2013-09-30 10:03:40' WHERE (process_id='8')
(2) HOLDS THE LOCK(S):
RECORD LOCKS space id 594 page no 3 n bits 208 index PRIMARY of table dbxxx.m17_index_process trx id 918EE3E7 lock mode S locks rec but not gap
(2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 594 page no 3 n bits 208 index PRIMARY of table dbxxx.m17_index_process trx id 918EE3E7 lock_mode X locks rec but not gap waiting
* WE ROLL BACK TRANSACTION (1)
Best Regards.
Rasmus
Seems deadlocks are due to indexing processes. Try disabling automatic indexes Magento - Programmatically Disable Automatic Indexing
and doing them manually.
Also try disabling cron for some time and check if issues reoccur.
Its possible that many store admins saving products from different stores. In that case product save may be causing deadlock with index processes.
Thanks

Is a deadlock possible when updating and deleting different rows in a table?

In Oracle 10+ versions, can update and delete on the same table cause deadlocks even if they are operating on different rows of same table concurrently?
The table has primary key made-up of two columns, and do not have any FK associated/refereed with any other table. And there is no parent/child relation with other table
What I believe is, it will not create a deadlock, but I'm facing a issue in my application.
adding the oracle trace :
The following deadlock is not an ORACLE error. It is a deadlock due to user error in the design of an application or from issuing incorrect ad-hoc SQL. The following information may aid in determining the deadlock:
Deadlock graph:
---------Blocker(s)-------- ---------Waiter(s)---------
Resource Name process session holds waits process session holds waits
TX-0007003e-0081d6c3 45 790 X 104 20 X
TX-00080043-0085e6be 104 20 X 45 790 X
session 790: DID 0001-002D-000035F9 session 20: DID 0001-0068-000007F6
session 20: DID 0001-0068-000007F6 session 790: DID 0001-002D-000035F9
Rows waited on:
Session 790: obj - rowid = 0000F0C8 - AAAPDIAAMAAAEfIAAA
(dictionary objn - 61640, file - 12, block - 18376, slot - 0)
Session 20: obj - rowid = 0000F0C8 - AAAPDIAAMAAAEfGAAA
(dictionary objn - 61640, file - 12, block - 18374, slot - 0)
----- Information for the OTHER waiting sessions ----- Session 20:
sid: 20 ser: 4225 audsid: 57496371 user: 72/RPT_TABLE
flags: (0x45) USR/- flags_idl: (0x1) BSY/-/-/-/-/-
flags2: (0x40009) -/-/INC
pid: 104 O/S info: user: oracle, term: UNKNOWN, ospid: 20798
image: oracle#caidb10p-node1
client details:
O/S info: user: gtsgen, term: unknown, ospid: 1234
machine: caiapp08p-node0.nam.nsroot.net program: JDBC Thin Client
application name: JDBC Thin Client, hash value=2546894660
current SQL:
delete from RPT_TABLE.TEMP_TABLE_T1 where TEMP_T1_ID=:1
----- End of information for the OTHER waiting sessions -----
Information for THIS session:
----- Current SQL Statement for this session (sql_id=bsaxpc2bdps9q) ----- UPDATE RPT_TABLE.TEMP_TABLE_T1 temp1 SET temp1.CLIENT_ID = (SELECT MIN(INVMAP.CLIENT_ID) FROM LI_REF.REF_CLIENT_MAP INVMAP WHERE INVMAP.F_CODE = :B2 AND INVMAP.AID = temp1.ID AND temp1.R_ID=:B1 )
----- PL/SQL Stack -----
----- PL/SQL Call Stack -----
object line object
handle number name
45887d750 24 procedure RPT_TABLE.T1_UPDATE_StoredProc
6399ba188 1 anonymous block
If you could update your question with the deadlock graph, that would be useful information. (When your application encounters a deadlock, Oracle will raise an ORA-00060, and a tracefile will be written to the user_dump_dest.) If you look in the trace file, you'll find a section called the "Deadlock Graph". If you can post that, and also post the statement that caused the deadlock and other statements involved in the deadlock, then we can begin to draw some conclusions. (All the information I requested is available in the trace file.)
As Alessandro mentioned, it's possible for sessions locking different rows in the same table to deadlock due to unindexed foreign keys on the child table of a parent/child relationship. Also, It's possible that you could have deadlocks on two sessions updating different rows of the same table, even if the table is not part of a parent/child relationship, if, for example, the table has a shortage of ITL entries.
Again, post the information requested above, and I'm confident we can determine the root cause of your deadlock.
Added on 7/30/2012 **
Adding the following, now that the deadlock trace file has been supplied:
Ok, first off, based on the trace file contents, this is a simple deadlock due to sessions overlapping/colliding on the rows they are trying to lock. Despite your previous comments about the deadlock being on different rows, I'm here to tell you that this particular deadlock is due to row-level locking on the same rows.
The fact that the deadlock graph shows mode the lock is held in is 'X' (exclusive) and the mode the lock is waited on is 'X', tells me this is simple row-level locking.
In this case, SID 20 is executing "delete from RPT_TABLE.TEMP_TABLE_T1 where TEMP_T1_ID=:1" and already has a lock on rowid AAAPDIAAMAAAEfIAAA.
Meanwhile, SID 790 is executing "RPT_TABLE.T1_UPDATE_StoredProc", while already holding a lock on rowid AAAPDIAAMAAAEfGAAA.
Note from the "Rows waited on" section of the tracefile, that SID 20 is waiting on the row that SID 790 holds and SID 790 is waiting on the row that SID 20 is holding. This is a classic deadlock.
Some additional information:
Enqueue type is TX (see the deadlock graph), so, this is definitely not locking due to unindexed foreign keys. If it were locking due to unindexed FKs, the enqueue type would be TM, not TX. (There is at least one other case where TM enqueues are involved, and it's not unindexed FKs. So, don't assume that TM enqueue always means unindexed FKs.)
The mode the lock is being waited on is 'X' (exclusive), so this is row-level locking. If the mode waited on was 'S' (shared), then it would not be row-level locking. Rather, it could be ITL shortage or PK or UK enforcement.
Hope that helps!
I don't know if you have foreign keys involved in your application but it could probably be the source of your locks. If so take a look at these links:
http://docs.oracle.com/cd/E11882_01/server.112/e16508/consist.htm#BABCAHDJ
http://docs.oracle.com/cd/E11882_01/server.112/e16508/datainte.htm#CNCPT1657
Oracle Database maximizes the concurrency control of parent keys in relation to dependent foreign keys. Locking behaviour depends on whether foreign key columns are indexed. If foreign keys are not indexed, then the child table will probably be locked more frequently, deadlocks will occur, and concurrency will be decreased. For this reason foreign keys should almost always be indexed. The only exception is when the matching unique or primary key is never updated or deleted.
Locks and Unindexed Foreign Keys
When both of the following conditions are true, the database acquires a full table lock on the child table:
No index exists on the foreign key column of the child table.
A session modifies a primary key in the parent table (for example, deletes a row or modifies primary key attributes) or merges rows into the parent table. Inserts into the parent table do not acquire table locks on the child table.
If this is not your case try to provide more informations about it. Tell us about the Kind of locks Holden/requested by the sessions and take a look at the system tables V$LOCK, V$LOCKED_OBJECT, DBA_DDL_LOCKS, DBA_DML_LOCKS or V$SESSION_WAIT.

Identifying and Resolving Oracle ITL Deadlock

I have an Oracle DB package that is routinely causing what I believe is an ITL (Interested Transaction List) deadlock. The relevant portion of a trace file is below.
Deadlock graph:
---------Blocker(s)-------- ---------Waiter(s)---------
Resource Name process session holds waits process session holds waits
TM-0000cb52-00000000 22 131 S 23 143 SS
TM-0000ceec-00000000 23 143 SX 32 138 SX SSX
TM-0000cb52-00000000 30 138 SX 22 131 S
session 131: DID 0001-0016-00000D1C session 143: DID 0001-0017-000055D5
session 143: DID 0001-0017-000055D5 session 138: DID 0001-001E-000067A0
session 138: DID 0001-001E-000067A0 session 131: DID 0001-0016-00000D1C
Rows waited on:
Session 143: no row
Session 138: no row
Session 131: no row
There are no bit-map indexes on this table, so that's not the cause. As far as I can tell, the lack of "Rows waited on" plus the "S" in the Waiter waits column likely indicates that this is an ITL deadlock. Also, the table is written to quite often (roughly 8 insert or updates concurrently, as often as 240 times a minute), so an ITL deadlock seems like a strong possibility.
I've increased the INITRANS parameter of the table and it's indexes to 100 and increased the PCT_FREE on the table from 10 to 20 (then rebuilt the indexes), but the deadlocks are still occurring. The deadlock seems to happen most often during an update, but that could just be a coincidence, as I've only traced it a couple of times.
My questions are two-fold:
1) Is this actually an ITL deadlock?
2) If it is an ITL deadlock, what else can be done to avoid it?
It turns out that this was not an ITL deadlock issue at all, but rather an issue with un-indexed foreign keys. I discovered this thanks to dpbradley's answer, which clued me into the fact that it wasn't an ITL issue and prompted me to find out what the other causes of a deadlock with "no rows" might be.
The best indication of ITL pressure is from the performance views:
select event, total_waits, time_waited, average_wait
from v$system_event
where event like 'enq: TX%'
order by 2 desc;
shows TX contention waits, and
select OBJECT_NAME, SUBOBJECT_NAME, TABLESPACE_NAME,
OBJECT_TYPE, STATISTIC_NAME, VALUE
from v$segment_statistics
where statistic_name = 'ITL waits'
and value > 0
order by value desc;
shows the tables and indexes involved.
(Like all v$ views, the results are from the point in time when the instance was started.)
If this shows that you do indeed have ITL waits, then the INITRANS and PCTFREE parameters are the main knobs to turn (but INITRANS = 100 sounds pretty high to me and these do cost space).
If ITL waits are not a problem, then the application code needs to be examined.
The best option is to increase it as needed (start from default 10 and increment by 10). If you see reduction in ITL waits, you're set. Usually these related parameters are adjusted by trial and error both in Oracle and SQL Server. Adjusting these parameters in real-time won't be that much of an issue, unless the resource is extremely busy. You can use the following query to see after each increment, if the ITL waits either go away or is highly reduced:
SELECT t.OWNER, t.OBJECT_NAME, t.OBJECT_TYPE, t.STATISTIC_NAME, t.VALUE
FROM v$segment_statistics t
WHERE t.STATISTIC_NAME = 'ITL waits' AND t.VALUE > 0
ORDER BY t.value desc;
We have carried out several tunings for Oracle deadlock scenarios due to ITL waits using this method. NOTE: Make sure the index is rebuilt, if the initrans is modified for indexes. Also ensure that statistics are not stale.
For a quick check SQL Tuning Advisor can be utilized to see the full state of the query/index and statistics.

Resources