I have a query running in Oracle, which may or may not be hung. It's been running for ~10 hours now, but based on the amount of data I'm loading that may not be unreasonable.
I was looking at the session in gv$session and was wondering if there's a way to translate that information to see if there's actually any activity going on, or if the query is stuck waiting for a lock or otherwise hung.
I've already read the documentation for this view here. I'm mostly looking for tips from anyone whose had experience debugging these types of issues in Oracle.
Thanks!
In gv$session, the event column tells you what wait event your session is currently waiting on. If your session is waiting on some sort of lock held by another session, the event will tell you that (for example, it will be "enq: TX - row lock contention" if you are enqueued waiting to lock a row held by another session) and blocking_instance and blocking_session will be populated with the instance and session ID of the holder of the lock. You can also look at seconds_in_wait (if wait_time=0) to determine how many seconds the session has spent in the current wait event. That should at least tell you whether your session is currently "stuck" but it doesn't tell you if your query is ever really going to finish-- if there is a bad plan, it's entirely possible that you've got "good" wait events like waits for disk I/O that indicate the session is doing something but that the query is never really going to finish.
Based on some further research and Ollie's comment I came up with these queries that help debug the issue:
select s.sid,
s.username,
s.machine,
s.osuser,
cpu_time,
(elapsed_time/1000000)/60 as minutes,
sql_text
from gv$sqlarea a, gv$session s
where s.sql_id = a.sql_id
and s.machine like '####';
select lo.*,
a.sql_text
from gv$sqlarea a, gv$session_longops lo
where lo.sql_id = a.sql_id
and lo.sid = ####
order by lo.start_time;
This will be Helpful to Check the Current Running Session
select a.SID, a.SERIAL#, c.OBJECT_NAME
from v$session a, v$locked_object b, user_objects c
where a.SID=b.SESSION_ID and b.OBJECT_ID=c.OBJECT_ID
Related
Why am I getting this database error when I update a table?
ERROR at line 1:
ORA-00054: resource busy and acquire with NOWAIT specified or timeout expired
Your table is already locked by some query. For example, you may have executed "select for update" and have not yet committed/rollbacked and fired another select query. Do a commit/rollback before executing your query.
from here ORA-00054: resource busy and acquire with NOWAIT specified
You can also look up the sql,username,machine,port information and get to the actual process which holds the connection
SELECT O.OBJECT_NAME, S.SID, S.SERIAL#, P.SPID, S.PROGRAM,S.USERNAME,
S.MACHINE,S.PORT , S.LOGON_TIME,SQ.SQL_FULLTEXT
FROM V$LOCKED_OBJECT L, DBA_OBJECTS O, V$SESSION S,
V$PROCESS P, V$SQL SQ
WHERE L.OBJECT_ID = O.OBJECT_ID
AND L.SESSION_ID = S.SID AND S.PADDR = P.ADDR
AND S.SQL_ADDRESS = SQ.ADDRESS;
Please Kill Oracle Session
Use below query to check active session info
SELECT
O.OBJECT_NAME,
S.SID,
S.SERIAL#,
P.SPID,
S.PROGRAM,
SQ.SQL_FULLTEXT,
S.LOGON_TIME
FROM
V$LOCKED_OBJECT L,
DBA_OBJECTS O,
V$SESSION S,
V$PROCESS P,
V$SQL SQ
WHERE
L.OBJECT_ID = O.OBJECT_ID
AND L.SESSION_ID = S.SID
AND S.PADDR = P.ADDR
AND S.SQL_ADDRESS = SQ.ADDRESS;
kill like
alter system kill session 'SID,SERIAL#';
(For example, alter system kill session '13,36543';)
Reference
http://abeytom.blogspot.com/2012/08/finding-and-fixing-ora-00054-resource.html
There is a very easy work around for this problem.
If you run a 10046 trace on your session (google this... too much to explain). You will see that before any DDL operation Oracle does the following:
LOCK TABLE 'TABLE_NAME' NO WAIT
So if another session has an open transaction you get an error. So the fix is... drum roll please. Issue your own lock before the DDL and leave out the 'NO WAIT'.
Special Note:
if you are doing splitting/dropping partitions oracle just locks the partition.
-- so yo can just lock the partition subpartition.
So...
The following steps fix the problem.
LOCK TABLE 'TABLE NAME'; -- you will 'wait' (developers call this hanging). until the session with the open transaction, commits. This is a queue. so there may be several sessions ahead of you. but you will NOT error out.
Execute DDL. Your DDL will then run a lock with the NO WAIT. However, your session has aquired the lock. So you are good.
DDL auto-commits. This frees the locks.
DML statements will 'wait' or as developers call it 'hang' while the table is locked.
I use this in code that runs from a job to drop partitions. It works fine. It is in a database that is constantly inserting at a rate of several hundred inserts/second. No errors.
if you are wondering. Doing this in 11g. I have done this in 10g before as well in the past.
This error happens when the resource is busy. Check if you have any referential constraints in the query. Or even the tables that you have mentioned in the query may be busy. They might be engaged with some other job which will be definitely listed in the following query results:
SELECT * FROM V$SESSION WHERE STATUS = 'ACTIVE'
Find the SID,
SELECT * FROM V$OPEN_CURSOR WHERE SID = --the id
In my case, I was quite sure it was one of my own sessions which was blocking. Therefore, it was safe to do the following:
I found the offending session with:
SELECT * FROM V$SESSION WHERE OSUSER='my_local_username';
The session was inactive, but it still held the lock somehow. Note, that you may need to use some other WHERE condition in your case (e.g. try USERNAME or MACHINE fields).
Killed the session using the ID and SERIAL# acquired above:
alter system kill session '<id>, <serial#>';
Edited by #thermz: If none of the previous open-session queries work try this one. This query can help you to avoid syntax errors while killing sessions:
SELECT 'ALTER SYSTEM KILL SESSION '''||SID||','||SERIAL#||''' immediate;' FROM V$SESSION WHERE OSUSER='my_local_username_on_OS'
This happens when a session other than the one used to alter a table is holding a lock likely because of a DML (update/delete/insert). If you are developing a new system, it is likely that you or someone in your team issues the update statement and you could kill the session without much consequence. Or you could commit from that session once you know who has the session open.
If you have access to a SQL admin system use it to find the offending session. And perhaps kill it.
You could use v$session and v$lock and others but I suggest you google how to find that session and then how to kill it.
In a production system, it really depends. For oracle 10g and older, you could execute
LOCK TABLE mytable in exclusive mode;
alter table mytable modify mycolumn varchar2(5);
In a separate session but have the following ready in case it takes too long.
alter system kill session '....
It depends on what system do you have, older systems are more likely to not commit every single time. That is a problem since there may be long standing locks. So your lock would prevent any new locks and wait for a lock that who knows when will be released. That is why you have the other statement ready. Or you could look for PLSQL scripts out there that do similar things automatically.
In version 11g there is a new environment variable that sets a wait time. I think it likely does something similar to what I described. Mind you that locking issues don't go away.
ALTER SYSTEM SET ddl_lock_timeout=20;
alter table mytable modify mycolumn varchar2(5);
Finally it may be best to wait until there are few users in the system to do this kind of maintenance.
select
c.owner,
c.object_name,
c.object_type,
b.sid,
b.serial#,
b.status,
b.osuser,
b.machine
from
v$locked_object a,
v$session b,
dba_objects c
where
b.sid = a.session_id
and
a.object_id = c.object_id;
ALTER SYSTEM KILL SESSION 'sid,serial#';
As mentioned in other answers, this error is caused by concurrent DML operations running in other sessions. This causes Oracle to fail to lock the table for DDL with the default NOWAIT option.
For those without admin permissions in the database or who cannot kill/interrupt the other sessions, you can also precede your DDL operation with:
alter session set DDL_LOCK_TIMEOUT = 30;
--Run your DDL command, e.g.: alter table, etc.
I was receiving this error repeatedly in a database with background jobs doing large insert/update operations, and altering this parameter in the session allowed the DDL to continue after a few seconds of waiting for the lock.
For further information, see the comment from rshdev on this answer, this entry on oracle-base or the official docs on DDL_LOCK_TIMEOUT.
Just check for process holding the session and Kill it. Its back to normal.
Below SQL will find your process
SELECT s.inst_id,
s.sid,
s.serial#,
p.spid,
s.username,
s.program FROM gv$session s
JOIN gv$process p ON p.addr = s.paddr AND p.inst_id = s.inst_id;
Then kill it
ALTER SYSTEM KILL SESSION 'sid,serial#'
OR
some example I found online seems to need the instance id as well
alter system kill session '130,620,#1';
I had this error happen when I had 2 scripts I was running. I had:
A SQL*Plus session connected directly using a schema user account (account #1)
Another SQL*Plus session connected using a different schema user account (account #2), but connecting across a database link as the first account
I ran a table drop, then table creation as account #1.
I ran a table update on account #2's session. Did not commit changes.
Re-ran table drop/creation script as account #1. Got error on the drop table x command.
I solved it by running COMMIT; in the SQL*Plus session of account #2.
Your problem looks like you are mixing DML & DDL operations. See this URL which explains this issue:
http://www.orafaq.com/forum/t/54714/2/
I managed to hit this error when simply creating a table! There was obviously no contention problem on a table that didn't yet exist. The CREATE TABLE statement contained a CONSTRAINT fk_name FOREIGN KEY clause referencing a well-populated table. I had to:
Remove the FOREIGN KEY clause from the CREATE TABLE statement
Create an INDEX on the FK column
Create the FK
I solved this problem by closing one of my IDE tabs.
PL/SQL Developer
Version 10.0.5.1710
I also face the similar Issue. Nothing programmer has to do to resolve this error. I informed to my oracle DBA team. They kill the session and worked like a charm.
Solution given by Shashi's link is the best... no needs to contact dba or someone else
make a backup
create table xxxx_backup as select * from xxxx;
delete all rows
delete from xxxx;
commit;
insert your backup.
insert into xxxx (select * from xxxx_backup);
commit;
What am I missing here? I am trying to test identifying long running queries.
I have a test table with about 400 million rows called mytest.
I ran select * from mytest in sqlplus
In another window, I ran the script below to see my long running query
select s.username, s.sid, s.serial#, s.schemaname,
s.program, s.osuser, s.status, s.last_call_et
from v$session s
where last_call_et >= 1 – this is just for testing
My long running query does not show up in the result from the query above. If I change the criteria to be >=0, then I see my query showing the status as INACTIVE and last_call_et of 0 despite the fact that the query is still running. What can I do to see my long running queries like the select * from... above so that I can kill it?
Thanks
First, you need to understand what a query like select * from mytest is really doing under the covers because that's generally not going to be a long-running query. Oracle doesn't ever need to materialize that result set and isn't going to read all the data as the result of a single call. Instead, what goes on is a series of calls each of which cause Oracle to do a little bit of work. The conversation goes something like this.
Client: Hey Oracle, run the query for me: select * from mytest
Oracle: Sure thing (last_call_et resets to 0 to reflect that a new call started). I've generated a query plan and opened a cursor,
here's a handle (note that no work has been done yet to actually
execute the query)
Client: Cool, thanks. Using this cursor handle,
fetch me the next 50 rows (the fetch size is a client-side setting)
Oracle: Will do (last_call_et resets to 0 to reflect that a new call started). I started full scanning the table, read a couple of
blocks, and got 50 rows. Here you go.
Client: OK, I've processed
those. Using this cursor handle, fetch the next 50 rows
Repeat until all the data is fetched
At no point in this process is Oracle ever really being asked to do more than read a handful of blocks to get the 50 rows (or whatever the fetch size the client is requesting). At any point, the client could simply not request the next batch of data so Oracle doesn't need to do anything long-running. Oracle doesn't track the application think time between requests for more data-- it has no idea whether the client is a GUI that is in a tight loop fetching data or whether it is displaying a result to a human and waiting for a human to hit the "next" button. The vast majority of the time, the session is going to be INACTIVE because it's mostly waiting for the client to request the next batch of data (which it generally won't do until it had formatted the last batch of data for display and done the work to display it).
When most people talk about a long-running query, they're talking about a query that Oracle is actively processing for a relatively long time with no waits on a client to fetch the data.
You can use the below script to find long running query:
select * from
(
select
opname,
start_time,
target,
sofar,
totalwork,
units,
elapsed_seconds,
message
from
v$session_longops
order by start_time desc
)
where rownum <=1;
is there a way to know that of the say 100 DDL and DML statement which i am executing through jdbc , is something stuck.
i need to find out for progress bar that in db sql statements are executing and are not hung, so that can inform to user.
Is there a way to find this out.
There are any number of Oracle data dictionary views that you might use given the relatively vague requirements (what, exactly, is your definition of "hung", for example).
I'd probably start, though, with v$session assuming you can identify the sid and serial# of the particular session that you want to monitor (which may be aided by looking at the various columns in v$session). STATUS will tell you whether the session is actively executing a SQL statement at that particular instant. The sql_id will (generally) let you join to v$sqlarea or other views that tell you what statement is currently executing. The event column will tell you what the session is waiting on (i.e. reading from disk, waiting on CPU, waiting to acquire a lock, etc.).
The sql_id from v$session will also let you join to v$sqlstats which periodically updates with things like the amount of logical I/O a particular SQL statement has generated which would let you see that the currently active statement is doing something (whether that something is useful or whether it will terminate in our lifetime would be much more difficult).
Depending on what the code is doing, there may be one or more rows in v$session_longops that you can use to track the progress of longer-running operations-- using this effectively, though, will require that the third-party code issues the sort of long-running SQL operations that Oracle can monitor automatically (i.e. table scans of tables that have a reasonable amount of data) or that the code is instrumented to use v$session_longops to track its own progress.
Depending on what version of Oracle you're using, you might also be able to use the v$sql_monitor view to monitor SQL in real time.
Why am I getting this database error when I update a table?
ERROR at line 1:
ORA-00054: resource busy and acquire with NOWAIT specified or timeout expired
Your table is already locked by some query. For example, you may have executed "select for update" and have not yet committed/rollbacked and fired another select query. Do a commit/rollback before executing your query.
from here ORA-00054: resource busy and acquire with NOWAIT specified
You can also look up the sql,username,machine,port information and get to the actual process which holds the connection
SELECT O.OBJECT_NAME, S.SID, S.SERIAL#, P.SPID, S.PROGRAM,S.USERNAME,
S.MACHINE,S.PORT , S.LOGON_TIME,SQ.SQL_FULLTEXT
FROM V$LOCKED_OBJECT L, DBA_OBJECTS O, V$SESSION S,
V$PROCESS P, V$SQL SQ
WHERE L.OBJECT_ID = O.OBJECT_ID
AND L.SESSION_ID = S.SID AND S.PADDR = P.ADDR
AND S.SQL_ADDRESS = SQ.ADDRESS;
Please Kill Oracle Session
Use below query to check active session info
SELECT
O.OBJECT_NAME,
S.SID,
S.SERIAL#,
P.SPID,
S.PROGRAM,
SQ.SQL_FULLTEXT,
S.LOGON_TIME
FROM
V$LOCKED_OBJECT L,
DBA_OBJECTS O,
V$SESSION S,
V$PROCESS P,
V$SQL SQ
WHERE
L.OBJECT_ID = O.OBJECT_ID
AND L.SESSION_ID = S.SID
AND S.PADDR = P.ADDR
AND S.SQL_ADDRESS = SQ.ADDRESS;
kill like
alter system kill session 'SID,SERIAL#';
(For example, alter system kill session '13,36543';)
Reference
http://abeytom.blogspot.com/2012/08/finding-and-fixing-ora-00054-resource.html
There is a very easy work around for this problem.
If you run a 10046 trace on your session (google this... too much to explain). You will see that before any DDL operation Oracle does the following:
LOCK TABLE 'TABLE_NAME' NO WAIT
So if another session has an open transaction you get an error. So the fix is... drum roll please. Issue your own lock before the DDL and leave out the 'NO WAIT'.
Special Note:
if you are doing splitting/dropping partitions oracle just locks the partition.
-- so yo can just lock the partition subpartition.
So...
The following steps fix the problem.
LOCK TABLE 'TABLE NAME'; -- you will 'wait' (developers call this hanging). until the session with the open transaction, commits. This is a queue. so there may be several sessions ahead of you. but you will NOT error out.
Execute DDL. Your DDL will then run a lock with the NO WAIT. However, your session has aquired the lock. So you are good.
DDL auto-commits. This frees the locks.
DML statements will 'wait' or as developers call it 'hang' while the table is locked.
I use this in code that runs from a job to drop partitions. It works fine. It is in a database that is constantly inserting at a rate of several hundred inserts/second. No errors.
if you are wondering. Doing this in 11g. I have done this in 10g before as well in the past.
This error happens when the resource is busy. Check if you have any referential constraints in the query. Or even the tables that you have mentioned in the query may be busy. They might be engaged with some other job which will be definitely listed in the following query results:
SELECT * FROM V$SESSION WHERE STATUS = 'ACTIVE'
Find the SID,
SELECT * FROM V$OPEN_CURSOR WHERE SID = --the id
In my case, I was quite sure it was one of my own sessions which was blocking. Therefore, it was safe to do the following:
I found the offending session with:
SELECT * FROM V$SESSION WHERE OSUSER='my_local_username';
The session was inactive, but it still held the lock somehow. Note, that you may need to use some other WHERE condition in your case (e.g. try USERNAME or MACHINE fields).
Killed the session using the ID and SERIAL# acquired above:
alter system kill session '<id>, <serial#>';
Edited by #thermz: If none of the previous open-session queries work try this one. This query can help you to avoid syntax errors while killing sessions:
SELECT 'ALTER SYSTEM KILL SESSION '''||SID||','||SERIAL#||''' immediate;' FROM V$SESSION WHERE OSUSER='my_local_username_on_OS'
This happens when a session other than the one used to alter a table is holding a lock likely because of a DML (update/delete/insert). If you are developing a new system, it is likely that you or someone in your team issues the update statement and you could kill the session without much consequence. Or you could commit from that session once you know who has the session open.
If you have access to a SQL admin system use it to find the offending session. And perhaps kill it.
You could use v$session and v$lock and others but I suggest you google how to find that session and then how to kill it.
In a production system, it really depends. For oracle 10g and older, you could execute
LOCK TABLE mytable in exclusive mode;
alter table mytable modify mycolumn varchar2(5);
In a separate session but have the following ready in case it takes too long.
alter system kill session '....
It depends on what system do you have, older systems are more likely to not commit every single time. That is a problem since there may be long standing locks. So your lock would prevent any new locks and wait for a lock that who knows when will be released. That is why you have the other statement ready. Or you could look for PLSQL scripts out there that do similar things automatically.
In version 11g there is a new environment variable that sets a wait time. I think it likely does something similar to what I described. Mind you that locking issues don't go away.
ALTER SYSTEM SET ddl_lock_timeout=20;
alter table mytable modify mycolumn varchar2(5);
Finally it may be best to wait until there are few users in the system to do this kind of maintenance.
select
c.owner,
c.object_name,
c.object_type,
b.sid,
b.serial#,
b.status,
b.osuser,
b.machine
from
v$locked_object a,
v$session b,
dba_objects c
where
b.sid = a.session_id
and
a.object_id = c.object_id;
ALTER SYSTEM KILL SESSION 'sid,serial#';
As mentioned in other answers, this error is caused by concurrent DML operations running in other sessions. This causes Oracle to fail to lock the table for DDL with the default NOWAIT option.
For those without admin permissions in the database or who cannot kill/interrupt the other sessions, you can also precede your DDL operation with:
alter session set DDL_LOCK_TIMEOUT = 30;
--Run your DDL command, e.g.: alter table, etc.
I was receiving this error repeatedly in a database with background jobs doing large insert/update operations, and altering this parameter in the session allowed the DDL to continue after a few seconds of waiting for the lock.
For further information, see the comment from rshdev on this answer, this entry on oracle-base or the official docs on DDL_LOCK_TIMEOUT.
Just check for process holding the session and Kill it. Its back to normal.
Below SQL will find your process
SELECT s.inst_id,
s.sid,
s.serial#,
p.spid,
s.username,
s.program FROM gv$session s
JOIN gv$process p ON p.addr = s.paddr AND p.inst_id = s.inst_id;
Then kill it
ALTER SYSTEM KILL SESSION 'sid,serial#'
OR
some example I found online seems to need the instance id as well
alter system kill session '130,620,#1';
I had this error happen when I had 2 scripts I was running. I had:
A SQL*Plus session connected directly using a schema user account (account #1)
Another SQL*Plus session connected using a different schema user account (account #2), but connecting across a database link as the first account
I ran a table drop, then table creation as account #1.
I ran a table update on account #2's session. Did not commit changes.
Re-ran table drop/creation script as account #1. Got error on the drop table x command.
I solved it by running COMMIT; in the SQL*Plus session of account #2.
Your problem looks like you are mixing DML & DDL operations. See this URL which explains this issue:
http://www.orafaq.com/forum/t/54714/2/
I managed to hit this error when simply creating a table! There was obviously no contention problem on a table that didn't yet exist. The CREATE TABLE statement contained a CONSTRAINT fk_name FOREIGN KEY clause referencing a well-populated table. I had to:
Remove the FOREIGN KEY clause from the CREATE TABLE statement
Create an INDEX on the FK column
Create the FK
I solved this problem by closing one of my IDE tabs.
PL/SQL Developer
Version 10.0.5.1710
I also face the similar Issue. Nothing programmer has to do to resolve this error. I informed to my oracle DBA team. They kill the session and worked like a charm.
Solution given by Shashi's link is the best... no needs to contact dba or someone else
make a backup
create table xxxx_backup as select * from xxxx;
delete all rows
delete from xxxx;
commit;
insert your backup.
insert into xxxx (select * from xxxx_backup);
commit;
Right now I have a few new applications being developed against an Oracle Database, and sometimes they crash or fail to end correctly, etc... anyways the problem is they sometimes seem to leave their connections open, and I need to cleanup after them.
My question is if there is a way from the database-side of things to determine dead connections and clean them up?
Here's a page referring to connection timeout parameters you can set in Oracle 11g. I think the 'Abandon Connection Timeout' is what you're looking for.
You may also be interested in killing them. Running this script in SQL*Plus will give you a list of "kill" statements. You can pick out the ones you want to kill based on the sid and run those. Oracle has some of it's own internal connections, do not kill them.
SELECT 'alter system kill session ''' || sid || ',' || serial# || '''; ' || sql_id death
FROM v$session
/
I believe you are looking for the SQLNet.ora parameter EXPIRE_TIME which tells the database to send a probe to the client every few minutes to verify that the connection is still alive.
Here's how to identify the session to kill (you will need SID and SERIAL# to kill it). Should I mention that you need to make sure you're killing the right session? sys_context('userenv','sid') gets the SID of your own session.
SELECT s.inst_id,
s.sid,
s.serial#,
p.spid,
s.username,
s.osuser,
s.program
FROM gv$session s
JOIN gv$process p ON p.addr = s.paddr AND p.inst_id = s.inst_id
WHERE s.type != 'BACKGROUND';
You can then issue alter system kill session '[sid],[serial#]' as suggested by WW.
However the alter system kill session command does not forcibly kill the session, rather it asks the session to die. If the session is really hung, you will find that the request hangs for 60 seconds and then returns ORA-00031 Session marked for kill. And the session is still there.
In that case, first check that the session isn't rolling back a large transaction (cross reference the SID and SERIAL# from the above):
SELECT s.username,
s.osuser,
s.sid,
s.serial#,
t.used_ublk,
t.used_urec,
rs.segment_name,
r.rssize,
r.status
FROM v$transaction t,
v$session s,
v$rollstat r,
dba_rollback_segs rs
WHERE s.saddr = t.ses_addr
AND t.xidusn = r.usn
AND rs.segment_id = t.xidusn
ORDER BY t.used_ublk DESC;
If a transaction is rolling back you will see USED_UREC decreasing. Leave it to complete rollback.
Otherwise, ALTER SYSTEM DISCONNECT SESSION '[sid],[serial#]' IMMEDIATE; will forcibly disconnect the session and roll back the open transaction.
All the above info came from here.