Have anyone faced a similar issue below:
Session with more number of targets.
TargetDB: Oracle
Target Schema: abc
Number of targets: 6
As per session logs, Writer thread is always busy 100%.
Writer volume is 5 million and takes 3.5 hours to complete.
Regards,
Sridar Venkatesan
Related
My 10g oracle prod database have performance problem. Some queries begun to return in 20 seconds which was comes in milliseconds. I get AWR report and top3 wait event shown below. I searched but i couldnt understand as well.
Can someone explain this events ? Thanks,
Event Waits Time(s) Avg Wait(ms) % Total Call Time Wait Class
---------------------- ---------- ------- ------------ ----------------- ----------
direct path write temp 11,941,557 866,004 73 29.8 User I/O FEBRUARY
direct path write temp 16,197,445 957,129 59 17.2 User I/O MARCH
db file scattered read 5,826,190 58,095 10 2.0 User I/O FEBRUARY
db file scattered read 10,128,657 70,408 7 1.3 User I/O MARCH
direct path read temp 34,197,762 324,663 9 11.2 User I/O FEBRUARY
direct path read temp 88,688,686 507,715 6 9.1 User I/O MARCH
Two of your wait events are related to sorting: direct path write temp and direct path read temp. These indicate an increase in sorting on disk rather than in memory; disk I/O is always slower.
So, what has changed regarding memory allocation usage? Perhaps you need to revisit the values of SORT_AREA_SIZE or PGA_AGGREGATE_TARGET init parameters (depending on whether you are using Automatic PGA memory). Here is a query which calculates the memory/disk sort ratio:
SELECT 100 * (mem.value - dsk.value)/(mem.value) AS sort_ratio
FROM v$sysstat mem
cross join v$sysstat dsk
WHERE mem.name = 'sorts (memory)'
AND dsk.name ='sorts (disk)'
In an OLTP application we would expect this to be over 95%.
The other thing is, instead of looking at macro events you need to look at the specific queries which are running much slower. What has changed with regards to them? Lots more data? New indexes or dropped indexes? Refreshed statistics?
"SORT_RATIO ---------- 99.9985462"
So, sorts are higher but not too high. You need to focus on specific queries.
"in march we begun to user phyton application for some new queries. reason can be this ? "
Could be. Application change is always the prime suspect when our system exhibits different behavior.
I have a same session running in production and UAT.All it does is seslects the data ( around 6k in both environments).Expression transformation (to hard code few columns) and then inserting into a table ( which does not have partitions).
The problem I am facing is PROD session is taking more than 30 minutes where as UAT is done within 5 minutes.
I have backtracked the timining to many days and its following the same pattern.When compared the session properties between the two.There is no difference at all.
When checked the session log its the reading of rows which is taking time(same count and query in UAT also)Could you please let me know how to proceed with this:
PROD:
Severity Timestamp Node Thread Message Code Message
INFO 4/26/2016 11:07:18 AM node02_WPPWM02A0004 WRITER_1_*_1 WRT_8167 Start loading table [FACT_] at: Tue Apr 26 01:37:17 2016
INFO 4/26/2016 11:26:48 AM node02_WPPWM02A0004 READER_1_1_1 BLKR_16019 Read [6102] rows, read [0] error rows for source table [STG_] instance name [STG]
UAT:
Severity Timestamp Node Thread Message Code Message
INFO 4/26/2016 11:40:53 AM node02_WUPWM02A0004 WRITER_1_*_1 WRT_8167 Start loading table [FACT] at: Tue Apr 26 01:10:53 2016
INFO 4/26/2016 11:43:10 AM node02_WUPWM02A0004 READER_1_1_1 BLKR_16019 Read [6209] rows, read [0] error rows for source table [STG] instance name [STG]
Follow the below steps
1) Open the session log and search for 'Busy'
2) Find the Busy statistics which has a very high Busy Percentage
3) if it is with reader , just run the query in production and UA and try to check the retrieval time. If its high in production then there is a need to tune the query or create indexes or create partitions at table level and informatica level etc., (depend on your project limitations)
4) if it is writer try to increase few informatica options like 'Maximum memory allocated for auto memory attributes' and 'Maximum percentage of total memory allowed..." depending on your server configuration
5) Also try to use informatica partitions while loading into target (Provided the target is partitioned on a particular column)
6) Also some times there is a possibility that cache creation takes time due to huge tables being used as lookups( Refer busy percentage of lookup also). In that case also target waits for the rows to come to the writer thread as they are still transforming
we need to tune the lookup by overriding the default query with tuned version of query
Also search for the following keywords
"Timeout based Commit point" - generally occurs when a writer thread waits for long time
"No more lookup cache " - generally occurs whenever there is a huge data and index cache to be build and no space available on the disk as multiple jobs will be running in production utilizing the same cache folder
Thanks and Regards
Raj
Perhaps, you should check the Query's Explain plan in UAT and PROD. Working on the plan in PROD can help. Similar thing happen with me earlier. We checked the SQL plan and found that it is different in prod when compared to UAT. Had to work with the DBA's to change the plan.
This may have been asked numerous times but none of them helped me so far.
Here's some history:
QueryTimeOut: 120 secs
Database:DB2
App Server: JBoss
Framework: Struts 2
I've one query which fetches around a million records. Yes, we need to fetch it all at once for caching purpose, sadly can't change the design.
Now, we've 2 servers Primary and DR. In DR server, the query is getting executed within 30 secs, so no timeout issue there. But in Primary serverit is getting time out due to some unknown reason. Sometimes it is getting timed out in rs.next() and sometime in pstmt.executeQuery().
All DB indexes, connection pool etc are in place. The explain plan shows, there are no full table scan as well.
My Analysis:
Since query is not the issue here, there might be issue in Network delay?
How can I get the root cause behind this timeout. How can I make sure there are no connection leakage? (Since all connection are closed properly).
Any way to recover from the timeout and again execute the query with increased timeout value for e.g: pstmt.setQueryTimeOut(600)? <- Note that this has no effect whatsoever. Don't know why..!
Appreciate any inputs.
Thank You!
I have an Azure website running about 100K requests/hour and it connects to Azure SQL S2 database with about 8GB throughput/day. I've spent a lot of time optimizing the database indexes, queries, etc. Normally the Data IO, CPU and Log IO percentages are well behaved in the 20% range.
A recent portion of the data throughput is retained for supporting our customers. I have a nightly maintenance procedure that removes obsolete data to manage database size. This mostly works well with the exception of removing image blobs in a varbinary(max) field.
The nightly procedure has a loop that sets 10 records varbinary(max) field to null at a time, waits a couple seconds, then sets the next 10. Nightly total for this loop is about 2000.
This loop will run for about 45 - 60 minutes and then stop running with no return to my remote Sql Agent job and no error reported. A second and sometimes third running of the procedure is necessary to finish setting the desired blobs to null.
In an attempt to alleviate the load on the nightly procedure, I started running a job once every 30 seconds throughout the day - it sets one blob to null each time.
Normally this trickle job is fine and runs in 1 - 6 seconds. However, once or twice a day something goes wrong and I can find no explanation for it. The Data I/O percentage peaks at 100% and stays there for 30 - 60 minutes or longer. This causes the database responsiveness to suffer and the website performance goes with it. The trickle job also reports running for this extended period of time. If I stop the Sql Agent job, it can take a few minutes to stop but the Data I/O continues at 100% for the 30 - 60 minute period.
The web service requests and database demands are relatively steady throughout the business day - no volatile demands that would explain this. No database deadlocks or other errors are reported. It's as if the database hits some kind of backlog limit where its ability to keep up suddenly drops and then it can't catch up until something that is jammed finally clears. Then the performance will suddenly return to normal.
Do you have any ideas what might be causing this intermittent and unpredictable issue? Any ideas what I could look at when one of these events is happening to determine why the Data I/O is 100% for an extended period of time? Thank you.
If you are on SQL DB V12, you may also consider using the Query Store feature to root cause this performance problem. It's now in public preview.
In order to turn on Query Store just run the following statement:
ALTER DATABASE your_db SET QUERY_STORE = ON;
I don't use Advanced Queueing at all, but amount of AQ$_PLSQL_NTFNnnnn scheduler jobs grows continuously.
Currently there are 8 such jobs. And because of them - I need to refresh maximum simultaneous jobs running count.
About 2 months ago it was ok with limit of 10, currently I have limit 15 and because of 8 "unnecessary" (at least for me) that jobs - I need to increase it to even 20 or 25 :-S
So, what are they used for? Can I just drop/disable them?
UPD: I've increased number of simultaneous jobs to 25 and in this night the amount of AQ... jobs rose up to 25 :-S Is it a joke?!
It sounds to me like something is using AQ somewhere in your database.
I googled around a bit, and there is some possibly useful information here - http://www.ora600.be/node/5547 - mostly the hidden parameter _srvntfn_max_concurrent_jobs that apparently limits the total number of jobs running for notifications.
Information seems to be hard to come by, but apparently notifications go into the table sys.alert_qt, so you could try having a look in there to see what appears.
You could also have a look in the ALL_QUEUES and other ALL_QUEUE* tables to see if there are any queues on your database you are not aware of.
I am assuming you are using Oracle 11gR1 or 11gR2?
When using a PL/SQL Callback function for processing the AQ queue, we have seen these jobs being generated. You can check this table to find any registered subscriptions:
select * from dba_subscr_registrations;
More about AQ PL/SQL Callback