Am newbie to ADF framework. Am developing a REST API by using this framework, and it works perfectly in integrated WebLogic on JDeveloper 12c (12.2.1), but in the production environment am getting an exception while performing the request. I feel like am missing some library in WebLogic.
production URL: http://192.168.2.4:7001/EcomCMS-RESTWebService-context-root/rest/CMS_API/ecomBill/01/child/EcomBillVO?finder=orderRowFinder;OrderNo=2017/156
server log :
<Mar 15, 2018, 10:01:05,953 AM AST> <Error> <HTTP> <BEA-101017> <[ServletContext#741490999[app:EcomCMS_Project1_EcomCMS module:EcomCMS-RESTWebService-context-root path:null spec-version:3.1]] Root cause of ServletException.
java.lang.NoSuchMethodError: oracle.adf.internal.model.rest.core.http.header.HeaderConfiguratorInfo.setSuppressedHeaders(Ljava/util/Set;)V
at oracle.adf.model.servlet.rest.ResourceServlet.execute(ResourceServlet.java:268)
at oracle.adf.model.servlet.rest.ResourceServlet.doGet(ResourceServlet.java:88)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at oracle.adf.model.servlet.rest.ResourceServlet.service(ResourceServlet.java:135)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
Truncated. see log file for complete stacktrace>
<Mar 15, 2018, 10:01:05,966 AM AST> <Notice> <Diagnostics> <BEA-320068> <Watch "UncheckedException" in module "Module-FMWDFW" with severity "Notice" on server "AdminServer" has triggered at Mar 15, 2018 10:01:05 AM AST. Notification details:
WatchRuleType: Log
WatchRule: (SEVERITY = 'Error') AND ((MSGID = 'WL-101020') OR (MSGID = 'WL-101017') OR (MSGID = 'WL-000802') OR (MSGID = 'BEA-101020') OR (MSGID = 'BEA-101017') OR (MSGID = 'BEA-000802'))
WatchData: MESSAGE = [ServletContext#741490999[app:EcomCMS_Project1_EcomCMS module:EcomCMS-RESTWebService-context-root path:null spec-version:3.1]] Root cause of ServletException.
java.lang.NoSuchMethodError: oracle.adf.internal.model.rest.core.http.header.HeaderConfiguratorInfo.setSuppressedHeaders(Ljava/util/Set;)V
at oracle.adf.model.servlet.rest.ResourceServlet.execute(ResourceServlet.java:268)
at oracle.adf.model.servlet.rest.ResourceServlet.doGet(ResourceServlet.java:88)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at oracle.adf.model.servlet.rest.ResourceServlet.service(ResourceServlet.java:135)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:286)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:260)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:137)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:350)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at oracle.adf.model.servlet.ADFBindingFilter.doFilter(ADFBindingFilter.java:207)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at oracle.wsm.agent.handler.servlet.SecurityFilter.doFilter(SecurityFilter.java:321)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:141)
at java.security.AccessController.doPrivileged(Native Method)
at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:315)
at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:650)
at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:124)
at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:232)
at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:94)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:248)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3683)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3649)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:326)
at weblogic.security.service.SecurityManager.runAsForUserCode(SecurityManager.java:197)
at weblogic.servlet.provider.WlsSecurityProvider.runAsForUserCode(WlsSecurityProvider.java:203)
at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:71)
at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2433)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2281)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2259)
at weblogic.servlet.internal.ServletRequestImpl.runInternal(ServletRequestImpl.java:1691)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1651)
at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:270)
at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:348)
at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:333)
at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:54)
at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:640)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:406)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:346)
SUPP_ATTRS = {severity-value=8, rid=0, partition-id=0, partition-name=DOMAIN} SERVER = AdminServer TIMESTAMP = 1521097265953 USERID = <WLS Kernel> MACHINE = WIN-L4R7T5PSL2G MSGID = BEA-101017 DATE = Mar 15, 2018, 10:01:05,953 AM AST SUBSYSTEM = HTTP CONTEXTID = 0556875b-2ef3-4871-93b2-3427f3afa4e8-0000007d TXID = SEVERITY = Error THREAD = [ACTIVE] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'
WatchAlarmType: AutomaticReset
WatchAlarmResetPeriod: 30000>
<Mar 15, 2018, 10:01:06,776 AM AST> <Emergency> <oracle.dfw.incident> <BEA-000000> <incident 25 created with problem key "DFW-99998 [java.lang.NoSuchMethodError][oracle.adf.model.servlet.rest.ResourceServlet.execute][EcomCMS_Project1_EcomCMS]">
Have you installed the ADF Runtime on your stand-alone WebLogic?
How did you create the EAR file for your REST service deployment? Did you include the model project in that ear file as a dependency?
We deployed our application to Azure. It is using the Azure Redis Cache and we are experiencing quite a few timeouts. Namely:
[TimeoutException: Timeout performing GET textobjectDetails__23290_TextObject, inst: 1, mgr: Inactive, queue: 5, qu=0, qs=5, qc=0, wr=0/0, in=56864/0]
[TimeoutException: Timeout performing GET featured_series_CachedSeries, inst: 1, mgr: Inactive, queue: 4, qu=0, qs=4, qc=0, wr=0/0, in=44470/0]
[TimeoutException: Timeout performing GET SeriesByFranchiseId_1_CachedSeries, inst: 1, mgr: Inactive, queue: 3, qu=0, qs=3, qc=0, wr=0/0, in=11252/0]
[TimeoutException: Timeout performing GET media_silo-1-1-0_Media, inst: 1, mgr: Inactive, queue: 3, qu=0, qs=3, qc=0, wr=0/0, in=15188/0]
[TimeoutException: Timeout performing GET textobjectDetails__3092_TextObject, inst: 3, mgr: Inactive, queue: 7, qu=0, qs=7, qc=0, wr=0/0, in=65536/0]
[TimeoutException: Timeout performing GET textobjectbytype_104__TextObject, inst: 11, mgr: Inactive, queue: 9, qu=0, qs=9, qc=0, wr=0/0, in=65536/0]
[TimeoutException: Timeout performing GET groupnews_2_14_TextObject, inst: 1, mgr: Inactive, queue: 7, qu=0, qs=7, qc=0, wr=0/0, in=65536/0]
[TimeoutException: Timeout performing GET archived_news_by_group_13586_1_TextObject, inst: 2, mgr: Inactive, queue: 7, qu=0, qs=7, qc=0, wr=0/0, in=65536/0]
[TimeoutException: Timeout performing GET textobjectDetails__24404_TextObject, inst: 11, mgr: Inactive, queue: 12, qu=0, qs=12, qc=0, wr=0/0, in=65536/0]
[TimeoutException: Timeout performing GET standings_232_lcds_fixtures, inst: 2, mgr: Inactive, queue: 11, qu=0, qs=11, qc=0, wr=0/0, in=65536/0]
[TimeoutException: Timeout performing GET player_name1099_Player, inst: 4, mgr: Inactive, queue: 11, qu=0, qs=11, qc=0, wr=0/0, in=65536/0]
[TimeoutException: Timeout performing GET groupnews_1_6_TextObject, inst: 4, mgr: Inactive, queue: 9, qu=0, qs=9, qc=0, wr=0/0, in=65536/0]
[TimeoutException: Timeout performing GET archivednewscount__20789_TextObject, inst: 2, mgr: Inactive, queue: 11, qu=0, qs=11, qc=0, wr=0/0, in=65536/0]
[TimeoutException: Timeout performing GET media_id3648_Media, inst: 1, mgr: Inactive, queue: 10, qu=0, qs=10, qc=0, wr=0/0, in=65536/0]
The body of the exception is the same for all of them:
StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server):509
StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server):25
StackExchange.Redis.RedisDatabase.StringGet(RedisKey key, CommandFlags flags):16
AB.SiteCaching.Providers.RedisDataSource+<>c__DisplayClasse`1.<RetrieveCacheObject>b__b():0
Microsoft.Practices.TransientFaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func):115
AB.SiteCaching.Providers.RedisDataSource.RetrieveCacheObject[T](String fullCacheKey):56
AB.SiteCaching.Providers.RedisDataSource.RetrieveCached[T](String key, Func`1 onNotCached, TimeSpan timeOut):61
DataAccess.Data.Caching.CachedSeries.GetSeriesByFranchiseId(Int32 franchiseId):64
Shared.Services.SeriesService.LatestYearByFranchiseId(Int32 franchiseId):0
AllBlacksdotcom.Controllers.FixturesController._MostRecentYearOfFixtures(Int32 franchiseId):0
(unknown).lambda_method(Closure , ControllerBase , Object[] ):-1
System.Web.Mvc.ActionMethodDispatcher.Execute(ControllerBase controller, Object[] parameters):0
System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary`2 parameters):87
System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary`2 parameters):0
System.Web.Mvc.Async.AsyncControllerActionInvoker.<BeginInvokeSynchronousActionMethod>b__39(IAsyncResult asyncResult, ActionInvocation innerInvokeState):0
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResult`2.CallEndDelegate(IAsyncResult asyncResult):0
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1.End():41
System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethod(IAsyncResult asyncResult):0
System.Web.Mvc.Async.AsyncControllerActionInvoker+AsyncInvocationWithFilters.<InvokeActionMethodFilterAsynchronouslyRecursive>b__3d():20
System.Web.Mvc.Async.AsyncControllerActionInvoker+AsyncInvocationWithFilters+<>c__DisplayClass46.<InvokeActionMethodFilterAsynchronouslyRecursive>b__3f():134
System.Web.Mvc.Async.AsyncControllerActionInvoker+<>c__DisplayClass33.<BeginInvokeActionMethodWithFilters>b__32(IAsyncResult asyncResult):0
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResult`1.CallEndDelegate(IAsyncResult asyncResult):0
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1.End():41
System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethodWithFilters(IAsyncResult asyncResult):0
System.Web.Mvc.Async.AsyncControllerActionInvoker+<>c__DisplayClass21+<>c__DisplayClass2b.<BeginInvokeAction>b__1c():0
System.Web.Mvc.Async.AsyncControllerActionInvoker+<>c__DisplayClass21.<BeginInvokeAction>b__1e(IAsyncResult asyncResult):65
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResult`1.CallEndDelegate(IAsyncResult asyncResult):0
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1.End():41
System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeAction(IAsyncResult asyncResult):0
System.Web.Mvc.Controller.<BeginExecuteCore>b__1d(IAsyncResult asyncResult, ExecuteCoreState innerState):0
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult):0
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1.End():41
System.Web.Mvc.Controller.EndExecuteCore(IAsyncResult asyncResult):0
System.Web.Mvc.Controller.<BeginExecute>b__15(IAsyncResult asyncResult, Controller controller):0
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult):0
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1.End():41
System.Web.Mvc.Controller.EndExecute(IAsyncResult asyncResult):0
System.Web.Mvc.Controller.System.Web.Mvc.Async.IAsyncController.EndExecute(IAsyncResult asyncResult):0
System.Web.Mvc.MvcHandler.<BeginProcessRequest>b__5(IAsyncResult asyncResult, ProcessRequestState innerState):0
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult):0
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1.End():41
System.Web.Mvc.MvcHandler.EndProcessRequest(IAsyncResult asyncResult):0
System.Web.Mvc.MvcHandler.System.Web.IHttpAsyncHandler.EndProcessRequest(IAsyncResult result):0
System.Web.Mvc.HttpHandlerUtil+ServerExecuteHttpHandlerAsyncWrapper+<>c__DisplayClassa.<EndProcessRequest>b__9():0
System.Web.Mvc.HttpHandlerUtil+ServerExecuteHttpHandlerWrapper+<>c__DisplayClass4.<Wrap>b__3():0
System.Web.Mvc.HttpHandlerUtil+ServerExecuteHttpHandlerWrapper.Wrap[TResult](Func`1 func):0
System.Web.Mvc.HttpHandlerUtil+ServerExecuteHttpHandlerWrapper.Wrap(Action action):25
System.Web.Mvc.HttpHandlerUtil+ServerExecuteHttpHandlerAsyncWrapper.EndProcessRequest(IAsyncResult result):32
System.Web.HttpServerUtility.ExecuteInternal(IHttpHandler handler, TextWriter writer, Boolean preserveForm, Boolean setPreviousPage, VirtualPath path, VirtualPath filePath, String physPath, Exception error, String queryStringOverride):772
Please see our time-out settings:
retryTimeoutInMilliseconds = "5000"
connectionTimeoutInMilliseconds = "5000"
operationTimeoutInMilliseconds = "1000"
How do I approach these time-outs? Would increasing operationTimeoutInMilliseconds do the trick? I also read about g-zip compression being helpful for decreasing time that it takes to read the data from Redis.
All nuget packages related to Redis are on their latest versions. We are using C1 version of Azure Redis Cache (would increasing to C2 help?).
A few points that improved our situation:
Protobuf-net instead of BinaryFormatter
I recommend using protobuf-net as it will reduce the size of values that you want to store in your cache.
public interface ICacheDataSerializer
{
byte[] Serialize(object o);
T Deserialize<T>(byte[] stream);
}
public class ProtobufNetSerializer : ICacheDataSerializer
{
public byte[] Serialize(object o)
{
using (var memoryStream = new MemoryStream())
{
Serializer.Serialize(memoryStream, o);
return memoryStream.ToArray();
}
}
public T Deserialize<T>(byte[] stream)
{
var memoryStream = new MemoryStream(stream);
return Serializer.Deserialize<T>(memoryStream);
}
}
Implement retry strategy
Implement this RedisCacheTransientErrorDetectionStrategy to handle timeout issues.
using Microsoft.Practices.TransientFaultHandling;
public class RedisCacheTransientErrorDetectionStrategy : ITransientErrorDetectionStrategy
{
/// <summary>
/// Custom Redis Transient Error Detenction Strategy must have been implemented to satisfy Redis exceptions.
/// </summary>
/// <param name="ex"></param>
/// <returns></returns>
public bool IsTransient(Exception ex)
{
if (ex == null) return false;
if (ex is TimeoutException) return true;
if (ex is RedisServerException) return true;
if (ex is RedisException) return true;
if (ex.InnerException != null)
{
return IsTransient(ex.InnerException);
}
return false;
}
}
Instantiate like this:
private readonly RetryPolicy _retryPolicy;
// CODE
var retryStrategy = new FixedInterval(3, TimeSpan.FromSeconds(2));
_retryPolicy = new RetryPolicy<RedisCacheTransientErrorDetectionStrategy>(retryStrategy);
Use like this:
var cachedString = _retryPolicy.ExecuteAction(() => dataCache.StringGet(fullCacheKey));
Review your code to minimize cache calls and values that you are storing in your cache. I reduced lots of errors by storing values more efficiently.
If none of this helps. Move to higher cache (we ended up using C3 instead of C1).
In all of your exception messages, you have something like "in=65536/0". This indicates that 65536 bytes have been received locally (in the kernel's socket buffer on the local machine) but have not yet been processed by StackExchange.Redis. This would point at an issue on the client app side of things that is keeping the system from processing your response. Two possibilities I can think of that could lead to this:
Your system does not have enough free thread pool threads to process the data fast enough. If you update to build 1.0.450 or later of StackExchange.Redis, the timeout messages will include statistics for how busy the thread pool is, which will help diagnose this issue. Alternatively, you can take use code like https://github.com/JonCole/SampleCode/blob/master/ThreadPoolMonitor/ThreadPoolLogger.cs to log this same information yourself whenever you get an exception.
You are receiving a very large object from the cache and the size of the object is not received within the timeout configured on StackExchange.Redis. When this happens, all calls that were queued onto the same connection behind the large item will timeout when the large item times out.
Anyway, hope this helps.
Go to Azure and get a bigger Redis instance. That t
We've come across the deadlock situation below when running multiple Java EE MDB instances that are trying to write to the DB:
[deadlocked thread] [ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)':
Thread '[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'' is waiting to acquire lock 'oracle.jdbc.driver.T4CConnection#90106ee' that is held by thread '[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)''
Stack trace:
oracle.sql.ARRAY.toBytes(ARRAY.java:673)
oracle.jdbc.driver.OraclePreparedStatement.setArrayCritical(OraclePreparedStatement.java:5985)
oracle.jdbc.driver.OraclePreparedStatement.setARRAYInternal(OraclePreparedStatement.java:5944)
oracle.jdbc.driver.OraclePreparedStatement.setObjectCritical(OraclePreparedStatement.java:8782)
oracle.jdbc.driver.OraclePreparedStatement.setObjectInternal(OraclePreparedStatement.java:8278)
oracle.jdbc.driver.OraclePreparedStatement.setObject(OraclePreparedStatement.java:8868)
oracle.jdbc.driver.OraclePreparedStatementWrapper.setObject(OraclePreparedStatementWrapper.java:240)
weblogic.jdbc.wrapper.PreparedStatement.setObject(PreparedStatement.java:287)
org.springframework.jdbc.core.StatementCreatorUtils.setValue(StatementCreatorUtils.java:356)
org.springframework.jdbc.core.StatementCreatorUtils.setParameterValueInternal(StatementCreatorUtils.java:216)
org.springframework.jdbc.core.StatementCreatorUtils.setParameterValue(StatementCreatorUtils.java:127)
org.springframework.jdbc.core.PreparedStatementCreatorFactory$PreparedStatementCreatorImpl.setValues(PreparedStatementCreatorFactory.java:298)
org.springframework.jdbc.object.BatchSqlUpdate$1.setValues(BatchSqlUpdate.java:192)
org.springframework.jdbc.core.JdbcTemplate$4.doInPreparedStatement(JdbcTemplate.java:892)
org.springframework.jdbc.core.JdbcTemplate$4.doInPreparedStatement(JdbcTemplate.java:1)
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:586)
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:614)
org.springframework.jdbc.core.JdbcTemplate.batchUpdate(JdbcTemplate.java:883)
org.springframework.jdbc.object.BatchSqlUpdate.flush(BatchSqlUpdate.java:184)
com.csfb.fao.rds.rfi.common.dao.storedprocs.SaveEarlyExceptionBatchStoredProc.execute(SaveEarlyExceptionBatchStoredProc.java:93)
com.csfb.fao.rds.rfi.common.dao.EarlyExceptionDAOImpl.saveEarlyExceptionBatch(EarlyExceptionDAOImpl.java:34)
com.csfb.fao.rds.rfi.application.rulesengine.RulesEngine.saveEarlyExceptions(RulesEngine.java:302)
com.csfb.fao.rds.rfi.application.rulesengine.RulesEngine.executeRules(RulesEngine.java:209)
com.csfb.fao.rds.rfi.application.rulesengine.RulesEngine.onMessage(RulesEngine.java:97)
com.csfb.fao.rds.feeds.process.BaseWorkerMDB.onMessage(BaseWorkerMDB.java:518)
weblogic.ejb.container.internal.MDListener.execute(MDListener.java:466)
weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:371)
weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:327)
weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4547)
weblogic.jms.client.JMSSession.execute(JMSSession.java:4233)
weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3709)
weblogic.jms.client.JMSSession.access$000(JMSSession.java:114)
weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:5058)
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
and...
[deadlocked thread] [ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)':
Thread '[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'' is waiting to acquire lock 'oracle.jdbc.driver.T4CConnection#b48b568' that is held by thread '[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)''
Stack trace:
oracle.sql.ARRAY.toBytes(ARRAY.java:673)
oracle.jdbc.driver.OraclePreparedStatement.setArrayCritical(OraclePreparedStatement.java:5985)
oracle.jdbc.driver.OraclePreparedStatement.setARRAYInternal(OraclePreparedStatement.java:5944)
oracle.jdbc.driver.OraclePreparedStatement.setObjectCritical(OraclePreparedStatement.java:8782)
oracle.jdbc.driver.OraclePreparedStatement.setObjectInternal(OraclePreparedStatement.java:8278)
oracle.jdbc.driver.OraclePreparedStatement.setObject(OraclePreparedStatement.java:8868)
oracle.jdbc.driver.OraclePreparedStatementWrapper.setObject(OraclePreparedStatementWrapper.java:240)
weblogic.jdbc.wrapper.PreparedStatement.setObject(PreparedStatement.java:287)
org.springframework.jdbc.core.StatementCreatorUtils.setValue(StatementCreatorUtils.java:356)
org.springframework.jdbc.core.StatementCreatorUtils.setParameterValueInternal(StatementCreatorUtils.java:216)
org.springframework.jdbc.core.StatementCreatorUtils.setParameterValue(StatementCreatorUtils.java:127)
org.springframework.jdbc.core.PreparedStatementCreatorFactory$PreparedStatementCreatorImpl.setValues(PreparedStatementCreatorFactory.java:298)
org.springframework.jdbc.object.BatchSqlUpdate$1.setValues(BatchSqlUpdate.java:192)
org.springframework.jdbc.core.JdbcTemplate$4.doInPreparedStatement(JdbcTemplate.java:892)
org.springframework.jdbc.core.JdbcTemplate$4.doInPreparedStatement(JdbcTemplate.java:1)
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:586)
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:614)
org.springframework.jdbc.core.JdbcTemplate.batchUpdate(JdbcTemplate.java:883)
org.springframework.jdbc.object.BatchSqlUpdate.flush(BatchSqlUpdate.java:184)
com.csfb.fao.rds.rfi.common.dao.storedprocs.SaveEarlyExceptionBatchStoredProc.execute(SaveEarlyExceptionBatchStoredProc.java:93)
com.csfb.fao.rds.rfi.common.dao.EarlyExceptionDAOImpl.saveEarlyExceptionBatch(EarlyExceptionDAOImpl.java:34)
com.csfb.fao.rds.rfi.application.rulesengine.RulesEngine.saveEarlyExceptions(RulesEngine.java:302)
com.csfb.fao.rds.rfi.application.rulesengine.RulesEngine.executeRules(RulesEngine.java:209)
com.csfb.fao.rds.rfi.application.rulesengine.RulesEngine.onMessage(RulesEngine.java:97)
com.csfb.fao.rds.feeds.process.BaseWorkerMDB.onMessage(BaseWorkerMDB.java:518)
weblogic.ejb.container.internal.MDListener.execute(MDListener.java:466)
weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:371)
weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:327)
weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4547)
weblogic.jms.client.JMSSession.execute(JMSSession.java:4233)
weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3709)
weblogic.jms.client.JMSSession.access$000(JMSSession.java:114)
weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:5058)
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Looking at the ARRAY.toBytes() method:
public byte[] toBytes()
throws SQLException
{
synchronized (getInternalConnection())
{
return this.descriptor.toBytes(this, this.enableBuffering);
}
}
..., it synchronizes on the following method (getInternalConnection() -> getPhysicalConnection()):
oracle.jdbc.internal.OracleConnection getPhysicalConnection()
{
if (this.physicalConnection == null)
{
try
{
this.physicalConnection = ((oracle.jdbc.internal.OracleConnection)new OracleDriver().defaultConnection());
}
catch (SQLException localSQLException)
{
}
}
return this.physicalConnection;
}
defaultConnection() does the following:
public Connection defaultConnection()
throws SQLException
{
if ((defaultConn == null) || (defaultConn.isClosed()))
{
synchronized (OracleDriver.class)
{
if ((defaultConn == null) || (defaultConn.isClosed()))
{
defaultConn = connect("jdbc:oracle:kprb:", new Properties());
}
}
}
return defaultConn;
}
So there's synchronizations on the connection instance and OracleDriver.class object... I can't see how this can deadlock. To get to the point of needing the lock on OracleDriver.class, the thread would already have the lock on the connection instance.... clearly I'm missing something.
We are creating the ARRAY type using the following code:
public ARRAY getOracleLongArray(DataSource ds, List<Long> vals, String typeName)
throws SQLException {
oracle.jdbc.OracleConnection conn = (oracle.jdbc.OracleConnection) ds
.getConnection();
Object[] data = vals.toArray(new Object[0]);
ARRAY mddArray = conn.createARRAY(typeName, data);
conn.close(); // Close the extra connection made here.
return mddArray;
}
Thanks
It looks like you are sharing that connection with multiple threads (by way of that defaultConnection() method. Don't do that, make sure a connection is used by only one thread at a time.
Mark had the answer:
We were using a Connection pool but passing the DataSource reference to it - every time we created a New ARRAY object it was getting a different Connection object from the Datasource (to that used by the actual db call) and thus creating a new db connection.
We wrapped the DataSource to always return the same connection and it fixed the issue.
Thanks