Unexpected number of Hibernate sessions created in a multi EAR context - spring

Question

We are facing an issue when using Spring JTA + Hibernate in a multiple EAR context.
We have multiple transactional services (using #Transactional).
Our services are POJOs that have a reference to a DAO.
We also have multiple persistence units. For sake of simplicity, lets assume that each service/dao uses a distinct PU.
If Service1 in EAR1 starts a transaction and calls service2 in EAR2 multiple times then :
Calls to Service2 will create a new hibernate session per call.
All the sessions will be flushed/committed at the end of the transactions (i.e. when the Service1 calls is done and commits the transaction). -
Important note : Spring and Hibernate are not loaded by the root classloader, but by the EAR classloaders.
This implies that Spring and Hibernate classes are loaded once per EAR.
Looking into the code and debugging led us to believe that the issue comes from thread local synchronization (i.e. the TransactionSynchronizationManager class).
What happens is:
A transaction is started by Service1 in EAR1
Service1 calls Service2 in EAR2
Because Service2 is also transacted the TransactionAspectSupport kicks in around the Service2 method that is being called.
Down the road the EntityManagerFactoryUtils will check if an EntityManager is registered with TransactionSynchronizationManager. This happens when SharedEntityManagerCreator calls EntityManagerFactoryUtils.doGetTransactionalEntityManager(). See call stack 1 below.
Because no EntityManager is registered, it will create a new EntityManager and a new Hibernate Session. See call stack 2 below.
When returning from the Service2 call, the synchronization is removed because it is considered as a new one. This happens in TransactionAspectSupport.commitTransactionAfterReturning(). See call stack 3 below.
Subsequent calls to Service2 will go through the process of creating a new entity manager + session + synchronization again.
Because all the sessions are also registered with the global transaction, they all get flushed/committed at the end of said transaction.
A side effect, is that if the calls to Service2 modify the same entity, then that entity gets stored multiple times to the RDBMS (i.e. multiple identical update queries are fired).
The same scenario with both Service1 and Service2 deployed in the same EAR yields completely different results:
Only one hibernate session is created
That unique session is flushed at the end of the transaction (and a single update query is fired)
In some of our scenarios, we are seeing hundreds of sessions being created over a short period of time and this is causing performance issues.
It is possible that moving Spring and Hibernate to the root ClassLoader could fix the issue but it does not feel right.
What if the the two EARs are deployed in separate JVMs ?
Moving spring and hibernate up in the ClassLoader hierarchy would also badly break our spring context setup...
Is this the expected behavior of Spring Transactions ?
Is it possible to avoid the thread local synchronization and use transaction synchronization ?
Is our setup (see below) wrong with regard to what we are trying to achieve ?
We are using :
Spring version : 3.2.8.RELEASE
Hibernate version : 4.1.12.Final
Here are some relevant fragments of our Spring setup :
<tx:jta-transaction-manager />
<bean class="org.springframework.orm.jpa.support.PersistenceAnnotationBeanPostProcessor" />
<bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean"
abstract="true">
<property name="jpaVendorAdapter">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
<property name="database" value="${jpa.database}" />
<property name="generateDdl" value="${jpa.generateDdl}" />
<property name="showSql" value="true" />
</bean>
</property>
<property name="dataSource" ref="coreDataSource" />
<property name="jpaProperties">
<props>
<prop key="hibernate.dialect">${hibernate.dialect}</prop>
<prop key="hibernate.transaction.jta.platform">${jpa.transaction.jta.platform}</prop>
<prop key="hibernate.jdbc.wrap_result_sets">${jpa.wrap_result_sets}</prop>
<prop key="hibernate.generate_statistics">${jpa.generateStatistics}</prop>
<prop key="hibernate.cache.region.factory_class">${jpa.cache.factory_class}</prop>
<prop key="hibernate.cache.use_second_level_cache">true</prop>
<prop key="hibernate.cache.use_query_cache">true</prop>
<prop key="hibernate.format_sql">${jpa.formatSql}</prop>
<prop key="hibernate.use_sql_comments">${jpa.useSqlComments}</prop>
</props>
</property>
</bean>
Where jpa.transaction.jta.platform = org.hibernate.service.jta.platform.internal.WeblogicJtaPlatform
<bean id="referentialCoreEntityManagerFactory" parent="entityManagerFactory">
<property name="persistenceXmlLocation" value="classpath:META-INF/referential-core-persistence.xml" />
<property name="persistenceUnitName" value="referential-core" />
</bean>
<?xml version="1.0" encoding="UTF-8"?>
<persistence xmlns="http://java.sun.com/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_1_0.xsd"
version="1.0">
<persistence-unit name="referential-core" transaction-type="JTA" />
</persistence>
Base class of our DAOs showing how the entity managers are defined:
public abstract class BaseDAO<E extends IBaseEntity> extends com.sungard.decalog.framework.domain.dao.jpa.BaseDAO<E> {
private EntityManager entityManager;
/**
* #param entityClass
*/
public <U extends E> BaseDAO(Class<U> entityClass) {
super(entityClass);
}
#Override
protected final EntityManager getEntityManager() {
return entityManager;
}
#PersistenceContext(unitName = "referential-core")
public void setEntityManager(EntityManager entityManager) {
this.entityManager = entityManager;
}
}
Call stack 1 : Creation of new EntityManager and Session:
Daemon Thread [[ACTIVE] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'] (Suspended (breakpoint at line 239 in SessionImpl))
-> SessionImpl.<init>(Connection, SessionFactoryImpl, SessionOwner, TransactionCoordinatorImpl, boolean, long, Interceptor, boolean, boolean, ConnectionReleaseMode, String) line: 239
SessionFactoryImpl$SessionBuilderImpl.openSession() line: 1597
EntityManagerImpl.getRawSession() line: 121
EntityManagerImpl.getSession() line: 97
EntityManagerImpl(AbstractEntityManagerImpl).joinTransaction(boolean) line: 1207
EntityManagerImpl(AbstractEntityManagerImpl).postInit() line: 178
-> EntityManagerImpl.<init>(EntityManagerFactoryImpl, PersistenceContextType, PersistenceUnitTransactionType, boolean, Class, Map) line: 89
EntityManagerFactoryImpl.createEntityManager(Map) line: 179
EntityManagerFactoryImpl.createEntityManager() line: 174
GeneratedMethodAccessor450.invoke(Object, Object[]) line: not available
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
Method.invoke(Object, Object...) line: 597
LocalContainerEntityManagerFactoryBean(AbstractEntityManagerFactoryBean).invokeProxyMethod(Method, Object[]) line: 376
AbstractEntityManagerFactoryBean$ManagedEntityManagerFactoryInvocationHandler.invoke(Object, Method, Object[]) line: 519
$Proxy116.createEntityManager() line: not available
-> EntityManagerFactoryUtils.doGetTransactionalEntityManager(EntityManagerFactory, Map) line: 202
SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(Object, Method, Object[]) line: 211
$Proxy118.find(Class, Object) line: not available
GenericLoaderCodifDAO(BaseDAO<E>).findById(Serializable) line: 57
GenericLoaderCodifService.findGenericLoaderCodif(String, String, String) line: 18
ReferentialFacade.findGenericLoaderCodif(String, String, String) line: 161
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
Method.invoke(Object, Object...) line: 597
AopUtils.invokeJoinpointUsingReflection(Object, Method, Object[]) line: 317
ReflectiveMethodInvocation.invokeJoinpoint() line: 183
ReflectiveMethodInvocation.proceed() line: 150
TransactionInterceptor$1.proceedWithInvocation() line: 96
TransactionInterceptor(TransactionAspectSupport).invokeWithinTransaction(Method, Class, TransactionAspectSupport$InvocationCallback) line: 260
TransactionInterceptor.invoke(MethodInvocation) line: 94
ReflectiveMethodInvocation.proceed() line: 172
JdkDynamicAopProxy.invoke(Object, Method, Object[]) line: 204
$Proxy124.findGenericLoaderCodif(String, String, String) line: not available
ReferentialFacadeBean_590diy_Impl(ReferentialFacadeBean).findGenericLoaderCodif(String, String, String) line: 271
ReferentialFacadeBean_590diy_EOImpl.__WL_invoke(Object, Object[], int) line: not available
SessionRemoteMethodInvoker.invoke(BaseRemoteObject, MethodDescriptor, Object[], int, String, Class<?>) line: 40
ReferentialFacadeBean_590diy_EOImpl.findGenericLoaderCodif(String, String, String) line: not available
ReferentialFacadeBean_590diy_EOImpl_WLSkel.invoke(int, Object[], Object) line: not available
ServerRequest.sendReceive() line: 174
ClusterableRemoteRef.invoke(RemoteReference, RuntimeMethodDescriptor, Object[], Method) line: 345
ClusterableRemoteRef.invoke(Remote, RuntimeMethodDescriptor, Object[], Method) line: 259
ReferentialFacadeBean_590diy_EOImpl_1036_WLStub.findGenericLoaderCodif(String, String, String) line: not available
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
Method.invoke(Object, Object...) line: 597
RmiClientInterceptorUtils.invokeRemoteMethod(MethodInvocation, Object) line: 116
SimpleRemoteStatelessSessionProxyFactoryBean(SimpleRemoteSlsbInvokerInterceptor).doInvoke(MethodInvocation) line: 99
SimpleRemoteStatelessSessionProxyFactoryBean(AbstractRemoteSlsbInvokerInterceptor).invokeInContext(MethodInvocation) line: 141
SimpleRemoteStatelessSessionProxyFactoryBean(AbstractSlsbInvokerInterceptor).invoke(MethodInvocation) line: 189
ReflectiveMethodInvocation.proceed() line: 172
JdkDynamicAopProxy.invoke(Object, Method, Object[]) line: 204
$Proxy314.findGenericLoaderCodif(String, String, String) line: not available
ReferentialService.findGenericLoaderCodif(String, String, String) line: 54
SessionService.refreshSession(ISessionContext, ISessionSnapshot, ISessionData) line: 1114
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
Method.invoke(Object, Object...) line: 597
AopUtils.invokeJoinpointUsingReflection(Object, Method, Object[]) line: 317
JdkDynamicAopProxy.invoke(Object, Method, Object[]) line: 198
$Proxy357.refreshSession(ISessionContext, ISessionSnapshot, ISessionData) line: not available
SessionFacadeService.loadSession(ISessionContext, ISessionSnapshot, ISessionData, ReloadSession) line: 880
SessionFacadeService.loadSession(long, ReloadSession) line: 848
...
<snip>
Call stack 2 : Registration of newly created EntityManager synchronization :
Daemon Thread [[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'] (Suspended)
TransactionSynchronizationManager.registerSynchronization(TransactionSynchronization) line: 289
EntityManagerFactoryUtils.doGetTransactionalEntityManager(EntityManagerFactory, Map) line: 211
SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(Object, Method, Object[]) line: 211
$Proxy118.find(Class, Object) line: not available
GenericLoaderCodifDAO(BaseDAO<E>).findById(Serializable) line: 57
GenericLoaderCodifService.findGenericLoaderCodif(String, String, String) line: 18
ReferentialFacade.findGenericLoaderCodif(String, String, String) line: 161
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
Method.invoke(Object, Object...) line: 597
AopUtils.invokeJoinpointUsingReflection(Object, Method, Object[]) line: 317
ReflectiveMethodInvocation.invokeJoinpoint() line: 183
ReflectiveMethodInvocation.proceed() line: 150
TransactionInterceptor$1.proceedWithInvocation() line: 96
TransactionInterceptor(TransactionAspectSupport).invokeWithinTransaction(Method, Class, TransactionAspectSupport$InvocationCallback) line: 260
TransactionInterceptor.invoke(MethodInvocation) line: 94
ReflectiveMethodInvocation.proceed() line: 172
JdkDynamicAopProxy.invoke(Object, Method, Object[]) line: 204
$Proxy124.findGenericLoaderCodif(String, String, String) line: not available
ReferentialFacadeBean_590diy_Impl(ReferentialFacadeBean).findGenericLoaderCodif(String, String, String) line: 271
ReferentialFacadeBean_590diy_EOImpl.__WL_invoke(Object, Object[], int) line: not available
SessionRemoteMethodInvoker.invoke(BaseRemoteObject, MethodDescriptor, Object[], int, String, Class<?>) line: 40
ReferentialFacadeBean_590diy_EOImpl.findGenericLoaderCodif(String, String, String) line: not available
ReferentialFacadeBean_590diy_EOImpl_WLSkel.invoke(int, Object[], Object) line: not available
ServerRequest.sendReceive() line: 174
ClusterableRemoteRef.invoke(RemoteReference, RuntimeMethodDescriptor, Object[], Method) line: 345
ClusterableRemoteRef.invoke(Remote, RuntimeMethodDescriptor, Object[], Method) line: 259
ReferentialFacadeBean_590diy_EOImpl_1036_WLStub.findGenericLoaderCodif(String, String, String) line: not available
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
Method.invoke(Object, Object...) line: 597
RmiClientInterceptorUtils.invokeRemoteMethod(MethodInvocation, Object) line: 116
SimpleRemoteStatelessSessionProxyFactoryBean(SimpleRemoteSlsbInvokerInterceptor).doInvoke(MethodInvocation) line: 99
SimpleRemoteStatelessSessionProxyFactoryBean(AbstractRemoteSlsbInvokerInterceptor).invokeInContext(MethodInvocation) line: 141
SimpleRemoteStatelessSessionProxyFactoryBean(AbstractSlsbInvokerInterceptor).invoke(MethodInvocation) line: 189
ReflectiveMethodInvocation.proceed() line: 172
JdkDynamicAopProxy.invoke(Object, Method, Object[]) line: 204
$Proxy314.findGenericLoaderCodif(String, String, String) line: not available
ReferentialService.findGenericLoaderCodif(String, String, String) line: 54
SessionService.refreshSession(ISessionContext, ISessionSnapshot, ISessionData) line: 1114
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
Method.invoke(Object, Object...) line: 597
AopUtils.invokeJoinpointUsingReflection(Object, Method, Object[]) line: 317
JdkDynamicAopProxy.invoke(Object, Method, Object[]) line: 198
$Proxy357.refreshSession(ISessionContext, ISessionSnapshot, ISessionData) line: not available
SessionFacadeService.loadSession(ISessionContext, ISessionSnapshot, ISessionData, ReloadSession) line: 880
SessionFacadeService.loadSession(long, ReloadSession) line: 848
...
<snip>
Call stack 3 : Clearing of EntityManager synchronization (this will cause a new one to be created for each Service2 call) :
Daemon Thread [[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'] (Suspended)
TransactionSynchronizationManager.clearSynchronization() line: 328
TransactionSynchronizationManager.clear() line: 464
WebLogicJtaTransactionManager(AbstractPlatformTransactionManager).cleanupAfterCompletion(DefaultTransactionStatus) line: 1006
WebLogicJtaTransactionManager(AbstractPlatformTransactionManager).processCommit(DefaultTransactionStatus) line: 805
WebLogicJtaTransactionManager(AbstractPlatformTransactionManager).commit(TransactionStatus) line: 724
TransactionInterceptor(TransactionAspectSupport).commitTransactionAfterReturning(TransactionAspectSupport$TransactionInfo) line: 475
TransactionInterceptor(TransactionAspectSupport).invokeWithinTransaction(Method, Class, TransactionAspectSupport$InvocationCallback) line: 270
TransactionInterceptor.invoke(MethodInvocation) line: 94
ReflectiveMethodInvocation.proceed() line: 172
JdkDynamicAopProxy.invoke(Object, Method, Object[]) line: 204
$Proxy124.findGenericLoaderCodif(String, String, String) line: not available
ReferentialFacadeBean_590diy_Impl(ReferentialFacadeBean).findGenericLoaderCodif(String, String, String) line: 271
ReferentialFacadeBean_590diy_EOImpl.__WL_invoke(Object, Object[], int) line: not available
SessionRemoteMethodInvoker.invoke(BaseRemoteObject, MethodDescriptor, Object[], int, String, Class<?>) line: 40
ReferentialFacadeBean_590diy_EOImpl.findGenericLoaderCodif(String, String, String) line: not available
ReferentialFacadeBean_590diy_EOImpl_WLSkel.invoke(int, Object[], Object) line: not available
ServerRequest.sendReceive() line: 174
ClusterableRemoteRef.invoke(RemoteReference, RuntimeMethodDescriptor, Object[], Method) line: 345
ClusterableRemoteRef.invoke(Remote, RuntimeMethodDescriptor, Object[], Method) line: 259
ReferentialFacadeBean_590diy_EOImpl_1036_WLStub.findGenericLoaderCodif(String, String, String) line: not available
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
Method.invoke(Object, Object...) line: 597
RmiClientInterceptorUtils.invokeRemoteMethod(MethodInvocation, Object) line: 116
SimpleRemoteStatelessSessionProxyFactoryBean(SimpleRemoteSlsbInvokerInterceptor).doInvoke(MethodInvocation) line: 99
SimpleRemoteStatelessSessionProxyFactoryBean(AbstractRemoteSlsbInvokerInterceptor).invokeInContext(MethodInvocation) line: 141
SimpleRemoteStatelessSessionProxyFactoryBean(AbstractSlsbInvokerInterceptor).invoke(MethodInvocation) line: 189
ReflectiveMethodInvocation.proceed() line: 172
JdkDynamicAopProxy.invoke(Object, Method, Object[]) line: 204
$Proxy314.findGenericLoaderCodif(String, String, String) line: not available
ReferentialService.findGenericLoaderCodif(String, String, String) line: 54
SessionService.refreshSession(ISessionContext, ISessionSnapshot, ISessionData) line: 1114
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
Method.invoke(Object, Object...) line: 597
AopUtils.invokeJoinpointUsingReflection(Object, Method, Object[]) line: 317
JdkDynamicAopProxy.invoke(Object, Method, Object[]) line: 198
$Proxy357.refreshSession(ISessionContext, ISessionSnapshot, ISessionData) line: not available
SessionFacadeService.loadSession(ISessionContext, ISessionSnapshot, ISessionData, ReloadSession) line: 880
SessionFacadeService.loadSession(long, ReloadSession) line: 848
...
<snip>

Answer 1

Try to avoid the use of shared classloaders, this can give rise to all sorts of problems with static variables being shared accidentally between applications.
For example static loggers will be shared, so if you turn on trace debug in one application, you will see the other application slowdown as well, this is just an example of what the shared classloader can cause as unintended side effect.
Something similar could be happening not necessarily in Spring or Hibernate but in some transitive dependency that they use, most of these libraries are meant to be deployed at the application classloader level.
Currently there is no really good reason to put spring and hibernate at a shared classloader, they hardly consume any memory compared to other things.
I personally would put Spring and Hibernate in each WAR instead of moving it further down to the JVM, that would actually make the application un-redeployable, and would try to reduce the use of shared libraries to a minimum.
Also cross EAR calls usually are not needed, it's a programming model that is mostly abandoned due to complexity problems. The usual solution is to ship the service/DAO layer in a separate jar, and deploy the jar in the two applications. The application code can then call the service layer directly without the need for a call to another EAR.
The native transaction manager auto-detected by tx:jta-transaction-manager is then no longer needed and can be replaced by something like JpaTransactionManager or HibernateTransactionManager.