I'm using Hikari CP (5.0.1), Hibernate (5.6.3), hibernate-hikaricp (6.0.0) and I'm getting the connections using HibernateUtil class.
This all works fine, without any issues.
Then I have this #Scheduled method that looks like this:
#Scheduled(fixedDelay = 5000)
public void runCalculations() {
try {
this.calculateKPIs();
} catch (Throwable t) {
logger.error("runCalculations() -> catched exception! - " + t.getMessage(), t);
}
}
private void calculateKPIs() {
Session sessApp = HibernateUtilApp.getSessionFactory().openSession();
Transaction txApp = sessApp.beginTransaction();
Integer val = (Integer) sessApp.createSQLQuery("some query").uniqueResult();
txApp.commit();
sessApp.close();
}
All is running absolutely fine until once upon a time there is a deadlock on the database level - I'm reading data (every 5 seconds) from tables that are sometimes changed by other program (every 30 minutes).
And when this deadlock happens there is this Exception catched and logged:
2022-04-19 08:30:39,760 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (scheduled-task-pool-5) Transaction (Process ID 442) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
2022-04-19 08:30:39,760 ERROR [com.yyy.DW.workers.BW.BWWorker] (scheduled-task-pool-5) BW runCalculations() -> catched exception! - org.hibernate.exception.LockAcquisitionException: could not execute query: javax.persistence.OptimisticLockException: org.hibernate.exception.LockAcquisitionException: could not execute query
at deployment.xxx.war//org.hibernate.internal.ExceptionConverterImpl.wrapLockException(ExceptionConverterImpl.java:277)
at deployment.xxx.war//org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:98)
at deployment.xxx.war//org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1626)
at deployment.xxx.war//org.hibernate.query.internal.AbstractProducedQuery.uniqueResult(AbstractProducedQuery.java:1659)
at deployment.xxx.war//com.yyy.DW.workers.BW.BWWorker.calculateGenericKPI(BWWorker.java:248)
at deployment.xxx.war//com.yyy.DW.workers.BW.BWWorker.runCalculations(BWWorker.java:64)
at jdk.internal.reflect.GeneratedMethodAccessor271.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at deployment.xxx.war//org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)
at deployment.xxx.war//org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.hibernate.exception.LockAcquisitionException: could not execute query
at deployment.xxx.war//org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:120)
at deployment.xxx.war//org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:37)
at deployment.xxx.war//org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
at deployment.xxx.war//org.hibernate.loader.Loader.doList(Loader.java:2871)
at deployment.xxx.war//org.hibernate.loader.Loader.doList(Loader.java:2850)
at deployment.xxx.war//org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2682)
at deployment.xxx.war//org.hibernate.loader.Loader.list(Loader.java:2677)
at deployment.xxx.war//org.hibernate.loader.custom.CustomLoader.list(CustomLoader.java:338)
at deployment.xxx.war//org.hibernate.internal.SessionImpl.listCustomQuery(SessionImpl.java:2181)
at deployment.xxx.war//org.hibernate.internal.AbstractSharedSessionContract.list(AbstractSharedSessionContract.java:1204)
at deployment.xxx.war//org.hibernate.query.internal.NativeQueryImpl.doList(NativeQueryImpl.java:177)
at deployment.xxx.war//org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1617)
... 14 more
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 442) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
at deployment.xxx.war//com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:265)
at deployment.xxx.war//com.microsoft.sqlserver.jdbc.SQLServerResultSet$FetchBuffer.nextRow(SQLServerResultSet.java:5471)
at deployment.xxx.war//com.microsoft.sqlserver.jdbc.SQLServerResultSet.fetchBufferNext(SQLServerResultSet.java:1794)
at deployment.xxx.war//com.microsoft.sqlserver.jdbc.SQLServerResultSet.next(SQLServerResultSet.java:1052)
at deployment.xxx.war//com.zaxxer.hikari.pool.HikariProxyResultSet.next(HikariProxyResultSet.java)
at deployment.xxx.war//org.hibernate.loader.Loader.getRowsFromResultSet(Loader.java:1043)
at deployment.xxx.war//org.hibernate.loader.Loader.processResultSet(Loader.java:998)
at deployment.xxx.war//org.hibernate.loader.Loader.doQuery(Loader.java:967)
at deployment.xxx.war//org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:357)
at deployment.xxx.war//org.hibernate.loader.Loader.doList(Loader.java:2868)
... 22 more
But after this exception is raised and catched, the #Scheduled job is never run again (other #Scheduled jobs are fine). The scheduler's task that ran the job (here scheduled-task-pool-5) continues to work and executes other tasks but this one is never run again.
Later in the logs I can see that HikariCP detected and unleaked the leaked connection from this code (as expected).
But I don't really understand why the #Scheduled job is not running again after the catched Exception.
Can anyone help please?
Related
I'm using a Spring Boot Scheduler to run a query on the DB daily to find some records based on a condition and update the records returned. Fetching the records using JPA works fine, but when I loop through them, update them, and try to save each updated record I get the following error:
Could not commit JPA transaction; nested exception is javax.persistence.RollbackException: Error while committing the transaction
Caused by: javax.persistence.RollbackException: Error while committing the transaction at org.hibernate.internal.ExceptionConverterImpl.convertCommitException(ExceptionConverterImpl.java:81) at org.hibernate.engine.transaction.internal.TransactionImpl.commit(TransactionImpl.java:104) at org.springframework.orm.jpa.JpaTransactionManager.doCommit(JpaTransactionManager.java:562) ... 30 more Caused by: java.lang.NullPointerException at com.xxx.yyy.config.JpaAuditingConfiguration.auditorProvider$lambda-0(JpaAuditingConfiguration.kt:15) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) at com.sun.proxy.$Proxy168.getCurrentAuditor(Unknown Source) at java.base/java.util.Optional.map(Optional.java:265) at org.springframework.data.auditing.AuditingHandler.getAuditor(AuditingHandler.java:109) at org.springframework.data.auditing.AuditingHandler.markModified(AuditingHandler.java:104) at org.springframework.data.jpa.domain.support.AuditingEntityListener.touchForUpdate(AuditingEntityListener.java:112).
Here is the scheduler code I have. If I run the same code inside my service and call it using an endpoint everything works fine:
#Component
class Scheduler(
private val repository: Repository
) {
#Scheduled(cron = "0 0 2 * * *")
fun expire() {
val records = repository.findRecords()
for (record in records) {
try {
// Call some external API using record.id but this part is commented out for now until the saving works
record.active = false
repository.save(record)
} catch (ex: Exception) {
logger.error("Error expiring record " + record.id)
logger.error("Exception: ${ex.printStackTrace()}")
continue
}
}
}
}
the null pointer exception happens in the JpaAuditingConfiguration config I use for storing the created_at and last_modified_at dates. Here is the code I have for that class:
#Configuration
#EnableJpaAuditing(auditorAwareRef = "auditorProvider")
class JpaAuditingConfiguration {
#Bean
fun auditorProvider(): AuditorAware<String> {
return AuditorAware { Optional.of(SecurityContextHolder.getContext().authentication.name) }
}
}
Your JpaAuditingConfiguration requires the security context to be non null when you make modifications. When you're running your task in a scheduler there is no active request, so no active session, and therefore your authentication is null.
Usually, this is solved by making a special app user and manually authenticating them in your scheduled task.
I am trying to insert some 50k records into db. We have used AbstractRoutingDataSource which resolve Datasource using TenantContext which is a utility class and has a private static final ThreadLocal CURRENT_TENANT = new ThreadLocal<>();
when I am using parallel stream or if I am trying to make the method #Async I am getting the below error
Code:
.parallelStream()
.forEach(row -> {
TenantContext.setCurrentTenant(centerCd);
someDao.insert(row);
});
Error:
org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is java.lang.IllegalStateException: Cannot determine target DataSource for lookup key [null]
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:305)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:378)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:474)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:289)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Cannot determine target DataSource for lookup key [null]
at org.springframework.jdbc.datasource.lookup.AbstractRoutingDataSource.determineTargetDataSource(AbstractRoutingDataSource.java:207)
at org.springframework.jdbc.datasource.lookup.AbstractRoutingDataSource.getConnection(AbstractRoutingDataSource.java:169)
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:262)
... 10 common frames omitted
It works exactly like you described: your TenantContext is exactly ThreadLocal and exists in a thread, which is initiated either by parallelStream() or Async method. (in reality, the call inside of the Async or forEach method is a run from Runnable)
The data source is attempted to be injected/resolved at start of the thread: because your transaction have to be started at the thread creation, before your Runnable gets into a run method. And at this moment of time you haven't yet specified your tenant, call TenantContext.setCurrentTenant(centerCd) is performed later in a run method implementation.
I would suggest applying such structure to your code:
class TenantAwareThread extends Thread {
public TenantAwareThread(Runnable target, TenantData tenantData) {
super(target);
TenantContext.setCurrentTenant(tenantData);
}
}
#Autowired
TaskExecutor executor;
void startTask(TenantData tenantData, RowData row) {
executor.execute(
new TenantAwareThread(() -> {
someDao.insert(row);
},
tenantData));
}
You create a new thread type which is aware of tenant data from the very beginning. And simply wraps your executions into such thread.
I'm using Spring Boot 1.5.3 and Atomikos 4.0.6. I have a DefaultMessageListenerContainer setup to listen a JMS queue and handle incoming messages in a transaction using the Spring JTATransactionManager configured to use Atomikos. The message handler calls a transactional service that tries to process the message inside a try-catch-block and in case of exceptions calls other transactional methods for logging purposes. The idea is that the transaction should be rolled back only after everything is logged and the encountered RuntimeException is thrown from the catch-block.
#Transactional
public void handleMessage(UnmarshalledMessage message) {
try {
Thing thing = repository.find(message.getId());
...
}
catch (Exception e) {
// NoResultException translated into EmptyResultDataAccessException
logger.logUsingTransactions(e.getMessage());
throw e;
}
}
However, what happens that the transaction is rolled back immediately after it's originally thrown inside repository.find(). When attempting to read from the database inside the catch-block, an exception is thrown since the transaction has been marked as rollback only:
c.atomikos.jdbc.AtomikosSQLException - Transaction is marked for rollback only or has timed out
com.atomikos.datasource.xa.session.InvalidSessionHandleStateException: Transaction is marked for rollback only or has timed out
at com.atomikos.datasource.xa.session.NotInBranchStateHandler.checkEnlistBeforeUse(NotInBranchStateHandler.java:39)
at com.atomikos.datasource.xa.session.TransactionContext.checkEnlistBeforeUse(TransactionContext.java:70)
at com.atomikos.datasource.xa.session.SessionHandleState.notifyBeforeUse(SessionHandleState.java:160)
at com.atomikos.jdbc.AtomikosConnectionProxy.enlist(AtomikosConnectionProxy.java:207)
at com.atomikos.jdbc.AtomikosConnectionProxy.invoke(AtomikosConnectionProxy.java:122)
at com.sun.proxy.$Proxy129.prepareStatement(Unknown Source)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.prepareStatement(DatabaseAccessor.java:1565)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.prepareStatement(DatabaseAccessor.java:1514)
at org.eclipse.persistence.internal.databaseaccess.DatabaseCall.prepareStatement(DatabaseCall.java:778)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:621)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:560)
at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:2055)
at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:570)
at org.eclipse.persistence.sessions.server.ClientSession.executeCall(ClientSession.java:258)
at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:242)
at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:228)
at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectOneRow(DatasourceCallQueryMechanism.java:714)
at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectOneRowFromTable(ExpressionQueryMechanism.java:2803)
at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectOneRow(ExpressionQueryMechanism.java:2756)
at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:555)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1175)
at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:904)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1134)
at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:441)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1222)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2896)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1857)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1839)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1790)
at org.eclipse.persistence.internal.jpa.EntityManagerImpl.executeQuery(EntityManagerImpl.java:911)
at org.eclipse.persistence.internal.jpa.EntityManagerImpl.findInternal(EntityManagerImpl.java:854)
at org.eclipse.persistence.internal.jpa.EntityManagerImpl.find(EntityManagerImpl.java:730)
at org.eclipse.persistence.internal.jpa.EntityManagerImpl.find(EntityManagerImpl.java:599)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
I'd like to know what is causing this behavior and how to resolve the issue. Note that this setup has been working correctly when run in Weblogic. For additional information, here is a transaction trace log when the exception is first encountered inside the repository method.
DEBUG o.s.t.jta.JtaTransactionManager - Initiating transaction commit
DEBUG o.s.t.jta.JtaTransactionManager - Creating new transaction with name [myMessageListenerContainer]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT
DEBUG o.s.t.jta.JtaTransactionManager - Participating in existing transaction
TRACE o.s.t.i.TransactionInterceptor - Getting transaction for [my.transactional.messagehandler.handleMessage]
DEBUG o.s.t.jta.JtaTransactionManager - Participating in existing transaction
TRACE o.s.t.i.TransactionInterceptor - Getting transaction for [my.repository.class.method]
TRACE o.s.t.i.TransactionInterceptor - Completing transaction for [my.repository.class.method] after exception: org.springframework.dao.EmptyResultDataAccessException: ProcessableMessage with id 443e73e7-0905-416b-9e03-4aaa2bbf09fb; nested exception is javax.persistence.NoResultException: ProcessableMessage with id [message-id]
TRACE o.s.t.i.RuleBasedTransactionAttribute - Applying rules to determine whether transaction should rollback on org.springframework.dao.EmptyResultDataAccessException: ProcessableMessage with id 443e73e7-0905-416b-9e03-4aaa2bbf09fb; nested exception is javax.persistence.NoResultException: ProcessableMessage with id [message-id]
TRACE o.s.t.i.RuleBasedTransactionAttribute - Winning rollback rule is: null
TRACE o.s.t.i.RuleBasedTransactionAttribute - No relevant rollback rule found: applying default rules
DEBUG o.s.t.jta.JtaTransactionManager - Participating transaction failed - marking existing transaction as rollback-only
DEBUG o.s.t.jta.JtaTransactionManager - Setting JTA transaction rollback-only
EDIT:
I am using JPA and the NoResultException is initially thrown in a following way:
public static <T> T mandatoryFind(EntityManager em, Class<T> type, Object id) throws NoResultException {
T found = em.find(type, id);
if (found == null) {
throw new NoResultException(type.getSimpleName() +" with id "+ id);
}
return found;
}
Which in turn is called from a class annotated with #Transactional(noRollbackFor = NoResultException.class). The exception is raised in a fairly normal use case - what I'm troubled with is why the transaction is rolled back before I can handle the exception?
It sounds like you do use JPA under the hood (based on your debug logs).
This is a typical behaviour then you do query a single result.
NoResultException:
Thrown by the persistence provider when Query.getSingleResult() or TypedQuery.getSingleResult()is executed on a query and there is no result to return. This exception will not cause the current transaction, if one is active, to be marked for rollback.
In order to avoid that behaviour you do query result as a List.
I've developed a monitoring application that uses Spring-3.0 Hibernate-3.6.5 with SpringHibernateTemplate as DAO. Database SQL Server 2008. This app has long running transaction every day.
#Repository("areaDaoHibernate")
public class AreaDAO implements IArea {
protected HibernateTemplate template = null;
#Autowired #Required
public void setSessionFactory(SessionFactory sessionFactory) {
template = new HibernateTemplate(sessionFactory);
}
// basic dao method
}
This app has some scheduling (implements Runable) via springbeans.
#Component("dailyTask")
#Scope("prototype")
public class DailyTask implements Runnable {
#Override
public void run() {
// running task
}
}
This app uses Apache DBCP for pooling.
<bean id="sessionFactory" class="org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean">
<property name="dataSource">
<ref bean="CustodyDataSource" />
</property>
…
</bean>
<bean id = "CustodyDataSource"
class = "org.apache.commons.dbcp.BasicDataSource"
p:driverClassName = "${jdbc1.driverClassName}"
p:url = "${jdbc1.url}"
p:username = "${jdbc1.username}"
p:password = "${jdbc1.password}"
p:maxIdle = "${jdbc1.maxIdle}"
p:maxActive = "${jdbc1.maxActive}"
p:validationQuery = "${jdbc1.validationQuery}"
p:testWhileIdle = "${jdbc1.testWhileIdle}" />
<bean id = "transactionManager"
class = "org.springframework.orm.hibernate3.HibernateTransactionManager"
p:sessionFactory-ref = "sessionFactory" />
I have problem when this app deadlock, the scheduling stop to run.
The error shows:
2012-12-27 12:57:20,861 WARN (JDBCExceptionReporter:233) - SQL Error: 1205, SQLState: 40001
2012-12-27 12:57:20,862 ERROR (JDBCExceptionReporter:234) - Transaction (Process ID 55) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
2012-12-27 12:57:20,866 ERROR (DailyTask:238) - org.springframework.dao.CannotAcquireLockException: could not update: [com.btpn.custody.entity.MasterData#00176ed4-b7a2-4da5-a266-ffaab3654050]; SQL [update T_CIF set __UPDATED_DATE=?, STATUS_MD=? where __SID=?]; nested exception is org.hibernate.exception.LockAcquisitionException: could not update: [com.btpn.custody.entity.MasterData#00176ed4-b7a2-4da5-a266-ffaab3654050]
at org.springframework.orm.hibernate3.SessionFactoryUtils.convertHibernateAccessException(SessionFactoryUtils.java:633)
at org.springframework.orm.hibernate3.HibernateTransactionManager.convertHibernateAccessException(HibernateTransactionManager.java:793)
at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:664)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:754)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:120)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy33.relateWithBranch(Unknown Source)
at com.btpn.custody.bean.DailyTask.relateMasterDataWithBranch(DailyTask.java:1166)
at com.btpn.custody.bean.DailyTask.manualInitial(DailyTask.java:213)
at com.btpn.custody.bean.DailyTask.run(DailyTask.java:197)
at java.lang.Thread.run(Unknown Source)
Caused by: org.hibernate.exception.LockAcquisitionException: could not update: [com.btpn.custody.entity.MasterData#00176ed4-b7a2-4da5-a266-ffaab3654050]
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:107)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2612)
at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:2494)
at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2821)
at org.hibernate.action.EntityUpdateAction.execute(EntityUpdateAction.java:113)
at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:273)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:265)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:185)
at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321)
at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:51)
at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1216)
at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:383)
at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:133)
at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:656)
... 11 more
Caused by: java.sql.SQLException: Transaction (Process ID 55) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:368)
at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:2820)
at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2258)
at net.sourceforge.jtds.jdbc.TdsCore.getMoreResults(TdsCore.java:632)
at net.sourceforge.jtds.jdbc.JtdsStatement.processResults(JtdsStatement.java:584)
at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQL(JtdsStatement.java:546)
at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeUpdate(JtdsPreparedStatement.java:504)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:102)
at org.hibernate.jdbc.NonBatchingBatcher.addToBatch(NonBatchingBatcher.java:46)
at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2590)
... 23 more
So my questions:
Should I change connection pool to avoid deadlock problem?
Why my scheduling bean stop run after deadlock happen?
The deadlock occurred in SQL Server. This is neither Spring, Hibernate, the connection pool nor SQL Servers fault. The problem is that your application is locking rows that either your application, or other applications are also trying to lock (in a deadlock manner).
The easiest fix is probably to have a retry-strategy for your job (just retry the whole transaction if it fails). You could also try to figure out exactly which applications / users were involved in the deadlock by looking at the SQL Server logs.
I'm trying to use Quartz 1.8.5 (and I also tried Quartz2.0 - same issue) with Spring 3.0.0.
However, I do not want to use Spring's utility wrappers for Quartz schedulers, triggers, jobs...
I would like to create all these objects programmatically from my application, just like in a
standalone Java app with no container - for many reasons, but mainly because i have requirements for
dynamic triggers and JobDetails...
So, I have my quartz.properties on the classpath, where I define a JobStoreTX:
org.quartz.scheduler.instanceName: NotificationsScheduler
org.quartz.scheduler.instanceId: AUTO
org.quartz.scheduler.skipUpdateCheck: true
org.quartz.threadPool.class: org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount: 3
org.quartz.threadPool.threadPriority: 5
org.quartz.jobStore.misfireThreshold: 60000
org.quartz.jobStore.class: org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.jobStore.driverDelegateClass: org.quartz.impl.jdbcjobstore.StdJDBCDelegate
org.quartz.jobStore.useProperties: false
org.quartz.jobStore.dataSource: qzDS
org.quartz.jobStore.tablePrefix: QRTZ_
org.quartz.jobStore.isClustered: false
org.quartz.dataSource.qzDS.driver: com.mysql.jdbc.Driver
org.quartz.dataSource.qzDS.URL: jdbc:mysql://localhost:3306/myschema
org.quartz.dataSource.qzDS.user: myuser
org.quartz.dataSource.qzDS.password: mypwd
org.quartz.dataSource.qzDS.maxConnections: 5
Then, I have a simple Initializer class, taken almost as is from example1 of Quartz distribution,
which is using an un-modified HelloJob from the same example:
public class NotificationInitializer {
public void init(){
try {
SchedulerFactory schdFact = new StdSchedulerFactory("quartz.properties");
Scheduler schd = schdFact.getScheduler();
schd.start();
JobDetail jd = new JobDetail("hellojob", Scheduler.DEFAULT_GROUP, HelloJob.class);
Trigger t = TriggerUtils.makeImmediateTrigger(2, 1000);
t.setStartTime(new Date());
schd.scheduleJob(jd, t);
// wait long enough so that the scheduler as an opportunity to run
..........
schd.shutdown(true);
} catch (SchedulerException e) {
.....
}
}
}
And I also have a simple unit test that calls this class:
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration(locations = "classpath:spring/applicationContext.xml")
public class NotificationInitializerTest {
private final static Logger logger = LoggerFactory
.getLogger(NotificationInitializerTest.class);
#BeforeClass
public static void setupClass() {
PropertyConfigurator.configure(Loader
.getResource("spring/log4j.properties"));
PropertyConfigurator.configure(Loader
.getResource("spring/quartz.properties"));
}
#Test
public void testInit() {
NotificationInitializer ni = new NotificationInitializer();
ni.init();
logger.info("NotificationInitializer.init() finished OK");
}
}
And when I run this unit test I'm getting the following errors - see the full stack trace below:
Caused by: org.springframework.beans.factory.BeanCreationException:
Error creating bean with name 'org.springframework.jdbc.datasource.init.DataSour ceInitializer#0':
Invocation of init method failed; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException:
Query was empty
Any idea what went wrong?
Also, I am setting up JobStoreTX for now - just to make sure I can get it working,
but eventually I would like to use the JobStoreCMT toplug intoSpring's transaction management.
I could not find any good documentation/examples on how to do that, especially if I don't want to use
Spring's wrappers for schedulers and hardcode all trigger definitions...
Thanks!
java.lang.IllegalStateException: Failed to load ApplicationContext
at org.springframework.test.context.TestContext.getApplicationContext(TestContext.java:308)
at org.springframework.test.context.support.DependencyInjectionTestExecutionListener.injectDeendencies (DependencyInjectionTestExecutionListener.java:109 )
at org.springframework.test.context.support.Dependenc yInjectionTestExecutionListener.prepareTestInstanc e(DependencyInjectionTestExecutionListener.java:75 )
at org.springframework.test.context.TestContextManage r.prepareTestInstance(TestContextManager.java:333)
at org.springframework.test.context.junit4.SpringJUni t4ClassRunner.createTest(SpringJUnit4ClassRunner.j ava:220)
at
.....
Caused by: org.springframework.beans.factory.BeanCreationExce ption: Error creating bean with name 'org.springframework.jdbc.datasource.init.DataSour ceInitializer#0': Invocation of init method failed; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorEx ception: Query was empty
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.initializeBean(Abstract AutowireCapableBeanFactory.java:1401)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.doCreateBean(AbstractAu towireCapableBeanFactory.java:512)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.createBean(AbstractAuto wireCapableBeanFactory.java:450)
at org.springframework.beans.factory.support.Abstract BeanFactory$1.getObject(AbstractBeanFactory.java:2 90)
at org.springframework.beans.factory.support.DefaultS ingletonBeanRegistry.getSingleton(DefaultSingleton BeanRegistry.java:222)
at org.springframework.beans.factory.support.Abstract BeanFactory.doGetBean(AbstractBeanFactory.java:287 )
at org.springframework.beans.factory.support.Abstract BeanFactory.getBean(AbstractBeanFactory.java:189)
at org.springframework.beans.factory.support.DefaultL istableBeanFactory.preInstantiateSingletons(Defaul tListableBeanFactory.java:557)
at org.springframework.context.support.AbstractApplic ationContext.finishBeanFactoryInitialization(Abstr actApplicationContext.java:842)
at org.springframework.context.support.AbstractApplic ationContext.refresh(AbstractApplicationContext.ja va:416)
at org.springframework.test.context.support.AbstractG enericContextLoader.loadContext(AbstractGenericCon textLoader.java:84)
at org.springframework.test.context.support.AbstractG enericContextLoader.loadContext(AbstractGenericCon textLoader.java:1)
at org.springframework.test.context.TestContext.loadA pplicationContext(TestContext.java:280)
at org.springframework.test.context.TestContext.getAp plicationContext(TestContext.java:304)
... 25 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorEx ception: Query was empty
at sun.reflect.NativeConstructorAccessorImpl.newInsta nce0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInsta nce(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newI nstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:40 6)
at com.mysql.jdbc.Util.getInstance(Util.java:381)
at com.mysql.jdbc.SQLError.createSQLException(SQLErro r.java:1030)
at com.mysql.jdbc.SQLError.createSQLException(SQLErro r.java:956)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.ja va:3491)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.ja va:3423)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:19 36)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java :2060)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionIm pl.java:2536)
at com.mysql.jdbc.StatementImpl.executeUpdate(Stateme ntImpl.java:1564)
at com.mysql.jdbc.StatementImpl.executeUpdate(Stateme ntImpl.java:1485)
at org.apache.commons.dbcp.DelegatingStatement.execut eUpdate(DelegatingStatement.java:228)
at org.springframework.jdbc.datasource.init.ResourceD atabasePopulator.executeSqlScript(ResourceDatabase Populator.java:157)
at org.springframework.jdbc.datasource.init.ResourceD atabasePopulator.populate(ResourceDatabasePopulato r.java:110)
at org.springframework.jdbc.datasource.init.DataSourc eInitializer.afterPropertiesSet(DataSourceInitiali zer.java:69)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.invokeInitMethods(Abstr actAutowireCapableBeanFactory.java:1460)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.initializeBean(Abstract AutowireCapableBeanFactory.java:1398)
...
I recently came across this issue too, (I was using Java7 under Mac OS X), the following worked for me:
first of all, test if the method java.net.InetAddress.getLocalHost() works, if it throws exception like:
Caused by: java.net.UnknownHostException: leo-mbp: nodename nor servname provided, or not known
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1293)
at java.net.InetAddress.getLocalHost(InetAddress.java:1469)
run the following to workaround it:
scutil --set HostName "localhost"
Hope it's helpful to you.
#blob : you were right on target. After spending hours trying to figure out why the NotificationInitializer was not initialized (as the error message suggested....) I finally tried to remove the Quartz DB initialization script from my Spring config - and everything worked like a charm!
<jdbc:initialize-database data-source="dataSource"
enabled="${database.initschema}">
<jdbc:script location="${database.schemaLoc}" />
<jdbc:script location="${database.seedLoc}" />
<!-- <jdbc:script location="classpath:db/tables_mysql_innodb_1.8.5.sql" /> -->
</jdbc:initialize-database>
I just wish Spring error messages were a bit more specific... :)
Now, I still don't understand why I cannot create the DB tables for Quartz this way - the same scripts works just fine when run directly on the DB. Maybe some weird sequence of startup actions in Spring?
thanks!
Only a guess: Check if you need to specify a connection validation query. Something like
validationQuery="select 1"
Changing "org.quartz.scheduler.instanceId" from AUTO to 10 in quartz.properties file worked for me.