possible dead-lock with nested transactions in grails service - spring

Question

I have a transactional grails service which is doing some (long running) processing. While processing, I would like to update a "percentComplete" value (which will ultimately be used to display a progress bar on the front-end). This obviously has to be written immediately (i.e. not as part of the current transaction), otherwise it is of no value.
Hence, I'm using the Grails Transaction Handling Plugin and specifically the "withNewTransaction" method to start up a new transaction, within which the percentComplete attribute is updated.
My problem is that, second time around in to the "updatePercentComplete()" method, the application hangs when attempting to exit the "withNewTransaction" method (presumably when it is trying to commit the transaction).
Notes:
if I comment out the session flushing in the main service method, there is no problem (btw, this flushing, along with clearing, is required for performance reasons in my real application)
using the hsql database, there is no problem (see DataSources.groovy).
also using the grails executor plugin, but I don't think this is causing any problems in itself (see the LibraryServiceTests, which also hangs and bypasses the plugin).
Where should I go looking to debug this problem? Maybe there are some log levels I can set in grails/hibernate somewhere to pin-point what is happening?
The thread doing the work seems to be getting stuck trying to read from a socket, see stack trace below. There don't appear to be any actual deadlocks in the JVM itself.
Here is my service class, link to full project below:
package withtransactiontest
class LibraryService {
static transactional = true
def sessionFactory
def loadBooks(library) {
int repeat = 5
repeat.times {
updatePercentComplete(library, it * (100 / repeat))
// Simulate some long running process in order to be able to see percentComplete getting written to DB.
Thread.sleep(2000)
// Update some property (within the default transaction)
library.name += "x"
library.save()
// Comment out the following, and the current method will complete successfully.
def session = sessionFactory?.currentSession
session?.flush()
}
library.percentComplete = 100
library.save()
}
void updatePercentComplete(library, val) {
println "before new transaction"
Library.withNewTransaction {
println "Percent complete: " + val
library.refresh()
library.percentComplete = val
library.save()
println "after save"
// Hang here, second time around.
}
println "Percent complete transaction committed"
library.refresh()
}
}
Link to grails project with integration test case which re-creates the problem
Stacktrace of "worker" thread:
Thread [pool-1-thread-1] (Suspended)
SocketInputStream.socketRead0(FileDescriptor, byte[], int, int, int) line: not available [native method]
SocketInputStream.read(byte[], int, int) line: 129
ReadAheadInputStream.fill(int) line: 114
ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(byte[], int, int) line: 161
ReadAheadInputStream.read(byte[], int, int) line: 189
MysqlIO.readFully(InputStream, byte[], int, int) line: 2499
MysqlIO.reuseAndReadPacket(Buffer, int) line: 2952
MysqlIO.reuseAndReadPacket(Buffer) line: 2941
MysqlIO.checkErrorPacket(int) line: 3489
MysqlIO.sendCommand(int, String, Buffer, boolean, String, int) line: 1959
MysqlIO.sqlQueryDirect(StatementImpl, String, String, Buffer, int, int, int, boolean, String, Field[]) line: 2113
JDBC4Connection(ConnectionImpl).execSQL(StatementImpl, String, int, Buffer, int, int, boolean, String, Field[], boolean) line: 2568
JDBC4PreparedStatement(PreparedStatement).executeInternal(int, Buffer, boolean, boolean, Field[], boolean) line: 2113
JDBC4PreparedStatement(PreparedStatement).executeUpdate(byte[][], InputStream[], boolean[], int[], boolean[], boolean) line: 2409
JDBC4PreparedStatement(PreparedStatement).executeUpdate(boolean, boolean) line: 2327
JDBC4PreparedStatement(PreparedStatement).executeUpdate() line: 2312
DelegatingPreparedStatement.executeUpdate() line: 105
DelegatingPreparedStatement.executeUpdate() line: 105
GroovyAwareSingleTableEntityPersister(AbstractEntityPersister).update(Serializable, Object[], Object[], Object, boolean[], int, Object, Object, String, SessionImplementor) line: 2435
GroovyAwareSingleTableEntityPersister(AbstractEntityPersister).updateOrInsert(Serializable, Object[], Object[], Object, boolean[], int, Object, Object, String, SessionImplementor) line: 2335
GroovyAwareSingleTableEntityPersister(AbstractEntityPersister).update(Serializable, Object[], int[], boolean, Object[], Object, Object, Object, SessionImplementor) line: 2635
EntityUpdateAction.execute() line: 115
ActionQueue.execute(Executable) line: 279
ActionQueue.executeActions(List) line: 263
ActionQueue.executeActions() line: 168
PatchedDefaultFlushEventListener.performExecutions(EventSource) line: 46
PatchedDefaultFlushEventListener(DefaultFlushEventListener).onFlush(FlushEvent) line: 50
SessionImpl.flush() line: 1027
SessionImpl.managedFlush() line: 365
JDBCTransaction.commit() line: 137
GrailsHibernateTransactionManager(HibernateTransactionManager).doCommit(DefaultTransactionStatus) line: 656
GrailsHibernateTransactionManager(AbstractPlatformTransactionManager).processCommit(DefaultTransactionStatus) line: 754
GrailsHibernateTransactionManager(AbstractPlatformTransactionManager).commit(TransactionStatus) line: 723
TransactionTemplate.execute(TransactionCallback) line: 147
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
Method.invoke(Object, Object...) line: 597
PojoMetaMethodSite$PojoCachedMethodSite.invoke(Object, Object[]) line: 188
PojoMetaMethodSite$PojoCachedMethodSite(PojoMetaMethodSite).call(Object, Object[]) line: 52
PojoMetaMethodSite$PojoCachedMethodSite(AbstractCallSite).call(Object, Object) line: 124
GroovyDynamicMethods$_doWith_closure1.doCall(Map, Map, Closure) line: 55
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
Method.invoke(Object, Object...) line: 597
PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(Object, Object[]) line: 266
PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce(PogoMetaMethodSite).callCurrent(GroovyObject, Object[]) line: 51
PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce(AbstractCallSite).callCurrent(GroovyObject, Object, Object, Object) line: 157
GroovyDynamicMethods$_doWith_closure1.call(Map, Map, Closure) line: not available
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
Method.invoke(Object, Object...) line: 597
PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(Object, Object[]) line: 266
PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce(PogoMetaMethodSite).call(Object, Object[]) line: 63
PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce(AbstractCallSite).call(Object, Object, Object, Object) line: 132
GroovyDynamicMethods$_doWith_closure3.doCall(Map, Closure) line: 65
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
Method.invoke(Object, Object...) line: 597
PogoMetaMethodSite$PogoCachedMethodSite.invoke(Object, Object[]) line: 225
PogoMetaMethodSite$PogoCachedMethodSite(PogoMetaMethodSite).callCurrent(GroovyObject, Object[]) line: 51
PogoMetaMethodSite$PogoCachedMethodSite(AbstractCallSite).callCurrent(GroovyObject, Object, Object) line: 153
GroovyDynamicMethods$_doWith_closure3.doCall(Closure) line: not available
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
Method.invoke(Object, Object...) line: 597
CachedMethod.invoke(Object, Object[]) line: 90
CachedMethod(MetaMethod).doMethodInvoke(Object, Object[]) line: 233
ExpandoMetaClass(MetaClassImpl).invokeMethod(Class, Object, String, Object[], boolean, boolean) line: 1058
ExpandoMetaClass.invokeMethod(Class, Object, String, Object[], boolean, boolean) line: 1070
ExpandoMetaClass(MetaClassImpl).invokeMethod(Object, String, Object[]) line: 886
GroovyDynamicMethods$_doWith_closure3(Closure).call(Object[]) line: 282
ClosureStaticMetaMethod.invoke(Object, Object[]) line: 59
StaticMetaMethodSite$StaticMetaMethodSiteNoUnwrapNoCoerce.invoke(Object, Object[]) line: 148
StaticMetaMethodSite$StaticMetaMethodSiteNoUnwrapNoCoerce(StaticMetaMethodSite).call(Object, Object[]) line: 88
StaticMetaMethodSite$StaticMetaMethodSiteNoUnwrapNoCoerce(AbstractCallSite).call(Object, Object) line: 124
LibraryService.updatePercentComplete(Object, Object) line: 34
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
Method.invoke(Object, Object...) line: 597
CachedMethod.invoke(Object, Object[]) line: 90
CachedMethod(MetaMethod).doMethodInvoke(Object, Object[]) line: 233
ExpandoMetaClass(MetaClassImpl).invokeMethod(Class, Object, String, Object[], boolean, boolean) line: 1058
ExpandoMetaClass.invokeMethod(Class, Object, String, Object[], boolean, boolean) line: 1070
ExpandoMetaClass(MetaClassImpl).invokeMethod(Object, String, Object[]) line: 886
ExpandoMetaClass(MetaClassImpl).invokeMethod(Class, Object, String, Object[], boolean, boolean) line: 1003
ExpandoMetaClass.invokeMethod(Class, Object, String, Object[], boolean, boolean) line: 1070
ExpandoMetaClass(MetaClassImpl).invokeMethod(Object, String, Object[]) line: 886
PogoMetaClassSite.callCurrent(GroovyObject, Object[]) line: 66
PogoMetaClassSite(AbstractCallSite).callCurrent(GroovyObject, Object, Object) line: 153
LibraryService$_loadBooks_closure1.doCall(Object) line: 13
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
Method.invoke(Object, Object...) line: 597
CachedMethod.invoke(Object, Object[]) line: 90
CachedMethod(MetaMethod).doMethodInvoke(Object, Object[]) line: 233
ExpandoMetaClass(MetaClassImpl).invokeMethod(Class, Object, String, Object[], boolean, boolean) line: 1058
ExpandoMetaClass.invokeMethod(Class, Object, String, Object[], boolean, boolean) line: 1070
ExpandoMetaClass(MetaClassImpl).invokeMethod(Object, String, Object[]) line: 886
LibraryService$_loadBooks_closure1(Closure).call(Object[]) line: 282
LibraryService$_loadBooks_closure1(Closure).call(Object) line: 295
DefaultGroovyMethods.times(Number, Closure) line: 9487
dgm$630.invoke(Object, Object[]) line: not available
PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(Object, Object[]) line: 270
PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce(PojoMetaMethodSite).call(Object, Object[]) line: 52
PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce(AbstractCallSite).call(Object, Object) line: 124
LibraryService.loadBooks(Object) line: 12
LibraryService$$FastClassByCGLIB$$f97d911e.invoke(int, Object, Object[]) line: not available
MethodProxy.invoke(Object, Object[]) line: 149
Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint() line: 688
Cglib2AopProxy$CglibMethodInvocation(ReflectiveMethodInvocation).proceed() line: 150
TransactionInterceptor.invoke(MethodInvocation) line: 110
Cglib2AopProxy$CglibMethodInvocation(ReflectiveMethodInvocation).proceed() line: 172
Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Object, Method, Object[], MethodProxy) line: 621
LibraryService$$EnhancerByCGLIB$$c098c6ba.loadBooks(Object) line: not available
LibraryService$loadBooks.call(Object, Object) line: not available
LibraryController$_closure4_closure9.doCall(Object) line: 30
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
Method.invoke(Object, Object...) line: 597
PogoMetaMethodSite$PogoCachedMethodSite.invoke(Object, Object[]) line: 225
PogoMetaMethodSite$PogoCachedMethodSite(PogoMetaMethodSite).callCurrent(GroovyObject, Object[]) line: 51
PogoMetaMethodSite$PogoCachedMethodSite(AbstractCallSite).callCurrent(GroovyObject, Object) line: 149
LibraryController$_closure4_closure9.doCall() line: not available
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
Method.invoke(Object, Object...) line: 597
CachedMethod.invoke(Object, Object[]) line: 90
CachedMethod(MetaMethod).doMethodInvoke(Object, Object[]) line: 233
ExpandoMetaClass(MetaClassImpl).invokeMethod(Class, Object, String, Object[], boolean, boolean) line: 1058
ExpandoMetaClass.invokeMethod(Class, Object, String, Object[], boolean, boolean) line: 1070
ExpandoMetaClass(MetaClassImpl).invokeMethod(Object, String, Object[]) line: 886
LibraryController$_closure4_closure9(Closure).call(Object[]) line: 282
ConvertedClosure.invokeCustom(Object, Method, Object[]) line: 51
ConvertedClosure(ConversionHandler).invoke(Object, Method, Object[]) line: 82
$Proxy6.call() line: not available
java_util_concurrent_Callable$call.call(Object) line: not available
PersistenceContextCallableWrapper$_call_closure1.doCall(Object) line: 36
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
Method.invoke(Object, Object...) line: 597
PogoMetaMethodSite$PogoCachedMethodSite.invoke(Object, Object[]) line: 225
PogoMetaMethodSite$PogoCachedMethodSite(PogoMetaMethodSite).callCurrent(GroovyObject, Object[]) line: 51
PogoMetaMethodSite$PogoCachedMethodSite(AbstractCallSite).callCurrent(GroovyObject, Object) line: 149
PersistenceContextCallableWrapper$_call_closure1.doCall() line: not available
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
Method.invoke(Object, Object...) line: 597
CachedMethod.invoke(Object, Object[]) line: 90
CachedMethod(MetaMethod).doMethodInvoke(Object, Object[]) line: 233
ExpandoMetaClass(MetaClassImpl).invokeMethod(Class, Object, String, Object[], boolean, boolean) line: 1058
ExpandoMetaClass.invokeMethod(Class, Object, String, Object[], boolean, boolean) line: 1070
ExpandoMetaClass(MetaClassImpl).invokeMethod(Object, String, Object[]) line: 886
PersistenceContextCallableWrapper$_call_closure1(Closure).call(Object[]) line: 282
PersistenceContextCallableWrapper$_call_closure1(Closure).call() line: 277
Closure$call.call(Object) line: not available
PersistenceContextCallableWrapper(PersistenceContextWrapper).wrap(Closure) line: 35
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
Method.invoke(Object, Object...) line: 597
PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(Object, Object[]) line: 266
PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce(PogoMetaMethodSite).callCurrent(GroovyObject, Object[]) line: 51
PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce(AbstractCallSite).callCurrent(GroovyObject, Object) line: 149
PersistenceContextCallableWrapper.call() line: 36
FutureTask$Sync.innerRun() line: 303
FutureTask.run() line: 138
ThreadPoolExecutor$Worker.runTask(Runnable) line: 886
ThreadPoolExecutor$Worker.run() line: 908
Thread.run() line: 680

Answer 1

Workaround is to introduce a new domain class, Progress, which has the property percentComplete and belongsTo Library. This is updated in the new (nested) transaction, rather than the Library instance itself.