Spring Batch database Connection is not available, request timed out - spring

During the work with Spring Batch for a project, I ran into the following problem. The project consists of:
a MySQL database (mysqld 10.2.13-MariaDB)
Spring Boot (2.1.1.RELEASE)
Spring Batch (4.1.0.RELEASE)
Spring Batch is configured to use the same datasource as the business logic for the JobRepository
The hikari connection pool size for the datasource is configured to have a size of 4 (which is the default when pusing the app to our CloudFoundry instance and injected during the auto-reconfiguration of Spring)
As a reference you can take a look into the sample project where I created a minimal project to reproduce the problem:
https://github.com/FlorianSW/spring-batch-connection-issue
The problem:
Given you've a controller, which handles one RequestMapping in which at least two things happen: The controller does an arbitrary action agains the business model database schema (e.g. saving or requesting an entity from the database) and afterwards starting a Spring Batch job through a call to JobLauncher#run. Spring Batch is configured to run the tasks asynchronous with a ThreadPoolTaskExecutor with the pool size of 1.
This works pretty fine, if the request mapping is queries only few times per second (round about 1-3 times). However, if the mapping is queried more than that, let's say 4+ (during testing I used 4 and up to 20 requests) times in a synchronized way (if testing locally), then the requests run into a deadlock, where some of them will be aborted with the following exception:
Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.dao.DataAccessResourceFailureException: Could not obtain last_insert_id(); nested exception is java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 5001ms.]
This problem can be mitigated when increasing the size of the connection pool to at least 7 (I don't know where this number could come from). After restarting the application and executing my test JavaScript code[1] to dispatch a number of requests I can easily increase the number of fetch requests to 150 and the application will handle these requests without any problems (as expected). Where, before increasing the pool size, numerous requests run into an timeout.
I'm not sure, if this qualifies as a bug or as an Improvement or whatever, and I'm not sure if the component is the documentation or something like that, however, my intention of this issue is:
Finding out, if Spring Batch, together with Spring Boot, requires a minimum number of available connections in the DB connection pool? If so, should this be mentioned in the documentation?
Or, if this is a bug as the JDBC connections used during the request processing are not returned to the pool in a reasonable short amount of time?
[1]
for (i = 0; i <= 150; i++) {fetch('http://localhost:8080/api/jobs/' + i, {method: 'PUT'})}

Related

Spring Boot DB connection release mode

Is there a way to control database connection release strategy in Spring Boot? Something like Hibernate's ConnectionReleaseMode?
My test code would be rougly following:
INSERT query
HTTP call
UPDATE query
INSERT and UPDATE queries are their own methods in a repository bean (either extending CrudRepository, or as a Mybatis #Mapper). HTTP call is in it's own bean.
Now my service bean is where I experiment with different #Transactional settings as my ultimate goal is to have these three steps executed within a single transaction.
The problem is, the HTTP call can take hundreds of millis and Spring is holding the database connection during that time. This quickly leads to an empty connection pool, while the connections themselves are idle.
I have done the same experiments with default configuration using spring-boot-starter-data-jpa and also using mybatis-spring-boot-starter.
The only thing which got me closer to my goal was setting spring.jpa.open-in-view=false for data-jpa, which would release db connections in case of not using #Transactional at all or with propagation set to NEVER. It doesn't work if it's all wrapped in a transactin though.
I'm feeling like I'm missing some vital part of the transaction concept in Spring. Though the Spring reference docs mentions release mode only in relation to JTA transaction manager + JEE container + Hibernate.
Transactions are bound to a connection/session so you can't release the connection in the middle of a transaction. You have to finish the transaction to close/release the connection. Keep in mind that a transaction could abort at any time, and since the HTTP call is not "part of" the transaction, but just runs while the transaction is in progress, it won't help you to keep the transaction open while the HTTP call is running. I'd suggest you go with one of the following solutions:
Use a circuit breaker to cancel the HTTP call if it hits a timeout to have an upper bound for how long a connection/transaction can be open/held
Move the HTTP call out of the transaction, either before or after the transaction
Use two database transactions
Note that you could use a transactional Job Scheduler to schedule a job in the first TX. The job could then, with at least once semantics, try to invoke the HTTP call and then continue with the second transaction or do some compensation if something fails.

Tracing memory leak in Spring Azure qPID JMS code

Im trying to trace and identify root cause for memory leak in our very small and simple Spring Boot application.
It uses following:
- Spring Boot 2.2.4
- azure-servicebus-jms-spring-boot-starter 2.2.1
- MSSQL
Function:
The app only dispatches Azure ServiceBus queue and stores data and sends data to other destination.
It is a small app so it starts easily with 64 megs of memory, despite I give it up to 256 megs via Xmx option. Important note is the queue is being dispatched using Spring default transacted mode with dedicated JmsTransactionManager who is actually inner TM of ChainedTransactionManager along with dbTM and additional outbound JMS TM. Both JMS ConnectionFactory objects are created as CachingConnectionFactory.
Behavior:
Once the app is started it seems OK. There is no traffic so I can see in the log it is opening transactions and closing when checking the queue (jms:message-driven-channel-adapter).
However after some time when there is still no traffic, no single message was consumed the memory starts climbing as monitored via JVVM.
There is an error thrown:
--2020-04-24 11:17:01.443 - WARN 39892 --- [er.container-10] o.s.j.l.DefaultMessageListenerContainer : Setup of JMS message listener invoker failed for destination 'MY QUEUE NAME HERE' - trying to recover. Cause: Heuristic completion: outcome state is rolled back; nested exception is org.springframework.transaction.TransactionSystemException: Could not commit JMS transaction; nested exception is javax.jms.IllegalStateException: The Session was closed due to an unrecoverable error.
... and after several minutes it reaches MAX of the heap and since that moment it is failing on OutOfMemory error in the thread opening JMS connections.
--2020-04-24 11:20:04.564 - WARN 39892 --- [windows.net:-1]] i.n.u.concurrent.AbstractEventExecutor : A task raised an exception. Task: org.apache.qpid.jms.provider.amqp.AmqpProvider$$Lambda$871/0x000000080199f840#1ed8f2b9
-
java.lang.OutOfMemoryError: Java heap space
at java.base/java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:61)
at java.base/java.nio.ByteBuffer.allocate(ByteBuffer.java:348)
at org.apache.qpid.proton.engine.impl.ByteBufferUtils.newWriteableBuffer(ByteBufferUtils.java:99)
at org.apache.qpid.proton.engine.impl.TransportOutputAdaptor.init_buffers(TransportOutputAdaptor.java:108)
at org.apache.qpid.proton.engine.impl.TransportOutputAdaptor.pending(TransportOutputAdaptor.java:56)
at org.apache.qpid.proton.engine.impl.SaslImpl$SwitchingSaslTransportWrapper.pending(SaslImpl.java:842)
at org.apache.qpid.proton.engine.impl.HandshakeSniffingTransportWrapper.pending(HandshakeSniffingTransportWrapper.java:138)
at org.apache.qpid.proton.engine.impl.TransportImpl.pending(TransportImpl.java:1577)
at org.apache.qpid.proton.engine.impl.TransportImpl.getOutputBuffer(TransportImpl.java:1526)
at org.apache.qpid.jms.provider.amqp.AmqpProvider.pumpToProtonTransport(AmqpProvider.java:994)
at org.apache.qpid.jms.provider.amqp.AmqpProvider.pumpToProtonTransport(AmqpProvider.java:985)
at org.apache.qpid.jms.provider.amqp.AmqpProvider.lambda$close$3(AmqpProvider.java:351)
at org.apache.qpid.jms.provider.amqp.AmqpProvider$$Lambda$871/0x000000080199f840.run(Unknown Source)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:510)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:518)
at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base/java.lang.Thread.run(Thread.java:835)
HeapDumps:
I took couple of heap snapshots during this whole process and looked at what gets increased.
I can see suspicious amount of ConcurrentHashMap/String/Byte[] objects.
Has anyone some clue/hint what can be wrong in this setup and libs: Spring Boot, Apache qPid used under the hood of the Azure JMS dependency etc.? Many thanks.
Update #1
I have clear evidence that the problem is either in Spring or azure service bus starter library - not automatically qPid client used. I would say the library has the bug rather than Spring, just my guess. This is how the failing setup looks like:
There are two JMS destinations and one DB, each having its transaction manager
There is ChainedTransactionManager wrapping above three TMs.
Spring integration app which connects to Azure ServiceBus queue via jms:message-driven-channel-adapter and setting the transaction manager on this component (as created in point 2)
Start the app., no traffic on the queue is needed, after 10 minutes the app will crash due to OutOfMemoryError ... within those 10 minutes I watch log on debug level and only thing which is happening is opening and closing transactions using ChainedTransactionManager ... also as written in the comments another important condition is the third JMS TransactionManager ... with 2 TMs it works and is stable, with 3 it will crash ...
Additional research and steps taken identified the most likely root cause Spring CachingConnectionFactory class. Once I removed that and used only native types the problem went away and memory consumption profile is very different and healthy.
I have to say I created CachingConnectionFactory using standard constructor and didnt further configure the behavior. However these Spring defaults clearly lead to memory leak as per my experience.
In past I had memory leak with ActiveMq which had to be resolved by using CachingConnectionFactory and now I have memory leak with Azure ServiceBus when using CachingConnectionFactory .. strange :) In both cases I see that as bugs because memory management should be correct regardless caching involved or not.
Marking this as my answer.
Tested case: The problem occurs when receiving and sending message both with its own TM and both JMS connectionFactories are type CachedConnectionFactory. At the end I tested the app. with inbound connection factory of type CachedConnectionFactory and outbound just native type ... no memory leak as well.

spring boot connection pool with long running requests

When a spring boot request starts, it obtains a connection from the pool. My question is - is this connection remains tied to the request thread (even if it is not executing any query) and only returned to the pool when request completes?
For example, if I'm doing something like:
Request starts
Execute a query (20ms)
Call external http service (1500ms)
Request completes
Would the connection obtained by this request thread remain occupied with the thread (and not available to other requests) for 20ms or 1520ms?
PS: I'm using Spring Boot 2.0 with HikariCP and I'm not using #Transactional.
Thanks.
If you have Spring MVC on your classpath, Spring will configure an instance of OpenEntityManagerInViewInterceptor automatically (in org.springframework.boot.autoconfigure.orm.jpa.JpaBaseConfiguration.JpaWebConfiguration), unless you configure spring.jpa.open-in-view to false. This interceptor will open an instance of EntityManager and bind it to the TransactionSynchronizationManager for the whole duration of the request.
What that means is that at the first use of a repository query, it will borrow a connection from the pool and keep it until the end of the request when the EntityManager is closed by the interceptor. It's the EntityManager's responsibility to return the connection to the pool. So, in that case, the connection will be borrowed for 1520ms in your example.
If you disable the spring.jpa.open-in-view option, a connection will be borrowed and returned at each use of a repository query given you don't use any explicit transaction. While this may look better, you have to keep in mind that managed instances of your entities will need to be reattached systematically on each persist operation so it may be costly. You will also loose the benefits of the EntityManager caching. To avoid this, use transactions to read/modify/persist your entities and avoid reattaching.
Finally, keep in mind that if you disable spring.jpa.open-in-view, since you don't have an EntityManager around for the duration of the request, you will need to make sure your lazy loaded relations are loaded inside a transaction otherwise you'll get the dreaded LazyInitializationException.
Basically it depends,
If you close conection it will be release back to pool and will be ready soon according to configuration (below) so it will be ~20ms (+ time to get back to pool)
If you don't close the connection it will wait until the configuration allow it, if it allow indefinite time, theoretically it can be causing a leak in your application and not return to pool until apllication is shutdown.
See answer about Hikari handling returning connections to pool:
The Hikari housekeeper runs every 30s, which closes any connections that are not in use and are older than maxLifetime. If there are more than minimumIdle number of connections the housekeeper will close connections that have been idle for longer than idleTimeout.
See more about max life time of connection:
by default Oracle does not enforce a max lifetime for connections (neither on JDBC driver side (1), nor on server side(2)). So in this respect, the "infrastructure-imposed connection time limit" is +infinity

Spring Application Context Initialization

In my Spring Application, at the time of context Initialization. The DB is not available, it will come up after some time (Due to DB redundancy). In this scenario, my spring application initialization should be delayed or the application should do retry for DB connectivity. How to achieve the same via Spring.
Srirama.
I'd suggest investigating the ApplicationContextInitializer. It is meant to be used for setting up your context before most of the magic of spring initialization happens.
I'm not sure if it is designed for your use case, but no beans are initialized when the initialize method is called during startup.
The example provided in the link deals with properties, but I see no reason why you should not create your own manually created connection and wait for it's readiness.
Thanks for the reply.
My application is initialized by means of C3P0. Here C3P0, is trying to reconnect to DB only for 30 times (the default configuration for acquireRetryAttempts) and after that it is saying failed to create the application beans.
Changed the configuration for acquireRetryAttempts to -1, so that C3P0 is retrying indefinitely till the DB connection is successful. Basically my app. initialization should be delayed till the DB comes up.
Srirama.

Can I use separate non connection pool data source for long running but infrequent tasks?

My application stack consists of Spring MVC, Hibernate and MySQL hosted on Apache tomcat 7.
I have set up Spring to manage transactions and Hibernate session factory is utilizing the tomcat dbcp connection pool backed datasource for getting the connection.
I have a use case in my application in which I have a run a long running task which is initiated through the web UI (say a button click). This task runs for let’s say 10 minutes then my connection pool starts to throw connection closed exceptions. This is obviously because of connection pool setting in which if the connection is not returned to pool after a specific time, it is marked as abandoned and later removed. I could solve this by tinkering with the timeout settings and increasing it to a large enough value. But I may have several other use cases like this and may not currently have idea how long those will run.
So I am thinking of another approach here.
This use case will be initiated not very often, so I may use a separate datasource definition without using connection pool. Of course I can set two transaction managers in Spring with different names “abc” and “xyz” and use the #Transactional(name=”abc”) and #Transactional(name=”xyz)”. Both these transaction managers would use their respective datasources – one with connection pool to support common use cases and one without connection pool to support long running transaction. This way I won’t have to worry about changing the timeout configurations.
Will this be a generally accepted solution or should I take the timeout configuration approach?
Avoiding to use the connection pool will cause problems if you don't have another way to limit the number of connections that your application can initiate. For example (trivial example of cours) if your going to launch your batch process each time a user clicks a button, make sure you limit the times they can do this task.
Another way would be to define a new jdbc resource in your application server (jdbc/batchprocess) and configure in this resource a longer timeout. Then change from one to another using dynamic datasource routing.
You can open Hibernate Sessions, supplying your own Connection:
sessionFactory.withOptions().connection( yourConnection ).openSession();

Resources