Is there a way to control database connection release strategy in Spring Boot? Something like Hibernate's ConnectionReleaseMode?
My test code would be rougly following:
INSERT query
HTTP call
UPDATE query
INSERT and UPDATE queries are their own methods in a repository bean (either extending CrudRepository, or as a Mybatis #Mapper). HTTP call is in it's own bean.
Now my service bean is where I experiment with different #Transactional settings as my ultimate goal is to have these three steps executed within a single transaction.
The problem is, the HTTP call can take hundreds of millis and Spring is holding the database connection during that time. This quickly leads to an empty connection pool, while the connections themselves are idle.
I have done the same experiments with default configuration using spring-boot-starter-data-jpa and also using mybatis-spring-boot-starter.
The only thing which got me closer to my goal was setting spring.jpa.open-in-view=false for data-jpa, which would release db connections in case of not using #Transactional at all or with propagation set to NEVER. It doesn't work if it's all wrapped in a transactin though.
I'm feeling like I'm missing some vital part of the transaction concept in Spring. Though the Spring reference docs mentions release mode only in relation to JTA transaction manager + JEE container + Hibernate.
Transactions are bound to a connection/session so you can't release the connection in the middle of a transaction. You have to finish the transaction to close/release the connection. Keep in mind that a transaction could abort at any time, and since the HTTP call is not "part of" the transaction, but just runs while the transaction is in progress, it won't help you to keep the transaction open while the HTTP call is running. I'd suggest you go with one of the following solutions:
Use a circuit breaker to cancel the HTTP call if it hits a timeout to have an upper bound for how long a connection/transaction can be open/held
Move the HTTP call out of the transaction, either before or after the transaction
Use two database transactions
Note that you could use a transactional Job Scheduler to schedule a job in the first TX. The job could then, with at least once semantics, try to invoke the HTTP call and then continue with the second transaction or do some compensation if something fails.
Related
My route looks like below
from("jms:queue:IN_QUEUE) //(A) Transactional Endpoint
.transacted("required") //(B) TX Policy with PROPAGATION_REQUIRED and JPATxManager
.bean("someBean", "readFromDB()") //(C) Read from DB
.bean("someBean", "writeToDB()") //(D) Write to DB
.to("file:/home/src?fileName=demo_${id}.txt")
I know the JMS consumer at (A) will fork out JMS Transaction on each poll and attaches to the
thread.
Also the transacted node in (B) will fork out JPA transaction after an exchange reaches there
and attaches to the thread.
Please find my questions below:
Can two different transactions get attached to a single thread (like the one above) ?
If Yes, which one should get suspended ?
What should be the commit and rollback order of the above mentioned route ?
Note: I didn't find any obvious answer from Camel In Action 2nd Ed book, So please guide me
Good afternoon,
This is a variation on your other question.
The:
from("jms:queue:IN_QUEUE) //(A) Transactional Endpoint
endpoint is transacted, meaning that you have marked the JMS component as transacted and the JMS sessions will be managed by a JmsTransactionManager.
.transacted("required") //(B) TX Policy with PROPAGATION_REQUIRED and
JPATxManager
This should not be a JPA transaction manager, but a JTA transaction manager (like Arjuna). As in your other question, you now have a JMS local transaction for reading you message, and local JPA transacted sessions for your DB access. You want the PlatformTransactionManager (a JTA transaction manager) to synchronize the local transactions for you.
As to your questions:
Can two different transactions get attached to a single thread (like the one above) ?
That really doesn't make any sense.
If Yes, which one should get suspended ?
Nothing will get suspended.
What should be the commit and rollback order of the above mentioned route ?
The DB read is not transactional and does not need to be committed. The file write will actually happen as the JTA transactional context is closed. This leaves the DB write. If that fails, then the DB read does not matter, the message will get put back on the source destination and the file write will not be called.
Enabling the DEBUG logging for the various transaction managers is very helpful.
I could go on about this with this in painful detail. This is more for burki. I think that you will really appreciate this. Very subtle, and it happens a lot.
from("jms:queue:SRC_QUEUE")
.transacted("required")
.to("jms1:queue:DEST_QUEUE")
What happens if the two endpoints are marked as transacted ... but... you do not have the 'transacted' line? Well, a JMS local transaction was started on the message listener. This will be committed as the route ends. There are two independent local JMS transactions. These are not synchronized by a JTA transaction manager.
What actually happens is that the commit for the message 'get' is called. There is no actual commit for the message 'put'. The message 'put' is committed when the JMS session is closed. This is in the JMS spec, that closing the connection inherently commits any transaction. So, because there is no linkage between the two components, the 'get' is committed, and then the 'put' session is closed.
This means that you can lose messages if there is an outage between the commit for the message 'get' and the session close for the message 'put'.
Does that make sense? There is no linkage between the local transactions, so Camel closes them in order, starting with committing the 'get' before calling the 'put'.
JTA transaction synchronization is the key. You still have local transaction resources (not XA), but they can be very well managed in a pretty lightweight JTA transactional context.
from("jms:queue:SRC_QUEUE")
.transacted("required")
.to("DB:transactedwrite")
.to("jms1:queue:DEST_QUEUE")
I couldn't be bothered to look up the correct syntax for a database insert, but you get the idea. In this case, you can get duplicate DB inserts if the JMS 'put' fails. This is not 'all or nothing' XA transactions. The transactions are committed in order. If one in the middle succeeds, then the next transaction fails, well the 'get' will get rolled back and you will get duplicates up to the point of failure.
Sorry, I can't answer your specific questions, but I can give some specific infos about the transactions of your route.
You've got 3 different "systems" with different transaction "scopes"
A JMS broker from which you consume
A database you read and write from and you configured a JPA TxManager for
A file system (no transactions at all) as destination
First of all, if you want to have transaction safety across JMS and the database you have to use XA transactions.
Then, it is unclear if you expect the JMS consumer to be transacted (because of the transacted() in your route) or if you really configured a JMS connection with local JMS transactions. I assume that you really consume transacted.
Let's talk about what you got without line B of your route:
You consume transacted from the broker
Camel processes the message through your route
When any error occurs during route processing, the message is not committed on the broker and therefore redelivered to your route
The transaction that is opened by a transactional consumer is kept open by Camel until the route is successfully processed.
So the only obvious problem would be an error after the database write, that triggers a redelivery and the database write is done once again. Probably the write is not idempotent and therefore must not happen twice.
So to solve these problems, you either have to use XA transactions or you simply use local JMS transactions and implement compensation logic for the "gaps" like the one described above.
The database transaction on the other hand has no benefit unless the read and write operation must be done in a transaction (but I have doubts that this is the case with two individual bean calls and a JMS consumer).
When a spring boot request starts, it obtains a connection from the pool. My question is - is this connection remains tied to the request thread (even if it is not executing any query) and only returned to the pool when request completes?
For example, if I'm doing something like:
Request starts
Execute a query (20ms)
Call external http service (1500ms)
Request completes
Would the connection obtained by this request thread remain occupied with the thread (and not available to other requests) for 20ms or 1520ms?
PS: I'm using Spring Boot 2.0 with HikariCP and I'm not using #Transactional.
Thanks.
If you have Spring MVC on your classpath, Spring will configure an instance of OpenEntityManagerInViewInterceptor automatically (in org.springframework.boot.autoconfigure.orm.jpa.JpaBaseConfiguration.JpaWebConfiguration), unless you configure spring.jpa.open-in-view to false. This interceptor will open an instance of EntityManager and bind it to the TransactionSynchronizationManager for the whole duration of the request.
What that means is that at the first use of a repository query, it will borrow a connection from the pool and keep it until the end of the request when the EntityManager is closed by the interceptor. It's the EntityManager's responsibility to return the connection to the pool. So, in that case, the connection will be borrowed for 1520ms in your example.
If you disable the spring.jpa.open-in-view option, a connection will be borrowed and returned at each use of a repository query given you don't use any explicit transaction. While this may look better, you have to keep in mind that managed instances of your entities will need to be reattached systematically on each persist operation so it may be costly. You will also loose the benefits of the EntityManager caching. To avoid this, use transactions to read/modify/persist your entities and avoid reattaching.
Finally, keep in mind that if you disable spring.jpa.open-in-view, since you don't have an EntityManager around for the duration of the request, you will need to make sure your lazy loaded relations are loaded inside a transaction otherwise you'll get the dreaded LazyInitializationException.
Basically it depends,
If you close conection it will be release back to pool and will be ready soon according to configuration (below) so it will be ~20ms (+ time to get back to pool)
If you don't close the connection it will wait until the configuration allow it, if it allow indefinite time, theoretically it can be causing a leak in your application and not return to pool until apllication is shutdown.
See answer about Hikari handling returning connections to pool:
The Hikari housekeeper runs every 30s, which closes any connections that are not in use and are older than maxLifetime. If there are more than minimumIdle number of connections the housekeeper will close connections that have been idle for longer than idleTimeout.
See more about max life time of connection:
by default Oracle does not enforce a max lifetime for connections (neither on JDBC driver side (1), nor on server side(2)). So in this respect, the "infrastructure-imposed connection time limit" is +infinity
I have recently had to support a colleague in verifying why some system tests are not passing in wildfly, system tests that pass consistently on weblogic and glass fish.
After analysing the log, it became clear the reason is related to a JMS message sent by a backed thread getting committed to a queue too soon, when the expectation was the message would be committed when the entry point Container Managed Transaction of an MDB commits. So the message is going out before the MDB that sends it is done running.
In weblogic, to achieve the expected behaviour, you need to make sure that when you take the connection factory given by the container , which is XA configured, you set the connection.createseesion with
transacted = true and
acknowledgement = session transacted.
In a process similar to the one depicted in this URL
http://www.mastertheboss.com/jboss-server/jboss-jms/sending-jms-messages-over-xa-with-wildfly-jboss-as
Except in the snippet above auto acknowledge is set and the first parameter is set to false.
In wildly when our weblogic and glass fish configuration is used, nothing is committed and the system behaves as if the JMS message sent were to be rolled back.
If configuration as in the example above were to be used, instead what would happen is that the JMS message is immediately and the consumer MDB immediately launches being trigerred before the producer transaction actually ends, causing the system test to fail.
According to the official JMS configuration, by using a connection-pooled factory with the transaction=XA attribute, the container should immediately bind the commit of the transaction to the lifecycle of the parent transaction.
See official documentation bellow in particular in respect to the Java:/JmsXa connection factory.
https://docs.jboss.org/author/display/WFLY10/Messaging+configuration
My colleague was initially using a non pooled connection factory, but the injection info reference has since then been fixed. I have tried all possible combinations of parameters in the shed message, but my outcome is sitll:
Either sent too soon or never sent.
To conclude all the other resources are XA. Namely the oracle db is using the XA driver.
Can anyone confirm if in wildly the send JMS message only when parent transaction commits is working and if so how the session is being configured?
I will check if perhaps my colleague has not made a mistake in terms of the configuration of the connection factory used by the Men's themselves to consume messages out of the queue.but if that one is also XA... Then it is a big problem.
So the issues is fixed.
The commit of the JMS message to the queue at the end of the transaction works perfectly.
The issue was two fold:
(a) First spot of code I was looking at address the issue was not correct. Someone had decided to write his own send telegram to queue API elsewhere, and was not using the central API for sneding telegrams, so any modification I to the injection connection factory was actually not taking effect. The stale connection factories were still being used.
(b) Once the correct API was spotted it was easy to make the mechanism work by using the widlfy XA pooled connection factory mentioned in the post above.
The one thing that was tweaked was the connection.CreationSession api.
The API in JEE 7 has been enlarged and it is now better documented than in jEE 6.
To send a JMS message in a container as part of an XA transaction one should do:
connection.createSession() without any parameters.
This can easily be seen in the connection javadoc:
https://docs.oracle.com/javaee/7/api/javax/jms/Connection.html
QUOTE 1:
This method has been superseded by the method createSession(int
sessionMode) which specifies the same information using a single
argument, and by the method createSession() which is for use in a Java
EE JTA transaction. Applications should consider using those methods
instead of this one.
QUOTE 2:
In a Java EE web or EJB container, when there is an active JTA
transaction in progress:
Both arguments transacted and acknowledgeMode are ignored. The session will participate in the JTA transaction and will be committed
or rolled back when that transaction is committed or rolled back, not
by calling the session's commit or rollback methods. Since both
arguments are ignored, developers are recommended to use
createSession(), which has no arguments, instead of this method.
Which means, the code snippet in:
http://www.mastertheboss.com/jboss-server/jboss-jms/sending-jms-messages-over-xa-with-wildfly-jboss-as
Is not appropriate. What one should be doing is creating the session without any parameter and leting the container handle the rest.
Which it does just fine.
I have a very basic JTA question.
We are using Spring's AOP to apply WebLogicJtaTransactionManager pointcuts on any method in a service class...and we set the tx:method timeout="60".
What is interesting is within that service, we run a select statement to retrieve records from the database (using Hibernate/JPA) and then go do some work that is not database/transactional related. That method takes longer than 60 seconds to run and a timeout exception is never thrown and the transaction is never rolled back. We don't update or create new objects to persist.
To add to that, we have JTA configured to timeout after 180 seconds on WebLogic Server...and the application still doesn't timeout.
The WLS docs say "The transaction timeout in seconds. If the transaction is still in the "active" state after this time (counting from begin()), it is automatically rolled back. Once the transaction moves on to the prepared state, however, this timeout parameter does not apply; the transaction is retried until all the resources are committed."
Would this behavior be related to the fact that there is nothing for the JTA to commit and therefore the timeout doesn't necessarily apply?
Or does it related to the fact that because there is no real database (insert/update/deletes) so the transaction is never in a true active state?
The JTA transaction timeout would only apply to XA based datasources. Can you confirm that that the datasource is setup with an XA database driver ?
I am using a connection pool and the Spring TransactionTemplate. If you want to shutdown the connection pool first all connection have to be returned to the pool, this means connection.close() has to be called. I have one thread using the TransactionTemplate for some queries and another thread that wants to call some shutdown method on the connection pool, but before doing this it first has to tell the TransactionTemplate to close all connection (actually only returning them to the pool).
How can this be done in Spring to immediately call close on the used connection?
If you are using Hibernate along with Spring please use:
hibernate.connection.release_mode=after_transaction
If you want to release connection just after transaction.
hibernate.connection.release_mode=after_statement
If you want to release connection after each statement
This two settings are the only ways I know that will make used connection being released faster than the default behavior. At least as far as Hibernate is concerned. If you are using other library please describe which.