I was reviewing a code which seems to lose connection to DB approximately once a month intermittently. It leaves no logs on DB side but server prints a log of Connection closed by peer.
Its a Spring Boot App (2.1.9) which uses Spring Integration to read from queues (JMS).
Under pom.xml spring-boot-started-jdbc , Hikari is excluded and tomcat is used.
The app is designed in such a way that it has an in-queue to get traffic, then series of internal queues to split the workload and finally an out queue to the destination.
This app has 4 datasources created all pointing to the same oracle database.
First datasource is used for all CRUD operations in the flow between in-queue and first internal queue. Second datasource is used for all CRUD operations in the flow between first and second internal queue. Third datasource is used between second internal queue and destination queue. Fourth datasource is not used anywhere which suggests some flow was commented but datasource was never removed.
Each datasource has different min and max connections but rest of the properties are same.
Wanted to understand is that a good practice to create
datasources for each spring integration flow pointing to same database?
Can it cause any problems to leave a datasource initialized without using it anywhere?
Is there any obvious reason why HikariCP shouldn't be chosen for production when lot of matrix online show Hikari is superior to Tomcat and Spring boot 2 has chosen it as default for a reason?
Related
In a Spring Boot application using Log4j2, I have setup a DataSource via yaml-configuration. But since Log4j2 needs a connection pool already in the startup phase for its JDBC Appender, a second data source must be provided separately. This results in the application having two connection pools running, both writing to the same database.
My first question is:
Are there any concerns about this setup, or is this the normal approach anyway?
Alternatively, the following setup would come to mind: we programmatically initialize the DataSource, so that we have access to it. Further on, we can then programmatically extend the existing Log4j2 configuration that doesn't have a JDBC Appender yet. By doing so we can pass the pooled DataSource that we also use for the application itself.
My second question is therefore:
Are there any caveats to using the same DataSource for both the application and the Log4j2 logging framework?
Many thanks for any information that helps to clarify the situation.
TLDR:
1.Where does atomikos keep its trasnaction records so it can function and how is it best to "secure" them against losing/corrupting them. I use Docker/Docker-Compose and Spring Boot.
2.If the app dies unexpectdly, do all the transactions roll back because atomikos automagically started the appropiate SQL/JMS transactions with the sources?
Details
Hello.
I am starting to use atomikos in Spring Boot to enable XA transactions (in my case simply one datasource and an ActiveMQ/JMS broker.
I have been Googling and can't find a more "in depth" blog or site that explains in a bit more details how Atomikos works under the hood.
I do see the atomikos transaction logs in the form:
{"id":"127.0.1.1.tm159765570032300035","wasCommitted":true,"participants":[{"uri":"127.0.1.1.tm35","state":"COMMITTING","expires":1597655710327,"resourceName":"dataSource"},{"uri":"127.0.1.1.tm36","state":"COMMITTING","expires":1597655710327,"resourceName":"jmsConnectionFactory"}]}
I can only assume these are simply actual "logs" and not like the kafka.db file in say ActiveMQ when using file persistence.
Questions
So does atomikos keep its state in memory?
If say an unexpected shutdown occurred half way through an XA transaction, will all the JTA sources simply rollback?
If the atomikos state is held in a file/database somewhere...are there some best practices when using docker-compose for example?
Example:
A #Component picks out a database entity "REQUIRES_PROCESSING" for queuing on to a JMS queue and then switches the entity to "PROCESSING".
The JMS message is queued but then the App unexpectedly dies before the database entity is set to "PROCESSING".
Will in this case, atomikos has now no state (it died with the app) but it created a SQL and ActiveMQ trasnaction that will simply timeout and everything rolls back?
The application reboots 5 seconds later...what happens here? Is this transaction picked up again somehow (i.e. atomikos has still got state somewhere?).
Will the rebooted app just grab the rolledback entity and simply retry as expected?
Are there any pitfalls here?
Thank you for any claification.
I am having multiple spring boot applications which are connected to kafka (clustrized with 3 brokers)and also i integrated transaction synchronization (chainedKafkaTransactionManager). so i want to know should i give the same TransactionIdPrefix value in kafka config for all the multiple application or diffrent one.
i tried giving a random generated TransactionIdPrefix for each application. but i think in some time in multi thread environment in Listeners method it will take old data from database (jpa repositories)
is it problem because of diffrent TransactionIdPrefix ?
It depends; if they are multiple instances of the same app and the transactions are started by consumers, the prefix must be the same, so that zombie fencing is handled properly when partitions move from one instance to another after a rebalance.
If the transactions are started by producers, the prefix must be unique in each instance.
If they are different applications they should have different prefixes, regardless of what starts the transaction.
I have been researching Spring Data Rest especially for cassandra and one of the questions my coworkers and I had was when does Spring Data connect to the database. We don't always want a rest controller to connect to the database so when does spring establish a connection if say we had a class extend the CRUDRepository? Does it connect to the database during the start of application itself? Is that something we can control?
For example, I implemented this example on Spring's website:
https://spring.io/guides/gs/accessing-data-rest/
At what point in the code does spring connect to the database?
Spring will connect to the DB as soon as the Datasource get initialized. Basically, Spring contexts will become alive somehow (Web listeners, manually calling them) and start creating beans. As soon as it reaches the Datasource, connection will be made and the connection pool will be populated.
Of course the above is based on a normal out of the box configuration and everything can be setup up to your taste.
So unless, you decide to control the connections yourself, DB connections will be sitting there waiting to be used.
Disagree with the above answer.
As part of research i initiated the datasource using a bean configuration and then changed my database password(not in my spring application but the real db username password)
The connection stays for a while and then in some point of time (maybe idle time) it stops working and throws credential exception.
This is enough to say the JPA does not keep the connection sitting and waiting to be used but uses some mechanism to occupy/release the db connection as per the need.
To process a large number of messages coming to a queue i need guarantee of at least one jms connection to be there at any time. I am using spring and spring allows to have multiple sessions on a single connection only. In case one and only connection fails, application will come to standstill till spring reconnects to the JMS bridge.
So how can i create more than one connection to a queue in Spring, also how can i do connection pooling here.
The answer to this depends on whether you are using Spring inside a J2EE container(jboss etc.) or in a standalone application.
Standalone - you'll find pooling connections to be a problem. Springs SingleConnectionFactory can be setup to renew the connection on an exception garaunteeing that at some point a connection will come online and start processing the queue again, but you'll still have the problem of waiting for that single connection to renew, plus depending on what messaging implementation your dealing with and how it does load balancing you may find yourself stuck with a connection to a single node in a cluster.
If you are running in a container you can rely on the containers connection factory which will be much more robust. JBoss Messaging in the container for instance will failover seamlessly to other nodes and handles pooling under the covers, but if your working in the container its usually easier to bail on JMS template which kind of sucks and use whatever that container provides.