My use case is to process 50 csv files and load the same to MySQL DB using spring xd filejdbc job module.
For this I have configured datasource connection pool in spring-xd-1.0.0.RELEASE\xd\config\servers.yml like:
spring:
datasource:
url: jdbc:mysql://localhost:3306/spring_xd
username: springxd
password: springxd
driverClassName: com.mysql.jdbc.Driver
maxActive: 75
maxIdle: 10
minIdle: 10
initialSize: 10
maxWait: 30000
validationQuery: select 1
validationInterval: 30000
testOnBorrow: true
testOnReturn: false
testWhileIdle: false
timeBetweenEvictionRunsMillis: 10000
minEvictableIdleTimeMillis: 60000
removeAbandoned: true
removeAbandonedTimeout: 300
logAbandoned: true
When singlenode starts spring xd is initializing this pool.
But the problem is that, when the filejdbc job launches its once again initializing a connection pool with the above configs. Since \spring-xd-1.0.0.RELEASE\xd\config\modules\job\filejdbc\filejdbc.properties is using the same connection pool configs from servers.yml
The csv files data are loaded to the DB only using the filejdbc connectionpool and not the spring xd.
Because of this my db connections are running out.
Now, when I reduce the connection in servers.yml to max active 20, and when a job launches, spring xd is throwing connection pool exhaust exception, since i have 50 csv files to load.
I simply can't configure the servers.xml max active to 10 for the spring xd database and filejdbc maxactive to 60 for my custom database, so that 50 files can be loaded when the filejdbc job launches. This always throws connection exhaust exception.
Please advise.
Details as below:
spring xd version - spring-xd-1.0.0.RELEASE in singlenode
java version - jdk 7
environment - windows
There is a bug in the local message bus where all partitions get executed simultaneously. In your case that causes the connection pool to get exhausted. We have opened a bug for this - https://jira.spring.io/browse/XD-2868.
Until this is fixed you could run with an another message bus like Redis or RabbitMQ and you should see your job complete. You can try by starting a Redis server and then start your singlenode with:
xd-singlenode --transport redis
Related
I connect my DB (AWS RDS) to Spring boot JPA, Then my number of connections increases dramatically.
it is 12 now, I think it works spring boot 5 + browser 5, workbench 1 +, and others?
How can I reduce my number of connections? How can I maintain this connection safely?
You should be looking for Database connection pooling.
Database connection pooling is a method used to keep database connections open so they can be reused and also it will be keeping the total number of connections within a limit that we are specifying.
The default connection pool in Spring Boot is HikariCP, all you have to do is configure it properly
Sample connection pool configuration,
spring.datasource.hikari.connection-timeout = 20000
spring.datasource.hikari.minimum-idle= 10
spring.datasource.hikari.maximum-pool-size= 10
spring.datasource.hikari.idle-timeout=10000
spring.datasource.hikari.max-lifetime= 1000
spring.datasource.hikari.auto-commit =true
The application i'm working on is in Spring Boot using Spring JDBCTemplate to connect to Teradata.
We face issues with Idle connections. we have about 6 different environments that create at some point 1672 sessions.
In order to limit the total pool size and the minimum idle connections i set it to:
hikari:
maximum-pool-size: 3
minimum-idle: 2
is there anything else recommend in otder to prevent the number of idle connections?
Thanks in advance
Assuming you're using Hikari for connection pool, you may use idleTimeout. Please refer Hikari configuration for all available properties.
I am creating microservices applications using spring boot. All these microservices will connect to a single database AWS Aurora Serverless. I am planning to configure the connection pooling in my applications using HikariCP. Based on my initial research, Aurora Serverless manages connection pooling automatically. My question are the following.
Do I still need to configure connection pooling in my microservices using HikariCP? If yes, what is the recommended configuration in spring boot application.yml considering these are microservices that would share the same database?
If connection pooling configuration is not needed in my microservices, should i disable it? And, how should I do it my application.yml.
I have initial hikaricp configuration as shown in my code below
Springboot version is 1.5.7.RELEASE
application.yml
spring:
datasource:
type: com.zaxxer.hikari.HikariDataSource
hikari:
connection-timeout: 30000
minimum-idle: 30
maximum-pool-size: 200
idle-timeout: 30000
... database details (url, password, etc...)
A connection pool is still useful since there will be a cost to build a connection from your app to their connection pool.
What is different is that when you don't have any activity during a significant amount of time (night/weekend/periods of no activity) or when speed is not a necessity you should put the minimum-idle parameter to zero.
This will make sure the number of connections goes down to zero and enable your aurora serverless db to go into pause.
When it is paused you pay only for storage costs.
We recently upgraded our application to Spring Boot 2 (2.0.3.RELEASE). Along with that Spring Data Redis was upgraded was upgraded to 2.0.8.RELEASE. We switched over from using Jedis to Lettuce as the boot upgrade docs say that is now the default.
I have the lettuce connection pool defined as:
spring:
redis:
ssl: true
lettuce:
pool:
max-active: 250
max-idle: 250
min-idle: 50
max-wait: 2000ms
shutdown-timeout: 100ms
I noticed today on one of the dev servers, we are getting a lot of thread blocking while doing Redis read/writes. The path looks like this:
io.lettuce.core.internal.AbstractInvocationHandler:invoke:80
|
|__io.lettuce.core.FutureSyncInvocationHandler:handleInvocation:62
|
|__io.lettuce.core.LettuceFutures:awaitOrCancel:112
|
|__io.lettuce.core.protocol.AsyncCommand:await:81
|
|__java.util.concurrent.locks.LockSupport:parkNanos:215
Sometimes we see the thread waiting for 2 seconds and some times we see it for 19 seconds.
I started looking into different options for Spring Redis like pipelining.
https://github.com/lettuce-io/lettuce-core/wiki/Asynchronous-API
The problem that I'm running into is that I can't find a way to get the LettuceConnection to be asynchronous. I see that LettuceConnection.getAsyncConnection() is called when pipelining. And from what I can tell, pipelining is only available when you use RedisTemplate. As of Spring Data Redis 2, RedisCacheManager does not use RedisTemplate anymore; it uses RedisCacheConfiguration.
So, my question is how do we do Async Redis using the Spring Cache Abstraction (RedisCacheManager)?
We have this behavior of MongoDB connection which is unable to be traced; the MongoDB connections after some application idle time starts getting MongoDB connection error for every request made by the application. Only after the restart of the AppServer ('Tomcat', the application is deployed on) the MongoDB connections are reacquired and works well.
Want to know if anyone have come across such an issue and probable configuration solution. One another way which I think is then to Exception handle and do a retry for getting the connection.
NOTE: Java 1.7, Spring 3.1.x have been used for Server code. MongoDB Version: 2.6.9. Mongo Driver: mongo-2.10.1
Use these options for performance and preventing this problem.
autoConnectRetry = true
connectTimeout = 3000
connectionsPerHost = 40
socketTimeout = 120000
threadsAllowedToBlockForConnectionMultiplier = 5
maxAutoConnectRetryTime=5
maxWaitTime=120000