PostgreSQL open connection takes too long - spring

I am currently developing a project using the following libs/frameworks:
hibernate: 5.2.11.Final
spring: 5.0.2.RELEASE
postgresql: 9.4.1212
mysql: 5.1.31
spring-data-jpa: 2.0.2.RELEASE
The project should be available in both Postgres and mysql. I am also using a profiler program called "Prefix" to track possible performance issues.
I realized that all my queries on postgresql were taking longer than the ones made on mysql, so I decided to look on profiler what was the issue.
Whenever I tried to establish a new connection to the database on mysql (DriverManager.getConnection), it took something between 5~10ms, and the same connection to postgresql is taking about 200ms+.
Do you guys know what could be making driver connections on postgresql take longer than mysql?
Here is how the db is configured:
public DataSource dataSource() {
DriverManagerDataSource dataSource = new DriverManagerDataSource();
...
dataSource.setDriverClassName(org.postgresql.Driver);
...
}
private Properties hibernateProperties() {
Properties properties = new Properties();
properties.put("spring.jpa.hibernate.ddl-auto", props.getHibernateJPAAutoConfig());
if (props.getDatabaseName().equals(ProjectConstants.DATABASE_POSTGRE)) {
properties.put("spring.jpa.properties.hibernate.dialect", "org.hibernate.dialect.PostgreSQL9Dialect");
properties.put("spring.jpa.properties.hibernate.jdbc.use_get_generated_keys", true);
properties.put("spring.jpa.properties.hibernate.temp.use_jdbc_metadata_defaults", false); //started using those properties because I read that it could make postgresql connections faster; however it did not work.
}
return properties;
}
public EntityManagerFactory entityManagerFactory() {
LocalContainerEntityManagerFactoryBean factory = new LocalContainerEntityManagerFactoryBean();
factory.setDataSource(dataSource());
factory.setJpaProperties(hibernateProperties());
...
}

Related

Getting too many connection for role when using DataSource

I have a Rest service and when it gets it has to do some insertion and updation to almost 25 database. So when I tried like the below code, it was working in my localhost but when I deploy to my staging server I was getting FATAL: too many connections for role "user123"
List<String> databaseUrls = null;
databaseUrls.forEach( databaseUrl -> {
DataSource dataSource = DataSourceBuilder.create()
.driverClassName("org.postgresql.Driver")
.url(databaseUrl)
.username("user123")
.password("some-password")
.build();
JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
jdbcTemplate.update("Some...Update...Query");
});
As per my understanding DataSource need not to be closed because it is never opened.
Note:
A DataSource implementation need not be closed, because it is never
“opened”. A DataSource is not a resource, is not connected to the
database, so it is not holding networking connections nor resources on
the database server. A DataSource is simply information needed when
making a connection to the database, with the database server's
network name or address, the user name, user password, and various
options you want specified when a connection is eventually made.
Can someone tell why I am getting this issue
The problem is in DataSourceBuilder, it actually creates of the connection pools which spawns some number of connections and keeps them running:
private static final String[] DATA_SOURCE_TYPE_NAMES = new String[] {
"org.apache.tomcat.jdbc.pool.DataSource",
"com.zaxxer.hikari.HikariDataSource",
"org.apache.commons.dbcp.BasicDataSource" };
Javadoc says:
/**
* Convenience class for building a {#link DataSource} with common implementations and
* properties. If Tomcat, HikariCP or Commons DBCP are on the classpath one of them will
* be selected (in that order with Tomcat first). In the interest of a uniform interface,
* and so that there can be a fallback to an embedded database if one can be detected on
* the classpath, only a small set of common configuration properties are supported. To
* inject additional properties into the result you can downcast it, or use
* <code>#ConfigurationProperties</code>.
*/
Try to use e.g. SingleConnectionDataSource, then your problem will gone:
List<String> databaseUrls = null;
Class.forName("org.postgresql.Driver");
databaseUrls.forEach( databaseUrl -> {
SingleConnectionDataSource dataSource;
try {
dataSource = new SingleConnectionDataSource(
databaseUrl, "user123", "some-password", true /*suppressClose*/);
JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
jdbcTemplate.update("Some...Update...Query");
} catch (Exception e) {
log.error("Failed to run queries for {}", databaseUrl, e);
} finally {
// release resources
if (dataSource != null) {
dataSource.destroy();
}
}
});
First thing it is very bad architecture decision to have single application managing 50 database. Anyway instead of creating DataSource in for loop, you should make use of Factory Design pattern to create DataSource for each DB. You should add some connection pooling mechanism to your system . HijariCP and TomcatPool are most widely used. Analyse logs of failure thread for any further issues.

Include newly added data sources into route Data Source object without restarting the application server

Implemented Spring's AbstractRoutingDatasource by dynamically determining the actual DataSource based on the current context.
Refered this article : https://www.baeldung.com/spring-abstract-routing-data-source.
Here on spring boot application start up . Created a map of contexts to datasource objects to configure our AbstractRoutingDataSource. All these client context details are fetched from a database table.
#Bean
#DependsOn("dataSource")
#Primary
public DataSource routeDataSource() {
RoutingDataSource routeDataSource = new RoutingDataSource();
DataSource defaultDataSource = (DataSource) applicationContext.getBean("dataSource");
List<EstCredentials> credentials = LocalDataSourcesDetailsLoader.getAllCredentails(defaultDataSource); // fetching from database table
localDataSourceRegistrationBean.registerDataSourceBeans(estCredentials);
routeDataSource.setDefaultTargetDataSource(defaultDataSource);
Map<Object, Object> targetDataSources = new HashMap<>();
for (Credentials credential : credentials) {
targetDataSources.put(credential.getEstCode().toString(),
(DataSource) applicationContext.getBean(credential.getEstCode().toString()));
}
routeDataSource.setTargetDataSources(targetDataSources);
return routeDataSource;
}
The problem is if i add a new client details, I cannot get that in routeDataSource. Obvious reason is that these values are set on start up.
How can I achieve to add new client context and I had to re intialize the routeDataSource object.
Planning to write a service to get all the client context newly added and reset the routeDataSource object, no need to restart the server each time any changes in the client details.
A simple solution to this situation is adding #RefreshScope to the bean definition:
#Bean
#Primary
#RefreshScope
public DataSource routeDataSource() {
RoutingDataSource routeDataSource = new RoutingDataSource();
DataSource defaultDataSource = (DataSource) applicationContext.getBean("dataSource");
List<EstCredentials> credentials = LocalDataSourcesDetailsLoader.getAllCredentails(defaultDataSource); // fetching from database table
localDataSourceRegistrationBean.registerDataSourceBeans(estCredentials);
routeDataSource.setDefaultTargetDataSource(defaultDataSource);
Map<Object, Object> targetDataSources = new HashMap<>();
for (Credentials credential : credentials) {
targetDataSources.put(credential.getEstCode().toString(),
(DataSource) applicationContext.getBean(credential.getEstCode().toString()));
}
routeDataSource.setTargetDataSources(targetDataSources);
return routeDataSource;
}
Add Spring Boot Actuator as a dependency:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
Then trigger the refresh endpoint POST to /actuator/refresh to update the DataSource (actually every refresh scoped bean).
So this will depend on how much you know about the datasources to be added, but you could set this up as a multi-tenant project. Another example of creating new datasources:
#Autowired private Map <String, Datasource> mars2DataSources;
public void addDataSourceAtRuntime() {
DataSourceBuilder dataSourcebuilder = DataSourcebuilder.create(
MultiTenantJPAConfiguration.class.getclassloader())
.driverclassName("org.postgresql.Driver")
.username("postgres")
.password("postgres")
.url("Jdbc: postgresql://localhost:5412/somedb");
mars2DataSources("tenantX", datasourcebuilder.build())
}
Given that you are using Oracle, you could also use its database change notification features.
Think of it as a listener in the JDBC driver that gets notified whenever something changes in your database table. So upon receiving a change, you could reinitialize/add datasources.
You can find a tutorial of how to do this here: https://docs.oracle.com/cd/E11882_01/java.112/e16548/dbchgnf.htm#JJDBC28820
Though, depending on your organization database notifications need some extra firewall settings for the communication to work.
Advantage: You do not need to manually call the REST Endpoint if something changes, (though Marcos Barberios answer is perfectly valid!)

Spring Boot - Reconnect to a database after its restart

I have an Spring Batch application, which runs every 10 minutes. It gets some data from a REST API and then it saves these data on a database.
Well, where is my problem now?
Sometimes the database (Oracle) may restart, or go offline (no idea, really). But the application doesn't seem to reconnect to the database. It just stays on an idle mode.
Spring Boot: 2.1.2.RELEASE
The application.yml looks like this:
app:
database:
jdbc-url: jdbc:oracle:thin:#<host>:<port>:<db>
username: <username>
password: <password>
driver-class-name: oracle.jdbc.OracleDriver
options:
show-sql: true
ddl-auto: none
dialect: org.hibernate.dialect.Oracle12cDialect
and then, I configure the DataSource like this:
public DataSource dataSource() {
HikariConfig configuration = new HikariConfig();
configuration.setJdbcUrl(properties.getJdbcUrl());
configuration.setUsername(properties.getUsername());
configuration.setPassword(properties.getPassword());
configuration.setDriverClassName(properties.getDriverClassName());
configuration.setLeakDetectionThreshold(60 * 1000);
return new HikariDataSource(configuration);
}
public LocalContainerEntityManagerFactoryBean entityManagerFactory(DataSource dataSource) {
LocalContainerEntityManagerFactoryBean em = new LocalContainerEntityManagerFactoryBean();
em.setDataSource(dataSource);
em.setPackagesToScan("xxx.xxx.xx");
JpaVendorAdapter vendorAdapter = new HibernateJpaVendorAdapter();
em.setJpaVendorAdapter(vendorAdapter);
Properties additionalProperties = properties();
em.setJpaProperties(additionalProperties);
return em;
}
public PlatformTransactionManager transactionManager(EntityManagerFactory emf) {
return new JpaTransactionManager(emf);
}
private Properties properties() {
Properties additionalProperties = new Properties();
additionalProperties.setProperty("hibernate.hbm2ddl.auto", properties.getOptions().getDdlAuto());
additionalProperties.setProperty("hibernate.dialect", properties.getOptions().getDialect());
additionalProperties.setProperty("hibernate.show_sql", properties.getOptions().getShowSql());
return additionalProperties;
}
To be honest, I am not really sure, if I have done anything wrong here in the configuration.
Thank you!
You should configure maxLifetime by setMaxLifetime for 30 minutes
configuration.setMaxLifetime(108000);
property controls the maximum lifetime of a connection in the pool. When a connection reaches this timeout, even if recently used, it will be retired from the pool. An in-use connection will never be retired, only when it is idle will it be removed.
We strongly recommend setting this value, and it should be at least 30 seconds less than any database or infrastructure imposed connection time limit.
by default Oracle does not enforce a max lifetime for connections

spring data jpa (hibernate+spring+jpa)

I got one requirement where each company has separate database with the same schema, in future new company might be added to same schema structure.I have to connect their company schema based on user login. So that I have to connect separate database each time when user login.I wan to use spring data JPA to take care of connection.Only I will configure new schema credential in properties file.Is there any way we can add dynamic database schema which uses same model classes.All should happen in run time only.
Sounds like you are asking about multi-tenant support. You can create a LocalContainerEntityManagerFactoryBean something like this:
#Bean
public LocalContainerEntityManagerFactoryBean entityManagerFactory(
DataSource dataSource,
MultiTenantConnectionProvider multiTenantConnectionProvider,
CurrentTenantIdentifierResolver tenantIdentifierResolver) {
LocalContainerEntityManagerFactoryBean emfBean = new LocalContainerEntityManagerFactoryBean();
emfBean.setDataSource(dataSource);
emfBean.setPackagesToScan(Application.class.getPackage().getName());
emfBean.setJpaVendorAdapter(jpaVendorAdapter());
Map<String, Object> jpaProperties = new HashMap<>();
jpaProperties.put("hibernate.dialect", "org.hibernate.dialect.PostgreSQLDialect");
// Improved naming strategy deprecated as of hibernate 5.0
// jpaProperties.put("hibernate.ejb.naming_strategy", "org.hibernate.cfg.ImprovedNamingStrategy");
jpaProperties.put("hibernate.implicit_naming_strategy", "org.hibernate.boot.model.naming.ImplicitNamingStrategyLegacyHbmImpl");
jpaProperties.put("hibernate.physical_naming_strategy", "com.example.config.HibernateLegacyImprovedNamingStrategy");
jpaProperties.put("hibernate.show_sql", "true");
jpaProperties.put("hibernate.format_sql", "true");
jpaProperties.put(org.hibernate.cfg.Environment.MULTI_TENANT, MultiTenancyStrategy.SCHEMA);
jpaProperties.put(org.hibernate.cfg.Environment.MULTI_TENANT_CONNECTION_PROVIDER, multiTenantConnectionProvider);
jpaProperties.put(org.hibernate.cfg.Environment.MULTI_TENANT_IDENTIFIER_RESOLVER, tenantIdentifierResolver);
emfBean.setJpaPropertyMap(jpaProperties);
return emfBean;
}
And provide implementations of MultiTenantConnectionProvider and CurrentTenantIdentifierResolver
As an example, for schema tenancy, the CurrentTenantIdentifierResolver could look up the current user from session to determine what schema to use.
The MultiTenantConnectionProvider could alter the connection with a command like SELECT set_schema_to(?) before returning the connection for use (and reset it on return).
#Override
public Connection getConnection(String tenantIdentifier) throws SQLException {
final Connection connection = getAnyConnection();
setSchemaTo(connection, tenantIdentifier);
return connection;
}
You'll need to read up on multi-tenancy and see what type you want along with which implementations you are using.

Managing deployment-specific configurations with Spring

What strategies have people developed for controlling deployment configurations with Spring? I've already extracted out the environmental details (e.g. jdbc connection parameters) into a properties file, but I'm looking for some way of managing deployment details that aren't simple strings. Specifically, I'm currently using a locally configured datasource while doing development, and JNDI on our application servers (DEV, QA).
I've got an applicationContext.xml with the following two lines,
<import resource="spring/datasource-local-oracle.xml"/>
<import resource="spring/datasource-jndi.xml"/>
and I comment out whichever datasource isn't being used in that instance. There's got to be a better way to do this though. Thoughts, ideas, suggestions?
Use #Bean to define your datasource in code, rather than in XML. That way you can apply conditional logic to how the bean is created. For example:
#Value("${url:jdbc:hsqldb:mem:memdb}")
String url;
// username, password, etc
#Value("${jndiName:}")
String jndiName;
#Bean
public DataSource dataSource() {
DataSource ds;
if (jndiName == "") {
BasicDataSource bds = new BasicDataSource();
bds.setDriverClassName(driverClassName);
bds.setUrl(url);
bds.setUsername(username);
bds.setPassword(password);
ds = bds;
} else {
JndiObjectFactoryBean = jndiFactory = new JndiObjectFactoryBean();
jndiFactory.setJndiName("java:/" + jndiName);
jndiFactory.afterPropertiesSet();
ds = (DataSource) jndiFactory.getObject();
}
return ds;
}

Resources