I have an Spring Batch application, which runs every 10 minutes. It gets some data from a REST API and then it saves these data on a database.
Well, where is my problem now?
Sometimes the database (Oracle) may restart, or go offline (no idea, really). But the application doesn't seem to reconnect to the database. It just stays on an idle mode.
Spring Boot: 2.1.2.RELEASE
The application.yml looks like this:
app:
database:
jdbc-url: jdbc:oracle:thin:#<host>:<port>:<db>
username: <username>
password: <password>
driver-class-name: oracle.jdbc.OracleDriver
options:
show-sql: true
ddl-auto: none
dialect: org.hibernate.dialect.Oracle12cDialect
and then, I configure the DataSource like this:
public DataSource dataSource() {
HikariConfig configuration = new HikariConfig();
configuration.setJdbcUrl(properties.getJdbcUrl());
configuration.setUsername(properties.getUsername());
configuration.setPassword(properties.getPassword());
configuration.setDriverClassName(properties.getDriverClassName());
configuration.setLeakDetectionThreshold(60 * 1000);
return new HikariDataSource(configuration);
}
public LocalContainerEntityManagerFactoryBean entityManagerFactory(DataSource dataSource) {
LocalContainerEntityManagerFactoryBean em = new LocalContainerEntityManagerFactoryBean();
em.setDataSource(dataSource);
em.setPackagesToScan("xxx.xxx.xx");
JpaVendorAdapter vendorAdapter = new HibernateJpaVendorAdapter();
em.setJpaVendorAdapter(vendorAdapter);
Properties additionalProperties = properties();
em.setJpaProperties(additionalProperties);
return em;
}
public PlatformTransactionManager transactionManager(EntityManagerFactory emf) {
return new JpaTransactionManager(emf);
}
private Properties properties() {
Properties additionalProperties = new Properties();
additionalProperties.setProperty("hibernate.hbm2ddl.auto", properties.getOptions().getDdlAuto());
additionalProperties.setProperty("hibernate.dialect", properties.getOptions().getDialect());
additionalProperties.setProperty("hibernate.show_sql", properties.getOptions().getShowSql());
return additionalProperties;
}
To be honest, I am not really sure, if I have done anything wrong here in the configuration.
Thank you!
You should configure maxLifetime by setMaxLifetime for 30 minutes
configuration.setMaxLifetime(108000);
property controls the maximum lifetime of a connection in the pool. When a connection reaches this timeout, even if recently used, it will be retired from the pool. An in-use connection will never be retired, only when it is idle will it be removed.
We strongly recommend setting this value, and it should be at least 30 seconds less than any database or infrastructure imposed connection time limit.
by default Oracle does not enforce a max lifetime for connections
Related
I am new to Spring boot development.
What I did?
I tested the sample MySQL integration with Spring boot and tested it works fine.
What I am trying?
I am trying to use Hibernate APIs with my existing project. So I create a util class for creating Hibernate sessionFactory.
My code:
public static void initSessionFactory() {
System.out.print("initSessionFactory");
if (sessionFactory == null) {
try {
Configuration configuration = new Configuration();
//***** I don't want configuration here. ******
// Hibernate settings equivalent to hibernate.cfg.xml's properties
//Properties settings = new Properties();
// settings.put(Environment.DRIVER, "com.mysql.cj.jdbc.Driver");
// settings.put(Environment.URL, "jdbc:mysql://localhost:3306/hibernate_db?useSSL=false");
// settings.put(Environment.USER, "root");
// settings.put(Environment.PASS, "root");
// settings.put(Environment.DIALECT, "org.hibernate.dialect.MySQL5Dialect");
// settings.put(Environment.CURRENT_SESSION_CONTEXT_CLASS, "thread");
// settings.put(Environment.HBM2DDL_AUTO, "create-drop");
// settings.put(Environment.SHOW_SQL, "true");
// configuration.setProperties(settings);
//configuration.addAnnotatedClass(Student.class);
ServiceRegistry serviceRegistry = new StandardServiceRegistryBuilder()
.applySettings(configuration.getProperties()).build();
sessionFactory = configuration.buildSessionFactory(serviceRegistry);
session = sessionFactory.openSession();
} catch (Exception e) {
e.printStackTrace();
}
}
}
My application.properties:(It works fine for without Hibernate)
spring.jpa.hibernate.ddl-auto=update
spring.datasource.url=jdbc:mysql://localhost:3306/tcc
spring.datasource.username=root
spring.datasource.password=password
spring.datasource.driverClassName=com.mysql.cj.jdbc.Driver
spring.jpa.database-platform=org.hibernate.dialect.MySQL8Dialect
spring.jpa.show-sql=true
My problem:
I already configured the jdbc environment & other configuration in my application.properties. It works fine. So I don't want to repeat the same configuration in my Java code. So I commented this configuration.
But without Java configuration it throws "The application must supply JDBC connections" error.
My Question:
How to set configuration from application.properties for Hibernate?
When you set the properties below in your app.properties, the Spring Boot will already make your DB connection ready when the application start.
spring.jpa.hibernate.ddl-auto=update
spring.datasource.url=jdbc:mysql://${MYSQL_HOST:localhost}:3306/db_example
spring.datasource.username=springuser
spring.datasource.password=ThePassword
spring.datasource.driver-class-name =com.mysql.jdbc.Driver
You do not need to create SessionFactory , ServiceRegistery etc. even if you do not want to manage DB connection manually.
This link also explain the steps in more detail. You will see that there is no any custom bean, factory or registerer to establish DB connection. Please see it.
https://spring.io/guides/gs/accessing-data-mysql/
Implemented Spring's AbstractRoutingDatasource by dynamically determining the actual DataSource based on the current context.
Refered this article : https://www.baeldung.com/spring-abstract-routing-data-source.
Here on spring boot application start up . Created a map of contexts to datasource objects to configure our AbstractRoutingDataSource. All these client context details are fetched from a database table.
#Bean
#DependsOn("dataSource")
#Primary
public DataSource routeDataSource() {
RoutingDataSource routeDataSource = new RoutingDataSource();
DataSource defaultDataSource = (DataSource) applicationContext.getBean("dataSource");
List<EstCredentials> credentials = LocalDataSourcesDetailsLoader.getAllCredentails(defaultDataSource); // fetching from database table
localDataSourceRegistrationBean.registerDataSourceBeans(estCredentials);
routeDataSource.setDefaultTargetDataSource(defaultDataSource);
Map<Object, Object> targetDataSources = new HashMap<>();
for (Credentials credential : credentials) {
targetDataSources.put(credential.getEstCode().toString(),
(DataSource) applicationContext.getBean(credential.getEstCode().toString()));
}
routeDataSource.setTargetDataSources(targetDataSources);
return routeDataSource;
}
The problem is if i add a new client details, I cannot get that in routeDataSource. Obvious reason is that these values are set on start up.
How can I achieve to add new client context and I had to re intialize the routeDataSource object.
Planning to write a service to get all the client context newly added and reset the routeDataSource object, no need to restart the server each time any changes in the client details.
A simple solution to this situation is adding #RefreshScope to the bean definition:
#Bean
#Primary
#RefreshScope
public DataSource routeDataSource() {
RoutingDataSource routeDataSource = new RoutingDataSource();
DataSource defaultDataSource = (DataSource) applicationContext.getBean("dataSource");
List<EstCredentials> credentials = LocalDataSourcesDetailsLoader.getAllCredentails(defaultDataSource); // fetching from database table
localDataSourceRegistrationBean.registerDataSourceBeans(estCredentials);
routeDataSource.setDefaultTargetDataSource(defaultDataSource);
Map<Object, Object> targetDataSources = new HashMap<>();
for (Credentials credential : credentials) {
targetDataSources.put(credential.getEstCode().toString(),
(DataSource) applicationContext.getBean(credential.getEstCode().toString()));
}
routeDataSource.setTargetDataSources(targetDataSources);
return routeDataSource;
}
Add Spring Boot Actuator as a dependency:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
Then trigger the refresh endpoint POST to /actuator/refresh to update the DataSource (actually every refresh scoped bean).
So this will depend on how much you know about the datasources to be added, but you could set this up as a multi-tenant project. Another example of creating new datasources:
#Autowired private Map <String, Datasource> mars2DataSources;
public void addDataSourceAtRuntime() {
DataSourceBuilder dataSourcebuilder = DataSourcebuilder.create(
MultiTenantJPAConfiguration.class.getclassloader())
.driverclassName("org.postgresql.Driver")
.username("postgres")
.password("postgres")
.url("Jdbc: postgresql://localhost:5412/somedb");
mars2DataSources("tenantX", datasourcebuilder.build())
}
Given that you are using Oracle, you could also use its database change notification features.
Think of it as a listener in the JDBC driver that gets notified whenever something changes in your database table. So upon receiving a change, you could reinitialize/add datasources.
You can find a tutorial of how to do this here: https://docs.oracle.com/cd/E11882_01/java.112/e16548/dbchgnf.htm#JJDBC28820
Though, depending on your organization database notifications need some extra firewall settings for the communication to work.
Advantage: You do not need to manually call the REST Endpoint if something changes, (though Marcos Barberios answer is perfectly valid!)
I am currently developing a project using the following libs/frameworks:
hibernate: 5.2.11.Final
spring: 5.0.2.RELEASE
postgresql: 9.4.1212
mysql: 5.1.31
spring-data-jpa: 2.0.2.RELEASE
The project should be available in both Postgres and mysql. I am also using a profiler program called "Prefix" to track possible performance issues.
I realized that all my queries on postgresql were taking longer than the ones made on mysql, so I decided to look on profiler what was the issue.
Whenever I tried to establish a new connection to the database on mysql (DriverManager.getConnection), it took something between 5~10ms, and the same connection to postgresql is taking about 200ms+.
Do you guys know what could be making driver connections on postgresql take longer than mysql?
Here is how the db is configured:
public DataSource dataSource() {
DriverManagerDataSource dataSource = new DriverManagerDataSource();
...
dataSource.setDriverClassName(org.postgresql.Driver);
...
}
private Properties hibernateProperties() {
Properties properties = new Properties();
properties.put("spring.jpa.hibernate.ddl-auto", props.getHibernateJPAAutoConfig());
if (props.getDatabaseName().equals(ProjectConstants.DATABASE_POSTGRE)) {
properties.put("spring.jpa.properties.hibernate.dialect", "org.hibernate.dialect.PostgreSQL9Dialect");
properties.put("spring.jpa.properties.hibernate.jdbc.use_get_generated_keys", true);
properties.put("spring.jpa.properties.hibernate.temp.use_jdbc_metadata_defaults", false); //started using those properties because I read that it could make postgresql connections faster; however it did not work.
}
return properties;
}
public EntityManagerFactory entityManagerFactory() {
LocalContainerEntityManagerFactoryBean factory = new LocalContainerEntityManagerFactoryBean();
factory.setDataSource(dataSource());
factory.setJpaProperties(hibernateProperties());
...
}
I got one requirement where each company has separate database with the same schema, in future new company might be added to same schema structure.I have to connect their company schema based on user login. So that I have to connect separate database each time when user login.I wan to use spring data JPA to take care of connection.Only I will configure new schema credential in properties file.Is there any way we can add dynamic database schema which uses same model classes.All should happen in run time only.
Sounds like you are asking about multi-tenant support. You can create a LocalContainerEntityManagerFactoryBean something like this:
#Bean
public LocalContainerEntityManagerFactoryBean entityManagerFactory(
DataSource dataSource,
MultiTenantConnectionProvider multiTenantConnectionProvider,
CurrentTenantIdentifierResolver tenantIdentifierResolver) {
LocalContainerEntityManagerFactoryBean emfBean = new LocalContainerEntityManagerFactoryBean();
emfBean.setDataSource(dataSource);
emfBean.setPackagesToScan(Application.class.getPackage().getName());
emfBean.setJpaVendorAdapter(jpaVendorAdapter());
Map<String, Object> jpaProperties = new HashMap<>();
jpaProperties.put("hibernate.dialect", "org.hibernate.dialect.PostgreSQLDialect");
// Improved naming strategy deprecated as of hibernate 5.0
// jpaProperties.put("hibernate.ejb.naming_strategy", "org.hibernate.cfg.ImprovedNamingStrategy");
jpaProperties.put("hibernate.implicit_naming_strategy", "org.hibernate.boot.model.naming.ImplicitNamingStrategyLegacyHbmImpl");
jpaProperties.put("hibernate.physical_naming_strategy", "com.example.config.HibernateLegacyImprovedNamingStrategy");
jpaProperties.put("hibernate.show_sql", "true");
jpaProperties.put("hibernate.format_sql", "true");
jpaProperties.put(org.hibernate.cfg.Environment.MULTI_TENANT, MultiTenancyStrategy.SCHEMA);
jpaProperties.put(org.hibernate.cfg.Environment.MULTI_TENANT_CONNECTION_PROVIDER, multiTenantConnectionProvider);
jpaProperties.put(org.hibernate.cfg.Environment.MULTI_TENANT_IDENTIFIER_RESOLVER, tenantIdentifierResolver);
emfBean.setJpaPropertyMap(jpaProperties);
return emfBean;
}
And provide implementations of MultiTenantConnectionProvider and CurrentTenantIdentifierResolver
As an example, for schema tenancy, the CurrentTenantIdentifierResolver could look up the current user from session to determine what schema to use.
The MultiTenantConnectionProvider could alter the connection with a command like SELECT set_schema_to(?) before returning the connection for use (and reset it on return).
#Override
public Connection getConnection(String tenantIdentifier) throws SQLException {
final Connection connection = getAnyConnection();
setSchemaTo(connection, tenantIdentifier);
return connection;
}
You'll need to read up on multi-tenancy and see what type you want along with which implementations you are using.
I need to set some specific Oracle JDBC connection properties in order to speed up batch INSERTs (defaultBatchValue) and mass SELECTs (defaultRowPrefetch).
I got suggestions how to achieve this with DBCP (Thanks to M. Deinum) but I would like to:
keep the default Tomcat jdbc connection pool
keep application.yml for configuration
I was thinking about a feature request to support spring.datasource.custom_connection_properties or similar in the future and because of this tried to pretent this was already possible. I did this by passing the relevant information while creating the DataSource and manipulated the creation of the DataSource like this:
#Bean
public DataSource dataSource() {
DataSource ds = null;
try {
Field props = DataSourceBuilder.class.getDeclaredField("properties");
props.setAccessible(true);
DataSourceBuilder builder = DataSourceBuilder.create();
Map<String, String> properties = (Map<String, String>) props.get(builder);
properties.put("defaultRowPrefetch", "1000");
properties.put("defaultBatchValue", "1000");
ds = builder.url( "jdbc:oracle:thin:#xyz:1521:abc" ).username( "ihave" ).password( "wonttell" ).build();
properties = (Map<String, String>) props.get(builder);
log.debug("properties after: {}", properties);
} ... leaving out the catches ...
}
log.debug("We are using this datasource: {}", ds);
return ds;
}
In the logs I can see that I am creating the correct DataSource:
2016-01-18 14:40:32.924 DEBUG 31204 --- [ main] d.a.e.a.c.config.DatabaseConfiguration : We are using this datasource: org.apache.tomcat.jdbc.pool.DataSource#19f040ba{ConnectionPool[defaultAutoCommit=null; ...
2016-01-18 14:40:32.919 DEBUG 31204 --- [ main] d.a.e.a.c.config.DatabaseConfiguration : properties after: {password=wonttell, driverClassName=oracle.jdbc.OracleDriver, defaultRowPrefetch=1000, defaultBatchValue=1000, url=jdbc:oracle:thin:#xyz:1521:abc, username=ihave}
The actuator shows me that my code replaced the datasource:
But the settings are not activated, which I can see while profiling the application. The defaultRowPrefetch is still at 10 which causes my SELECTs to be much slower than they would be if 1000 was activated.
Setting the pools connectionProperties should work. Those will be passed to the JDBC driver. Add this to application.properties:
spring.datasource.connectionProperties: defaultRowPrefetch=1000;defaultBatchValue=1000
Edit (some background information):
Note also that you can configure any of the DataSource implementation
specific properties via spring.datasource.*: refer to the
documentation of the connection pool implementation you are using for
more details.
source: spring-boot documentation
As Spring Boot is EOL for a long time I switched to Spring Boot 2.1 with its new default connection pool Hikari. Here the solution is even more simply and can be done in the application.properties or (like shown here) application.yml:
spring:
datasource:
hikari:
data-source-properties:
defaultRowPrefetch: 1000
(In a real-life config there would be several other configuration items but as they are not of interest for the question asked I simply left them out in my example)
Some additional information to complement the answer by #Cyril. If you want to upvote use his answer, not mine.
I was a little bit puzzled how easy it is to set additional connection properties that in the end get used while creating the database connection. So I did a little bit of research.
spring.datasource.connectionProperties is not mentioned in the reference. I created an issue because of this.
If I had used the Spring Boot YML editor, I would have seen which properties are supported. Here is what STS suggests when you create an application.yml and hit Ctrl+Space:
The dash does not matter because of relaxed binding but if you interpret it literally the propertys name is spring.datasource.connection-properties.
The correct setup in application.yml looks like this:
spring:
datasource:
connection-properties: defaultBatchValue=1000;defaultRowPrefetch=1000
...
This gets honored which is proven by my perf4j measurements of mass SELECTs.
Before:
2016-01-19 08:58:32.604 INFO 15108 --- [ main]
org.perf4j.TimingLogger : start[1453190311227]
time[1377] tag[get elements]
After:
2016-01-19 08:09:18.214 INFO 9152 --- [ main]
org.perf4j.TimingLogger : start[1453187358066]
time[147] tag[get elements]
The time taken to complete the SQL statement drops from 1377ms to 147, which is an enormous gain in performance.
After digging around in the Tomcat code for a bit, I found that the dataSource.getPoolProperties().getDbProperties() is the Properties object that will actually get used to generate connections for the pool.
If you use the BeanPostProcessor approach mentioned by #m-deinum, but instead use it to populate the dbProperties like so, you should be able to add the properties in a way that makes them stick and get passed to the Oracle driver.
import java.util.Properties;
import org.apache.tomcat.jdbc.pool.DataSource;
import org.apache.tomcat.jdbc.pool.PoolConfiguration;
#Component
public class OracleConfigurer implements BeanPostProcessor {
#Override
public Object postProcessBeforeInitialization(Object bean, String name) throws BeansException {
if (bean instanceof DataSource) {
DataSource dataSource = (DataSource)bean;
PoolConfiguration configuration = dataSource.getPoolProperties();
Properties properties = configuration.getDbProperties();
if (null == properties) properties = new Properties();
properties.put("defaultRowPrefetch", 1000);
properties.put("defaultBatchValue", 1000);
configuration.setDbProperties(properties);
}
return bean;
}
#Override
public Object postProcessAfterInitialization(Object bean, String name) throws BeansException {
return bean;
}
}