How to set custom connection properties on DataSource in Spring Boot 1.3.x with default Tomcat connection pool - spring-boot

I need to set some specific Oracle JDBC connection properties in order to speed up batch INSERTs (defaultBatchValue) and mass SELECTs (defaultRowPrefetch).
I got suggestions how to achieve this with DBCP (Thanks to M. Deinum) but I would like to:
keep the default Tomcat jdbc connection pool
keep application.yml for configuration
I was thinking about a feature request to support spring.datasource.custom_connection_properties or similar in the future and because of this tried to pretent this was already possible. I did this by passing the relevant information while creating the DataSource and manipulated the creation of the DataSource like this:
#Bean
public DataSource dataSource() {
DataSource ds = null;
try {
Field props = DataSourceBuilder.class.getDeclaredField("properties");
props.setAccessible(true);
DataSourceBuilder builder = DataSourceBuilder.create();
Map<String, String> properties = (Map<String, String>) props.get(builder);
properties.put("defaultRowPrefetch", "1000");
properties.put("defaultBatchValue", "1000");
ds = builder.url( "jdbc:oracle:thin:#xyz:1521:abc" ).username( "ihave" ).password( "wonttell" ).build();
properties = (Map<String, String>) props.get(builder);
log.debug("properties after: {}", properties);
} ... leaving out the catches ...
}
log.debug("We are using this datasource: {}", ds);
return ds;
}
In the logs I can see that I am creating the correct DataSource:
2016-01-18 14:40:32.924 DEBUG 31204 --- [ main] d.a.e.a.c.config.DatabaseConfiguration : We are using this datasource: org.apache.tomcat.jdbc.pool.DataSource#19f040ba{ConnectionPool[defaultAutoCommit=null; ...
2016-01-18 14:40:32.919 DEBUG 31204 --- [ main] d.a.e.a.c.config.DatabaseConfiguration : properties after: {password=wonttell, driverClassName=oracle.jdbc.OracleDriver, defaultRowPrefetch=1000, defaultBatchValue=1000, url=jdbc:oracle:thin:#xyz:1521:abc, username=ihave}
The actuator shows me that my code replaced the datasource:
But the settings are not activated, which I can see while profiling the application. The defaultRowPrefetch is still at 10 which causes my SELECTs to be much slower than they would be if 1000 was activated.

Setting the pools connectionProperties should work. Those will be passed to the JDBC driver. Add this to application.properties:
spring.datasource.connectionProperties: defaultRowPrefetch=1000;defaultBatchValue=1000
Edit (some background information):
Note also that you can configure any of the DataSource implementation
specific properties via spring.datasource.*: refer to the
documentation of the connection pool implementation you are using for
more details.
source: spring-boot documentation

As Spring Boot is EOL for a long time I switched to Spring Boot 2.1 with its new default connection pool Hikari. Here the solution is even more simply and can be done in the application.properties or (like shown here) application.yml:
spring:
datasource:
hikari:
data-source-properties:
defaultRowPrefetch: 1000
(In a real-life config there would be several other configuration items but as they are not of interest for the question asked I simply left them out in my example)

Some additional information to complement the answer by #Cyril. If you want to upvote use his answer, not mine.
I was a little bit puzzled how easy it is to set additional connection properties that in the end get used while creating the database connection. So I did a little bit of research.
spring.datasource.connectionProperties is not mentioned in the reference. I created an issue because of this.
If I had used the Spring Boot YML editor, I would have seen which properties are supported. Here is what STS suggests when you create an application.yml and hit Ctrl+Space:
The dash does not matter because of relaxed binding but if you interpret it literally the propertys name is spring.datasource.connection-properties.
The correct setup in application.yml looks like this:
spring:
datasource:
connection-properties: defaultBatchValue=1000;defaultRowPrefetch=1000
...
This gets honored which is proven by my perf4j measurements of mass SELECTs.
Before:
2016-01-19 08:58:32.604 INFO 15108 --- [ main]
org.perf4j.TimingLogger : start[1453190311227]
time[1377] tag[get elements]
After:
2016-01-19 08:09:18.214 INFO 9152 --- [ main]
org.perf4j.TimingLogger : start[1453187358066]
time[147] tag[get elements]
The time taken to complete the SQL statement drops from 1377ms to 147, which is an enormous gain in performance.

After digging around in the Tomcat code for a bit, I found that the dataSource.getPoolProperties().getDbProperties() is the Properties object that will actually get used to generate connections for the pool.
If you use the BeanPostProcessor approach mentioned by #m-deinum, but instead use it to populate the dbProperties like so, you should be able to add the properties in a way that makes them stick and get passed to the Oracle driver.
import java.util.Properties;
import org.apache.tomcat.jdbc.pool.DataSource;
import org.apache.tomcat.jdbc.pool.PoolConfiguration;
#Component
public class OracleConfigurer implements BeanPostProcessor {
#Override
public Object postProcessBeforeInitialization(Object bean, String name) throws BeansException {
if (bean instanceof DataSource) {
DataSource dataSource = (DataSource)bean;
PoolConfiguration configuration = dataSource.getPoolProperties();
Properties properties = configuration.getDbProperties();
if (null == properties) properties = new Properties();
properties.put("defaultRowPrefetch", 1000);
properties.put("defaultBatchValue", 1000);
configuration.setDbProperties(properties);
}
return bean;
}
#Override
public Object postProcessAfterInitialization(Object bean, String name) throws BeansException {
return bean;
}
}

Related

How to add Log4j2 JDBC Appender programmatically to an existing configuration in Spring Boot?

A short rant at the beginning, just because it has to be said:
I'm getting tired of reading the terrible documentation of Log4j2 for the umpteenth time and still not finding any solutions for my problems. The documentation is completely outdated, sample code is torn uselessly out of a context that is needed but not explained further and the explanations are consistently insufficient. It shouldn't be that only Log4j2 developers can use Log4j2 in-depth. Frameworks should make the work of other developers easier, which is definitely not the case here. Period and thanks.
Now to my actual problem:
I have a Spring Boot application that is primarily configured with yaml files. The DataSource however is set programmatically so that we have a handle to its bean. Log4j2 is initially set up using yaml configuration as well.
log4j2-spring.yaml:
Configuration:
name: Default
status: warn
Appenders:
Console:
name: Console
target: SYSTEM_OUT
PatternLayout:
pattern: "%d{yyyy-MM-dd HH:mm:ss} %-5level [%t] %c: %msg%n"
Loggers:
Root:
level: warn
AppenderRef:
- ref: Console
Logger:
- name: com.example
level: debug
additivity: false
AppenderRef:
- ref: Console
What I want to do now is to extend this initial configuration programmatically with a JDBC Appender using the already existing connection-pool. According to the documentation, the following should be done:
The recommended approach for customizing a configuration is to extend one of the standard Configuration classes, override the setup method to first do super.setup() and then add the custom Appenders, Filters and LoggerConfigs to the configuration before it is registered for use.
So here is my custom Log4j2Configuration which extends YamlConfiguration:
public class Log4j2Configuration extends YamlConfiguration {
/* private final Log4j2ConnectionSource connectionSource; */ // <-- needs to get somehow injected
public Log4j2Configuration(LoggerContext loggerContext, ConfigurationSource configSource) {
super(loggerContext, configSource);
}
#Override
public void setup() {
super.setup();
}
#Override
protected void doConfigure() {
super.doConfigure();
LoggerContext context = (LoggerContext) LogManager.getContext(false);
Configuration config = context.getConfiguration();
ColumnConfig[] columns = new ColumnConfig[]{
//...
};
Appender jdbcAppender = JdbcAppender.newBuilder()
.setName("DataBase")
.setTableName("application_log")
// .setConnectionSource(connectionSource)
.setColumnConfigs(columns)
.build();
jdbcAppender.start();
config.addAppender(jdbcAppender);
AppenderRef ref = AppenderRef.createAppenderRef("DataBase", null, null);
AppenderRef[] refs = new AppenderRef[]{ref};
/* Deprecated, but still in the Log4j2 documentation */
LoggerConfig loggerConfig = LoggerConfig.createLogger(
false,
Level.TRACE,
"com.example",
"true",
refs,
null,
config,
null);
loggerConfig.addAppender(jdbcAppender, null, null);
config.addLogger("com.example", loggerConfig);
context.updateLoggers();
}
}
The ConnectionSource exists as an implementation of AbstractConnectionSource in the Spring context and still needs to be injected into the Log4j2Configuration class. Once I know how the configuration process works I can try to find a solution for this.
Log4j2ConnectionSource:
#Configuration
public class Log4j2ConnectionSource extends AbstractConnectionSource {
private final DataSource dataSource;
public Log4j2ConnectionSource(#Autowired #NotNull DataSource dataSource) {
this.dataSource = dataSource;
}
#Override
public Connection getConnection() throws SQLException {
return dataSource.getConnection();
}
}
And finally the ConfigurationFactory as described here in the documentation (It is interesting that the method getConfiguration calls with new MyXMLConfiguration(source, configFile) a constructor that doesn't exist. Is witchcraft at play here?).
Log4j2ConfigurationFactory:
#Order(50)
#Plugin(name = "Log4j2ConfigurationFactory", category = ConfigurationFactory.CATEGORY)
public class Log4j2ConfigurationFactory extends YamlConfigurationFactory {
#Override
public Configuration getConfiguration(LoggerContext context, ConfigurationSource configSource) {
return new Log4j2Configuration(context, configSource);
}
#Override
public String[] getSupportedTypes() {
return new String[]{".yml", "*"};
}
}
Now that the set up is more or less done, the running Log4j2 configuration needs somehow to be updated. So somebody should call doConfigure() within Log4j2Configuration. Log4j2 doesn't seem to do anything here on its own. Spring Boot doesn't do anything either. And I unfortunately don't have any plan what do at all.
Therefore my request:
Can anyone please explain to me how to get Log4j2 to update its configuration?
Many thanks for any help.

Include newly added data sources into route Data Source object without restarting the application server

Implemented Spring's AbstractRoutingDatasource by dynamically determining the actual DataSource based on the current context.
Refered this article : https://www.baeldung.com/spring-abstract-routing-data-source.
Here on spring boot application start up . Created a map of contexts to datasource objects to configure our AbstractRoutingDataSource. All these client context details are fetched from a database table.
#Bean
#DependsOn("dataSource")
#Primary
public DataSource routeDataSource() {
RoutingDataSource routeDataSource = new RoutingDataSource();
DataSource defaultDataSource = (DataSource) applicationContext.getBean("dataSource");
List<EstCredentials> credentials = LocalDataSourcesDetailsLoader.getAllCredentails(defaultDataSource); // fetching from database table
localDataSourceRegistrationBean.registerDataSourceBeans(estCredentials);
routeDataSource.setDefaultTargetDataSource(defaultDataSource);
Map<Object, Object> targetDataSources = new HashMap<>();
for (Credentials credential : credentials) {
targetDataSources.put(credential.getEstCode().toString(),
(DataSource) applicationContext.getBean(credential.getEstCode().toString()));
}
routeDataSource.setTargetDataSources(targetDataSources);
return routeDataSource;
}
The problem is if i add a new client details, I cannot get that in routeDataSource. Obvious reason is that these values are set on start up.
How can I achieve to add new client context and I had to re intialize the routeDataSource object.
Planning to write a service to get all the client context newly added and reset the routeDataSource object, no need to restart the server each time any changes in the client details.
A simple solution to this situation is adding #RefreshScope to the bean definition:
#Bean
#Primary
#RefreshScope
public DataSource routeDataSource() {
RoutingDataSource routeDataSource = new RoutingDataSource();
DataSource defaultDataSource = (DataSource) applicationContext.getBean("dataSource");
List<EstCredentials> credentials = LocalDataSourcesDetailsLoader.getAllCredentails(defaultDataSource); // fetching from database table
localDataSourceRegistrationBean.registerDataSourceBeans(estCredentials);
routeDataSource.setDefaultTargetDataSource(defaultDataSource);
Map<Object, Object> targetDataSources = new HashMap<>();
for (Credentials credential : credentials) {
targetDataSources.put(credential.getEstCode().toString(),
(DataSource) applicationContext.getBean(credential.getEstCode().toString()));
}
routeDataSource.setTargetDataSources(targetDataSources);
return routeDataSource;
}
Add Spring Boot Actuator as a dependency:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
Then trigger the refresh endpoint POST to /actuator/refresh to update the DataSource (actually every refresh scoped bean).
So this will depend on how much you know about the datasources to be added, but you could set this up as a multi-tenant project. Another example of creating new datasources:
#Autowired private Map <String, Datasource> mars2DataSources;
public void addDataSourceAtRuntime() {
DataSourceBuilder dataSourcebuilder = DataSourcebuilder.create(
MultiTenantJPAConfiguration.class.getclassloader())
.driverclassName("org.postgresql.Driver")
.username("postgres")
.password("postgres")
.url("Jdbc: postgresql://localhost:5412/somedb");
mars2DataSources("tenantX", datasourcebuilder.build())
}
Given that you are using Oracle, you could also use its database change notification features.
Think of it as a listener in the JDBC driver that gets notified whenever something changes in your database table. So upon receiving a change, you could reinitialize/add datasources.
You can find a tutorial of how to do this here: https://docs.oracle.com/cd/E11882_01/java.112/e16548/dbchgnf.htm#JJDBC28820
Though, depending on your organization database notifications need some extra firewall settings for the communication to work.
Advantage: You do not need to manually call the REST Endpoint if something changes, (though Marcos Barberios answer is perfectly valid!)

Unable to connect to Locator via GFSH

I have started a GemFire Server and Locator via Spring Boot and when I try to connect to the Locator from GFSH, I getting the following issue:
gfsh> connect
Connecting to Locator at [host=localhost, port=10334] ..
Connection refused: connect
Below, is the Spring (Java) configuration:
#Configuration
#ComponentScan
#EnableGemfireRepositories(basePackages= "com.gemfire.demo")
#CacheServerApplication(locators = "localhost[10334]")
#EnableManager
public class GemfireConfiguration {
#Bean
Properties gemfireProperties() {
Properties gemfireProperties = new Properties();
gemfireProperties.setProperty("name", "SpringDataGemFireApplication");
gemfireProperties.setProperty("mcast-port", "0");
gemfireProperties.setProperty("log-level", "info");
return gemfireProperties;
}
#Bean
#Autowired
CacheFactoryBean gemfireCache() {
CacheFactoryBean gemfireCache = new CacheFactoryBean();
gemfireCache.setClose(true);
gemfireCache.setProperties(gemfireProperties());
return gemfireCache;
}
#Bean(name="employee")
#Autowired
LocalRegionFactoryBean<String, Employee> getEmployee(final GemFireCache cache) {
LocalRegionFactoryBean<String, Employee> employeeRegion = new LocalRegionFactoryBean<String, Employee>();
employeeRegion.setCache(cache);
employeeRegion.setClose(false);
employeeRegion.setName("employee");
employeeRegion.setPersistent(false);
employeeRegion.setDataPolicy(DataPolicy.PRELOADED);
return employeeRegion;
}
}
Ref:Spring Data Gemfire locator
as per John's advice, I have enabled the Manager though I am still unable to connect.
You are not able to connect to the Locator (using Gfsh) because you don't have a Locator (service, either standalone or embedded) running with just the Spring (Java) config shown above.
Note that the #CacheServerApplication(locators = "localhost[10334]") annotation, specifically with the locators attribute as you have configured above, does NOT start an embedded Locator. It simply allows this Spring Boot configured and bootstrapped Apache Geode or Pivotal GemFire peer Cache node to join an existing distributed system (cluster) with an "existing" Locator running on localhost, listening on port 10334.
For instance, you could have started a Locator using Gfsh (e.g. start locator --name=X ...), then started your Spring Boot application with the Spring (Java) config shown above and you would see the Spring Boot app as part of the cluster formed by the Gfsh started Locator.
It is simply a shortcut (convenience) to configure and start an "embedded" Locator, but to do so, you need to use the #EnableLocator annotation.
Therefore, to configure and start an (embedded) Locator service in the same Spring Boot application as the CacheServer (and Manager), you must also include the #EnableLocator annotation, like so:
#SpringBootApplicaton
#CacheServerApplication
#EnableLocator
#EnableManager(start = true)
public class GemFireServerApplication {
...
}
I have plenty of examples of this here, for instance here, and talk about this generally here, etc.
As a side note, your whole configuration (class) is confused and it is clear you don't quite understand what you are doing. For instance, declaring the gemfireProperties and gemfireCache beans in JavaConfig is redundant and unnecessary since you are using the #CacheServerApplication annotation. Your whole configuration could be simplified to:
#CacheServerApplication(
name = "SpringDataGemFireApplication",
locators = "localhost[10334]",
logLevel = "info"
)
#EnableLocator
#EnableManager(start = true)
#EnableGemfireRepositories(basePackages= "com.gemfire.demo")
#ComponentScan
public class GemfireConfiguration {
#Bean(name="employee")
LocalRegionFactoryBean<String, Employee> getEmployee(GemFireCache cache) {
LocalRegionFactoryBean<String, Employee> employeeRegion =
new LocalRegionFactoryBean<String, Employee>();
employeeRegion.setCache(cache);
employeeRegion.setClose(false);
employeeRegion.setName("employee");
employeeRegion.setPersistent(false);
employeeRegion.setDataPolicy(DataPolicy.PRELOADED);
return employeeRegion;
}
}
Two things:
1) First, I would be highly careful about using classpath component scanning (#ComponentScan). I am not a fan of this configuration approach, especially in production where things should be explicit.
2) I would encourage your to considering using the type-safe basePackageClasses attribute on the #EnableGemFireRepositorities annotation rather than the basePackages attribute. With basePackageClasses, you only need to refer to a single interface/class in the desired package (e.g. com.gemfire.demo) rather than every interface/class. The referenced interface/class serves as a pointer to identify the package to scan from, including all sub-packages. It is type-safe and when your interfaces/classes in that package are re-located, then your attribute is still valid after the refactoring.
Anyway...
Hope this helps.
-j

Optimizing JDBC fetch size by use of Spring Boots application.properties

My Spring Boot 1.3.1 based application relies on an Oracle 11.2 database and I want to tune the fetching of SELECT statement results.
JdbcTemplate offers public void setFetchSize(int fetchSize) to tune the fetch size, which for Oracle is preset to 10 by the driver:
Set the fetch size for this JdbcTemplate. This is important for
processing large result sets: Setting this higher than the default
value will increase processing speed at the cost of memory
consumption; setting this lower can avoid transferring row data that
will never be read by the application. Default is -1, indicating to
use the JDBC driver's default (i.e. to not pass a specific fetch size
setting on the driver).
The Oracle JDBC driver (I use ojdbc7.jar because it is downwards compatible) offers a defaultRowPrefetch parameter to increase the fetch size for the complete database connection.
According to the docs this parameter could be set this way:
java.util.Properties info = new java.util.Properties();
info.put ("user", "scott");
info.put ("password","tiger");
info.put ("defaultRowPrefetch","15");
getConnection ("jdbc:oracle:oci8:#",info);
But my application is configured using application.yml:
datasource:
url: jdbc:oracle:thin:#xyz:1521:abc
username: ${name}
password: ${password}
driver-class-name: oracle.jdbc.driver.OracleDriver
...
And even if I wanted to change that configuration to use spring.datasource.url=jdbc:... instead there is no way to set the fetch size globally according to this post.
Is there a more "Spring Boot style" approach or do I need to configure each template manually ?
A BeanPostProcessor will process all the beans in the ApplicationContext and that way you can add additional configuration or replace it totally if you would like.
You could create a BeanPostProcessor that would add the properties to the configured DataSource. The sample below assumes the use of commons-dbcp 1 or 2 if you use a different DataSource modify accordingly.
public class DataSourceConfiguringBeanPostProcessor implements BeanPostProcessor {
private final Map<String,String> properties = new HashMap<>;
public Object postProcessBeforeInitialization(Object bean, String beanName) throws BeansException {
if (bean instance BasicDataSource ) {
for (Map.Entry<String, String> prop : properties.entrySet()) {
((BasicDataSource) bean).addConnectionProperty(prop.getKey(), prop.getValue());
}
}
return bean;
}
public Object postProcessAfterInitialization(Object bean, String beanName) throws BeansException {
return bean;
}
public void setProperties(Map<String, String> properties) {
this.properties.putAll(properties);
}
}
Now you can add this to your configuration and it will add the properties to DataSource beans.
#Bean
public BeanPostProcessor dataSourcePostProcessor() {
DataSourceConfiguringBeanPostProcessor processor = new DataSourceConfiguringBeanPostProcessor();
Map<String, String> properties = new HashMap<>();
properties.put("defaultRowPrefetch", "15");
properties.put("defaultBatchValue", "25");
processor.setProperties(properties);
return processor;
}
That should do the trick for configuring the datasource.

Camel: use datasource configured by spring-boot

I have a project and in it I'm using spring-boot-jdbc-starter and it automatically configures a DataSource for me.
Now I added camel-spring-boot to project and I was able to successfully create routes from Beans of type RouteBuilder.
But when I'm using sql component of camel it can not find datasource. Is there any simple way to add Spring configured datasource to CamelContext? In samples of camel project they use spring xml for datasource configuration but I'm looking for a way with java config. This is what I tried:
#Configuration
public class SqlRouteBuilder extends RouteBuilder {
#Bean
public SqlComponent sqlComponent(DataSource dataSource) {
SqlComponent sqlComponent = new SqlComponent();
sqlComponent.setDataSource(dataSource);
return sqlComponent;
}
#Override
public void configure() throws Exception {
from("sql:SELECT * FROM tasks WHERE STATUS NOT LIKE 'completed'")
.to("mock:sql");
}
}
I have to publish it because although the answer is in the commentary, you may not notice it, and in my case such a configuration was necessary to run the process.
The use of the SQL component should look like this:
from("timer://dbQueryTimer?period=10s")
.routeId("DATABASE_QUERY_TIMER_ROUTE")
.to("sql:SELECT * FROM event_queue?dataSource=#dataSource")
.process(xchg -> {
List<Map<String, Object>> row = xchg.getIn().getBody(List.class);
row.stream()
.map((x) -> {
EventQueue eventQueue = new EventQueue();
eventQueue.setId((Long)x.get("id"));
eventQueue.setData((String)x.get("data"));
return eventQueue;
}).collect(Collectors.toList());
})
.log(LoggingLevel.INFO,"******Database query executed - body:${body}******");
Note the use of ?dataSource=#dataSource. The dataSource name points to the DataSource object configured by Spring, it can be changed to another one and thus use different DataSource in different routes.
Here is the sample/example code (Java DSL). For this I used
Spring boot
H2 embedded Database
Camel
on startup spring-boot, creates table and loads data. Then camel route, runs "select" to pull the data.
Here is the code:
public void configure() throws Exception {
from("timer://timer1?period=1000")
.setBody(constant("select * from Employee"))
.to("jdbc:dataSource")
.split().simple("${body}")
.log("process row ${body}")
full example in github

Resources