How to add Log4j2 JDBC Appender programmatically to an existing configuration in Spring Boot? - spring-boot

A short rant at the beginning, just because it has to be said:
I'm getting tired of reading the terrible documentation of Log4j2 for the umpteenth time and still not finding any solutions for my problems. The documentation is completely outdated, sample code is torn uselessly out of a context that is needed but not explained further and the explanations are consistently insufficient. It shouldn't be that only Log4j2 developers can use Log4j2 in-depth. Frameworks should make the work of other developers easier, which is definitely not the case here. Period and thanks.
Now to my actual problem:
I have a Spring Boot application that is primarily configured with yaml files. The DataSource however is set programmatically so that we have a handle to its bean. Log4j2 is initially set up using yaml configuration as well.
log4j2-spring.yaml:
Configuration:
name: Default
status: warn
Appenders:
Console:
name: Console
target: SYSTEM_OUT
PatternLayout:
pattern: "%d{yyyy-MM-dd HH:mm:ss} %-5level [%t] %c: %msg%n"
Loggers:
Root:
level: warn
AppenderRef:
- ref: Console
Logger:
- name: com.example
level: debug
additivity: false
AppenderRef:
- ref: Console
What I want to do now is to extend this initial configuration programmatically with a JDBC Appender using the already existing connection-pool. According to the documentation, the following should be done:
The recommended approach for customizing a configuration is to extend one of the standard Configuration classes, override the setup method to first do super.setup() and then add the custom Appenders, Filters and LoggerConfigs to the configuration before it is registered for use.
So here is my custom Log4j2Configuration which extends YamlConfiguration:
public class Log4j2Configuration extends YamlConfiguration {
/* private final Log4j2ConnectionSource connectionSource; */ // <-- needs to get somehow injected
public Log4j2Configuration(LoggerContext loggerContext, ConfigurationSource configSource) {
super(loggerContext, configSource);
}
#Override
public void setup() {
super.setup();
}
#Override
protected void doConfigure() {
super.doConfigure();
LoggerContext context = (LoggerContext) LogManager.getContext(false);
Configuration config = context.getConfiguration();
ColumnConfig[] columns = new ColumnConfig[]{
//...
};
Appender jdbcAppender = JdbcAppender.newBuilder()
.setName("DataBase")
.setTableName("application_log")
// .setConnectionSource(connectionSource)
.setColumnConfigs(columns)
.build();
jdbcAppender.start();
config.addAppender(jdbcAppender);
AppenderRef ref = AppenderRef.createAppenderRef("DataBase", null, null);
AppenderRef[] refs = new AppenderRef[]{ref};
/* Deprecated, but still in the Log4j2 documentation */
LoggerConfig loggerConfig = LoggerConfig.createLogger(
false,
Level.TRACE,
"com.example",
"true",
refs,
null,
config,
null);
loggerConfig.addAppender(jdbcAppender, null, null);
config.addLogger("com.example", loggerConfig);
context.updateLoggers();
}
}
The ConnectionSource exists as an implementation of AbstractConnectionSource in the Spring context and still needs to be injected into the Log4j2Configuration class. Once I know how the configuration process works I can try to find a solution for this.
Log4j2ConnectionSource:
#Configuration
public class Log4j2ConnectionSource extends AbstractConnectionSource {
private final DataSource dataSource;
public Log4j2ConnectionSource(#Autowired #NotNull DataSource dataSource) {
this.dataSource = dataSource;
}
#Override
public Connection getConnection() throws SQLException {
return dataSource.getConnection();
}
}
And finally the ConfigurationFactory as described here in the documentation (It is interesting that the method getConfiguration calls with new MyXMLConfiguration(source, configFile) a constructor that doesn't exist. Is witchcraft at play here?).
Log4j2ConfigurationFactory:
#Order(50)
#Plugin(name = "Log4j2ConfigurationFactory", category = ConfigurationFactory.CATEGORY)
public class Log4j2ConfigurationFactory extends YamlConfigurationFactory {
#Override
public Configuration getConfiguration(LoggerContext context, ConfigurationSource configSource) {
return new Log4j2Configuration(context, configSource);
}
#Override
public String[] getSupportedTypes() {
return new String[]{".yml", "*"};
}
}
Now that the set up is more or less done, the running Log4j2 configuration needs somehow to be updated. So somebody should call doConfigure() within Log4j2Configuration. Log4j2 doesn't seem to do anything here on its own. Spring Boot doesn't do anything either. And I unfortunately don't have any plan what do at all.
Therefore my request:
Can anyone please explain to me how to get Log4j2 to update its configuration?
Many thanks for any help.

Related

SpringBoot: How to update Logback file-pattern from a library (custom spring boot starter)

I would like to add TraceId to all log lines. I do that easily by:
Add traceId to MDC
MDC.put("TRACE_ID", sessionAware.getTraceId()+": ");
Update file-pattern in my application.properties (by adding: "%X{TRACE_ID}"):
logging.pattern.file=-%d{${LOG_DATEFORMAT_PATTERN:-yyyy-MM-dd HH:mm:ss.SSS}} ${LOG_LEVEL_PATTERN:-%5p} ${PID:- } --- [%t] %-40.40logger{39} : %X{TRACE_ID}%m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}
But I would like my CustomSpringBootStarter to set the file pattern. But it doesn't take effect when I update property "logging.pattern.file" from my CustomSpringBootStarter. Does anyone know the solution to this problem?
I have tried to set the "logging.pattern.file" property in the CustomSpringBootStarter's application.properties but it does not work.
I found a solution to the problem, very much inspired by this question.
I created the below EnvironmentPostProcessor in my CustomSpringBootStarter:
public class LifeLogBackConfiguration implements EnvironmentPostProcessor {
#Override
public void postProcessEnvironment(ConfigurableEnvironment environment, SpringApplication application) {
Map<String, String> properties = Collections.unmodifiableMap(Map.of(
"logging.pattern.file", LogConstants.FILE_LOG_PATTERN_LIFE,
"logging.pattern.console", LogConstants.CONSOLE_LOG_PATTERN_LIFE));
PropertySource propertySource = new OriginTrackedMapPropertySource("LogPatternsLife", properties, true);
environment.getPropertySources().addLast(propertySource);
}
}
,and registered it in: resources/META-INF/spring.factories
org.springframework.boot.env.EnvironmentPostProcessor=dk.topdanmark.life.lifespringbootstarter.log.LifeLogBackConfiguration
That solved the problem for me.

Problem with connection to Neo4j test container using Spring boot 2 and JUnit5

Problem with connection to Neo4j test container using Spring boot 2 and JUnit5
int test context. Container started successfully but spring.data.neo4j.uri property has a wrong default port:7687, I guess this URI must be the same when I call neo4jContainer.getBoltUrl().
Everything works fine in this case:
#Testcontainers
public class ExampleTest {
#Container
private static Neo4jContainer neo4jContainer = new Neo4jContainer()
.withAdminPassword(null); // Disable password
#Test
void testSomethingUsingBolt() {
// Retrieve the Bolt URL from the container
String boltUrl = neo4jContainer.getBoltUrl();
try (
Driver driver = GraphDatabase.driver(boltUrl, AuthTokens.none());
Session session = driver.session()
) {
long one = session.run("RETURN 1",
Collections.emptyMap()).next().get(0).asLong();
assertThat(one, is(1L));
} catch (Exception e) {
fail(e.getMessage());
}
}
}
But SessionFactory is not created for the application using autoconfiguration following to these recommendations - https://www.testcontainers.org/modules/databases/neo4j/
When I try to create own primary bean - SessionFactory in test context I get the message like this - "URI cannot be returned before the container is not loaded"
But Application runs and works perfect using autoconfiguration and neo4j started in a container, the same cannot be told about the test context
You cannot rely 100% on Spring Boot's auto configuration (for production) in this case because it will read the application.properties or use the default values for the connection.
To achieve what you want to, the key part is to create a custom (Neo4j-OGM) Configuration bean. The #DataNeo4jTest annotation is provided by the spring-boot-test-autoconfigure module.
#Testcontainers
#DataNeo4jTest
public class TestClass {
#TestConfiguration
static class Config {
#Bean
public org.neo4j.ogm.config.Configuration configuration() {
return new Configuration.Builder()
.uri(databaseServer.getBoltUrl())
.credentials("neo4j", databaseServer.getAdminPassword())
.build();
}
}
// your tests
}
For a broader explanation have a look at this blog post. Esp. the section Using with Neo4j-OGM and SDN.

What is the best alternative for #ConfigurationProperties locations?

#ConfigurationProperties locations is deprecated in Spring Boot 1.4.x and option is now removed in 1.5.x
I was using it like this: BucketTestConfig.java
For now with deprecation, I'm trying to set the system property spring.config.location for both production code and test code as an alternative.
./gradlew clean test is still failing although I set the system property.
What is the best alternative for deprecated #ConfigurationProperties locations in this case?
UPDATE:
Using SpringApplicationBuilder.properties() doesn't work in the test (BucketTestRepositoryTests).
Using SpringApplicationBuilder.listeners() doesn't work in the test (BucketTestRepositoryTests), either.
UPDATE (2nd):
There was no reason to depend on #ConfigurationProperties in my case, so I went with Yaml instead as follows: https://github.com/izeye/spring-boot-throwaway-branches/commit/a1290672dceea98706b1a258f8a17e2628ea01ee
So this question's title is invalid and this question can be deleted.
Follow this thread for more information.
Basically, this thread suggests two options
First option is to set spring.config.name to a list of the files you want to load:
new SpringApplicationBuilder(Application.class)
.properties("spring.config.name=application,mine")
.run(args);
Second options is to add listeners
new SpringApplicationBuilder(SanityCheckApplication.class)
.listeners(new LoadAdditionalProperties())
.run(args);
Content of LoadAdditionalProperties
#Component
public class LoadAdditionalProperties implements ApplicationListener<ApplicationEnvironmentPreparedEvent> {
private ResourceLoader loader = new DefaultResourceLoader();
#Override
public void onApplicationEvent(ApplicationEnvironmentPreparedEvent event) {
try {
Resource resource = loader.getResource("classpath:mine.properties");
PropertySource<?> propertySource = new PropertySourcesLoader().load(resource);
event.getEnvironment().getPropertySources().addLast(propertySource);
} catch (IOException ex) {
throw new IllegalStateException(ex);
}
}
}

Optimizing JDBC fetch size by use of Spring Boots application.properties

My Spring Boot 1.3.1 based application relies on an Oracle 11.2 database and I want to tune the fetching of SELECT statement results.
JdbcTemplate offers public void setFetchSize(int fetchSize) to tune the fetch size, which for Oracle is preset to 10 by the driver:
Set the fetch size for this JdbcTemplate. This is important for
processing large result sets: Setting this higher than the default
value will increase processing speed at the cost of memory
consumption; setting this lower can avoid transferring row data that
will never be read by the application. Default is -1, indicating to
use the JDBC driver's default (i.e. to not pass a specific fetch size
setting on the driver).
The Oracle JDBC driver (I use ojdbc7.jar because it is downwards compatible) offers a defaultRowPrefetch parameter to increase the fetch size for the complete database connection.
According to the docs this parameter could be set this way:
java.util.Properties info = new java.util.Properties();
info.put ("user", "scott");
info.put ("password","tiger");
info.put ("defaultRowPrefetch","15");
getConnection ("jdbc:oracle:oci8:#",info);
But my application is configured using application.yml:
datasource:
url: jdbc:oracle:thin:#xyz:1521:abc
username: ${name}
password: ${password}
driver-class-name: oracle.jdbc.driver.OracleDriver
...
And even if I wanted to change that configuration to use spring.datasource.url=jdbc:... instead there is no way to set the fetch size globally according to this post.
Is there a more "Spring Boot style" approach or do I need to configure each template manually ?
A BeanPostProcessor will process all the beans in the ApplicationContext and that way you can add additional configuration or replace it totally if you would like.
You could create a BeanPostProcessor that would add the properties to the configured DataSource. The sample below assumes the use of commons-dbcp 1 or 2 if you use a different DataSource modify accordingly.
public class DataSourceConfiguringBeanPostProcessor implements BeanPostProcessor {
private final Map<String,String> properties = new HashMap<>;
public Object postProcessBeforeInitialization(Object bean, String beanName) throws BeansException {
if (bean instance BasicDataSource ) {
for (Map.Entry<String, String> prop : properties.entrySet()) {
((BasicDataSource) bean).addConnectionProperty(prop.getKey(), prop.getValue());
}
}
return bean;
}
public Object postProcessAfterInitialization(Object bean, String beanName) throws BeansException {
return bean;
}
public void setProperties(Map<String, String> properties) {
this.properties.putAll(properties);
}
}
Now you can add this to your configuration and it will add the properties to DataSource beans.
#Bean
public BeanPostProcessor dataSourcePostProcessor() {
DataSourceConfiguringBeanPostProcessor processor = new DataSourceConfiguringBeanPostProcessor();
Map<String, String> properties = new HashMap<>();
properties.put("defaultRowPrefetch", "15");
properties.put("defaultBatchValue", "25");
processor.setProperties(properties);
return processor;
}
That should do the trick for configuring the datasource.

Camel: use datasource configured by spring-boot

I have a project and in it I'm using spring-boot-jdbc-starter and it automatically configures a DataSource for me.
Now I added camel-spring-boot to project and I was able to successfully create routes from Beans of type RouteBuilder.
But when I'm using sql component of camel it can not find datasource. Is there any simple way to add Spring configured datasource to CamelContext? In samples of camel project they use spring xml for datasource configuration but I'm looking for a way with java config. This is what I tried:
#Configuration
public class SqlRouteBuilder extends RouteBuilder {
#Bean
public SqlComponent sqlComponent(DataSource dataSource) {
SqlComponent sqlComponent = new SqlComponent();
sqlComponent.setDataSource(dataSource);
return sqlComponent;
}
#Override
public void configure() throws Exception {
from("sql:SELECT * FROM tasks WHERE STATUS NOT LIKE 'completed'")
.to("mock:sql");
}
}
I have to publish it because although the answer is in the commentary, you may not notice it, and in my case such a configuration was necessary to run the process.
The use of the SQL component should look like this:
from("timer://dbQueryTimer?period=10s")
.routeId("DATABASE_QUERY_TIMER_ROUTE")
.to("sql:SELECT * FROM event_queue?dataSource=#dataSource")
.process(xchg -> {
List<Map<String, Object>> row = xchg.getIn().getBody(List.class);
row.stream()
.map((x) -> {
EventQueue eventQueue = new EventQueue();
eventQueue.setId((Long)x.get("id"));
eventQueue.setData((String)x.get("data"));
return eventQueue;
}).collect(Collectors.toList());
})
.log(LoggingLevel.INFO,"******Database query executed - body:${body}******");
Note the use of ?dataSource=#dataSource. The dataSource name points to the DataSource object configured by Spring, it can be changed to another one and thus use different DataSource in different routes.
Here is the sample/example code (Java DSL). For this I used
Spring boot
H2 embedded Database
Camel
on startup spring-boot, creates table and loads data. Then camel route, runs "select" to pull the data.
Here is the code:
public void configure() throws Exception {
from("timer://timer1?period=1000")
.setBody(constant("select * from Employee"))
.to("jdbc:dataSource")
.split().simple("${body}")
.log("process row ${body}")
full example in github

Resources