I would like to add TraceId to all log lines. I do that easily by:
Add traceId to MDC
MDC.put("TRACE_ID", sessionAware.getTraceId()+": ");
Update file-pattern in my application.properties (by adding: "%X{TRACE_ID}"):
logging.pattern.file=-%d{${LOG_DATEFORMAT_PATTERN:-yyyy-MM-dd HH:mm:ss.SSS}} ${LOG_LEVEL_PATTERN:-%5p} ${PID:- } --- [%t] %-40.40logger{39} : %X{TRACE_ID}%m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}
But I would like my CustomSpringBootStarter to set the file pattern. But it doesn't take effect when I update property "logging.pattern.file" from my CustomSpringBootStarter. Does anyone know the solution to this problem?
I have tried to set the "logging.pattern.file" property in the CustomSpringBootStarter's application.properties but it does not work.
I found a solution to the problem, very much inspired by this question.
I created the below EnvironmentPostProcessor in my CustomSpringBootStarter:
public class LifeLogBackConfiguration implements EnvironmentPostProcessor {
#Override
public void postProcessEnvironment(ConfigurableEnvironment environment, SpringApplication application) {
Map<String, String> properties = Collections.unmodifiableMap(Map.of(
"logging.pattern.file", LogConstants.FILE_LOG_PATTERN_LIFE,
"logging.pattern.console", LogConstants.CONSOLE_LOG_PATTERN_LIFE));
PropertySource propertySource = new OriginTrackedMapPropertySource("LogPatternsLife", properties, true);
environment.getPropertySources().addLast(propertySource);
}
}
,and registered it in: resources/META-INF/spring.factories
org.springframework.boot.env.EnvironmentPostProcessor=dk.topdanmark.life.lifespringbootstarter.log.LifeLogBackConfiguration
That solved the problem for me.
Related
A short rant at the beginning, just because it has to be said:
I'm getting tired of reading the terrible documentation of Log4j2 for the umpteenth time and still not finding any solutions for my problems. The documentation is completely outdated, sample code is torn uselessly out of a context that is needed but not explained further and the explanations are consistently insufficient. It shouldn't be that only Log4j2 developers can use Log4j2 in-depth. Frameworks should make the work of other developers easier, which is definitely not the case here. Period and thanks.
Now to my actual problem:
I have a Spring Boot application that is primarily configured with yaml files. The DataSource however is set programmatically so that we have a handle to its bean. Log4j2 is initially set up using yaml configuration as well.
log4j2-spring.yaml:
Configuration:
name: Default
status: warn
Appenders:
Console:
name: Console
target: SYSTEM_OUT
PatternLayout:
pattern: "%d{yyyy-MM-dd HH:mm:ss} %-5level [%t] %c: %msg%n"
Loggers:
Root:
level: warn
AppenderRef:
- ref: Console
Logger:
- name: com.example
level: debug
additivity: false
AppenderRef:
- ref: Console
What I want to do now is to extend this initial configuration programmatically with a JDBC Appender using the already existing connection-pool. According to the documentation, the following should be done:
The recommended approach for customizing a configuration is to extend one of the standard Configuration classes, override the setup method to first do super.setup() and then add the custom Appenders, Filters and LoggerConfigs to the configuration before it is registered for use.
So here is my custom Log4j2Configuration which extends YamlConfiguration:
public class Log4j2Configuration extends YamlConfiguration {
/* private final Log4j2ConnectionSource connectionSource; */ // <-- needs to get somehow injected
public Log4j2Configuration(LoggerContext loggerContext, ConfigurationSource configSource) {
super(loggerContext, configSource);
}
#Override
public void setup() {
super.setup();
}
#Override
protected void doConfigure() {
super.doConfigure();
LoggerContext context = (LoggerContext) LogManager.getContext(false);
Configuration config = context.getConfiguration();
ColumnConfig[] columns = new ColumnConfig[]{
//...
};
Appender jdbcAppender = JdbcAppender.newBuilder()
.setName("DataBase")
.setTableName("application_log")
// .setConnectionSource(connectionSource)
.setColumnConfigs(columns)
.build();
jdbcAppender.start();
config.addAppender(jdbcAppender);
AppenderRef ref = AppenderRef.createAppenderRef("DataBase", null, null);
AppenderRef[] refs = new AppenderRef[]{ref};
/* Deprecated, but still in the Log4j2 documentation */
LoggerConfig loggerConfig = LoggerConfig.createLogger(
false,
Level.TRACE,
"com.example",
"true",
refs,
null,
config,
null);
loggerConfig.addAppender(jdbcAppender, null, null);
config.addLogger("com.example", loggerConfig);
context.updateLoggers();
}
}
The ConnectionSource exists as an implementation of AbstractConnectionSource in the Spring context and still needs to be injected into the Log4j2Configuration class. Once I know how the configuration process works I can try to find a solution for this.
Log4j2ConnectionSource:
#Configuration
public class Log4j2ConnectionSource extends AbstractConnectionSource {
private final DataSource dataSource;
public Log4j2ConnectionSource(#Autowired #NotNull DataSource dataSource) {
this.dataSource = dataSource;
}
#Override
public Connection getConnection() throws SQLException {
return dataSource.getConnection();
}
}
And finally the ConfigurationFactory as described here in the documentation (It is interesting that the method getConfiguration calls with new MyXMLConfiguration(source, configFile) a constructor that doesn't exist. Is witchcraft at play here?).
Log4j2ConfigurationFactory:
#Order(50)
#Plugin(name = "Log4j2ConfigurationFactory", category = ConfigurationFactory.CATEGORY)
public class Log4j2ConfigurationFactory extends YamlConfigurationFactory {
#Override
public Configuration getConfiguration(LoggerContext context, ConfigurationSource configSource) {
return new Log4j2Configuration(context, configSource);
}
#Override
public String[] getSupportedTypes() {
return new String[]{".yml", "*"};
}
}
Now that the set up is more or less done, the running Log4j2 configuration needs somehow to be updated. So somebody should call doConfigure() within Log4j2Configuration. Log4j2 doesn't seem to do anything here on its own. Spring Boot doesn't do anything either. And I unfortunately don't have any plan what do at all.
Therefore my request:
Can anyone please explain to me how to get Log4j2 to update its configuration?
Many thanks for any help.
I've a filter (OncePerRequestFilter) which basically intercepts incoming request and logs traceId, spanId etc. which works well,
this filter lies in a common module which is included in other projects to avoid including spring sleuth dependency in all of my micro-services, the reason why I've created it as a library because any changes to library will be common to all modules.
Now I've to add a new propagation key which need to be propagated to all services via http headers like trace and spanId for that I've extracted current span from HttpTracing and added a baggage key to it (as shown below)
Span span = httpTracing.tracing().tracer().currentSpan();
String corelationId =
StringUtils.isEmpty(request.getHeader(CORELATION_ID))
? "n/a"
: request.getHeader(CORELATION_ID);
ExtraFieldPropagation.set(CUSTOM_TRACE_ID_MDC_KEY_NAME, corelationId);
span.annotate("baggage_set");
span.tag(CUSTOM_TRACE_ID_MDC_KEY_NAME, corelationId);
I've added propagation-keys and whitelisted-mdc-keys to my application.yml (with my library) file like below
spring:
sleuth:
propagation-keys:
- x-corelationId
log:
slf4j:
whitelisted-mdc-keys:
- x-corelationId
After making this change in filter the corelationId is not available when I make a http call to another service with same app, basically keys are not getting propagated.
In your library you can implement ApplicationEnvironmentPreparedEvent listener and add the configuration you need there
Ex:
#Component
public class CustomApplicationListener implements ApplicationListener<ApplicationEvent> {
private static final Logger log = LoggerFactory.getLogger(LagortaApplicationListener.class);
public void onApplicationEvent(ApplicationEvent event) {
if (event instanceof ApplicationEnvironmentPreparedEvent) {
log.debug("Custom ApplicationEnvironmentPreparedEvent Listener");
ApplicationEnvironmentPreparedEvent envEvent = (ApplicationEnvironmentPreparedEvent) event;
ConfigurableEnvironment env = envEvent.getEnvironment();
Properties props = new Properties();
props.put("spring.sleuth.propagation-keys", "x-corelationId");
props.put("log.slf4j.whitelisted-mdc-keys:", "x-corelationId");
env.getPropertySources().addFirst(new PropertiesPropertySource("custom", props));
}
}
}
Then in your microservice you will register this custom listener
public static void main(String[] args) {
ConfigurableApplicationContext context = new SpringApplicationBuilder(MyApplication.class)
.listeners(new CustomApplicationListener()).run();
}
I've gone through documentation and seems like I need to add spring.sleuth.propagation-keys and whitelist them by using spring.sleuth.log.slf4j.whitelisted-mdc-keys
Yes you need to do this
is there another way to add these properties in common module so that I do not need to include them in each and every micro services.
Yes, you can use Spring Cloud Config server and a properties file called application.yml / application.properties that would set those properties for all microservices
The answer from Mahmoud works great when you want register the whitelisted-mdc-keys programatically.
An extra tip when you need these properties also in a test, then you can find the anwser in this post: How to register a ApplicationEnvironmentPreparedEvent in Spring Test
#ConfigurationProperties locations is deprecated in Spring Boot 1.4.x and option is now removed in 1.5.x
I was using it like this: BucketTestConfig.java
For now with deprecation, I'm trying to set the system property spring.config.location for both production code and test code as an alternative.
./gradlew clean test is still failing although I set the system property.
What is the best alternative for deprecated #ConfigurationProperties locations in this case?
UPDATE:
Using SpringApplicationBuilder.properties() doesn't work in the test (BucketTestRepositoryTests).
Using SpringApplicationBuilder.listeners() doesn't work in the test (BucketTestRepositoryTests), either.
UPDATE (2nd):
There was no reason to depend on #ConfigurationProperties in my case, so I went with Yaml instead as follows: https://github.com/izeye/spring-boot-throwaway-branches/commit/a1290672dceea98706b1a258f8a17e2628ea01ee
So this question's title is invalid and this question can be deleted.
Follow this thread for more information.
Basically, this thread suggests two options
First option is to set spring.config.name to a list of the files you want to load:
new SpringApplicationBuilder(Application.class)
.properties("spring.config.name=application,mine")
.run(args);
Second options is to add listeners
new SpringApplicationBuilder(SanityCheckApplication.class)
.listeners(new LoadAdditionalProperties())
.run(args);
Content of LoadAdditionalProperties
#Component
public class LoadAdditionalProperties implements ApplicationListener<ApplicationEnvironmentPreparedEvent> {
private ResourceLoader loader = new DefaultResourceLoader();
#Override
public void onApplicationEvent(ApplicationEnvironmentPreparedEvent event) {
try {
Resource resource = loader.getResource("classpath:mine.properties");
PropertySource<?> propertySource = new PropertySourcesLoader().load(resource);
event.getEnvironment().getPropertySources().addLast(propertySource);
} catch (IOException ex) {
throw new IllegalStateException(ex);
}
}
}
I'm using Spring Cloud Config in my spring-boot application and I need to write some custom code to handle properties to be read from my corporate password vault when property is flagged as such. I know spring cloud supports Hashicorp Vault, but that's not the one in case.
I don't want to hard-code specific properties to be retrieved from a different source, for example, I would have a properties file for application app1 with profile dev with values:
spring.datasource.url=jdbc:mysql://localhost/test
spring.datasource.username=dbuser
spring.datasource.password=dbpass
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
but for some other profiles such as prod, I would have:
spring.datasource.url=jdbc:mysql://localhost/test
spring.datasource.username=prod-user
spring.datasource.password=[[vault]]
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
So I need the custom property vault to intercept the property loaded whenever it finds a returned value equals to [[vault]] (or some other type of flag), and query from the corporate vault instead. In this case, my custom property loader would find the value of property spring.datasource.password from the corporate password vault. All other properties would still be returned as-is from values loaded by standard spring cloud config client.
I would like to do that using annotated code only, no XML configuration.
You can implement your own PropertySourceLocator and add entry to
spring.factories in directory META-INF.
#spring.factories
org.springframework.cloud.bootstrap.BootstrapConfiguration=/
foo.bar.MyPropertySourceLocator
Then you can you can refer to keys in your corporate password vault like a normal properties in spring.
spring.datasource.url=jdbc:mysql://localhost/test
spring.datasource.username=prod-user
spring.datasource.password=${lodaded.password.from.corporate.vault}
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
Implementation by HasiCorp: VaultPropertySourceLocatorSupport
While trying to solve the identical problem, I believe that I have come to work-around that may be acceptable.
Here is my solution below.
public class JBossVaultEnvironmentPostProcessor implements EnvironmentPostProcessor {
#Override
public void postProcessEnvironment(ConfigurableEnvironment environment, SpringApplication application) {
MutablePropertySources propertySources = environment.getPropertySources();
Map<String, String> sensitiveProperties = propertySources.stream()
.filter(propertySource -> propertySource instanceof EnumerablePropertySource)
.map(propertySource -> (EnumerablePropertySource<?>) propertySource)
.map(propertySource -> {
Map<String, String> vaultProperties = new HashMap<>();
String[] propertyNames = propertySource.getPropertyNames();
for (String propertyName : propertyNames) {
String propertyValue = propertySource.getProperty(propertyName).toString();
if (propertyValue.startsWith("VAULT::")) {
vaultProperties.put(propertyName, propertyValue);
}
}
return vaultProperties;
})
.reduce(new HashMap<>(), (m1, m2) -> {
m1.putAll(m2);
return m1;
});
Map<String, Object> vaultProperties = new HashMap<>();
sensitiveProperties.keySet().stream()
.forEach(key -> {
vaultProperties.put(key, VaultReader.readAttributeValue(sensitiveProperties.get(key)));
});
propertySources.addFirst(new MapPropertySource("vaultProperties", vaultProperties));
}
I need to set some specific Oracle JDBC connection properties in order to speed up batch INSERTs (defaultBatchValue) and mass SELECTs (defaultRowPrefetch).
I got suggestions how to achieve this with DBCP (Thanks to M. Deinum) but I would like to:
keep the default Tomcat jdbc connection pool
keep application.yml for configuration
I was thinking about a feature request to support spring.datasource.custom_connection_properties or similar in the future and because of this tried to pretent this was already possible. I did this by passing the relevant information while creating the DataSource and manipulated the creation of the DataSource like this:
#Bean
public DataSource dataSource() {
DataSource ds = null;
try {
Field props = DataSourceBuilder.class.getDeclaredField("properties");
props.setAccessible(true);
DataSourceBuilder builder = DataSourceBuilder.create();
Map<String, String> properties = (Map<String, String>) props.get(builder);
properties.put("defaultRowPrefetch", "1000");
properties.put("defaultBatchValue", "1000");
ds = builder.url( "jdbc:oracle:thin:#xyz:1521:abc" ).username( "ihave" ).password( "wonttell" ).build();
properties = (Map<String, String>) props.get(builder);
log.debug("properties after: {}", properties);
} ... leaving out the catches ...
}
log.debug("We are using this datasource: {}", ds);
return ds;
}
In the logs I can see that I am creating the correct DataSource:
2016-01-18 14:40:32.924 DEBUG 31204 --- [ main] d.a.e.a.c.config.DatabaseConfiguration : We are using this datasource: org.apache.tomcat.jdbc.pool.DataSource#19f040ba{ConnectionPool[defaultAutoCommit=null; ...
2016-01-18 14:40:32.919 DEBUG 31204 --- [ main] d.a.e.a.c.config.DatabaseConfiguration : properties after: {password=wonttell, driverClassName=oracle.jdbc.OracleDriver, defaultRowPrefetch=1000, defaultBatchValue=1000, url=jdbc:oracle:thin:#xyz:1521:abc, username=ihave}
The actuator shows me that my code replaced the datasource:
But the settings are not activated, which I can see while profiling the application. The defaultRowPrefetch is still at 10 which causes my SELECTs to be much slower than they would be if 1000 was activated.
Setting the pools connectionProperties should work. Those will be passed to the JDBC driver. Add this to application.properties:
spring.datasource.connectionProperties: defaultRowPrefetch=1000;defaultBatchValue=1000
Edit (some background information):
Note also that you can configure any of the DataSource implementation
specific properties via spring.datasource.*: refer to the
documentation of the connection pool implementation you are using for
more details.
source: spring-boot documentation
As Spring Boot is EOL for a long time I switched to Spring Boot 2.1 with its new default connection pool Hikari. Here the solution is even more simply and can be done in the application.properties or (like shown here) application.yml:
spring:
datasource:
hikari:
data-source-properties:
defaultRowPrefetch: 1000
(In a real-life config there would be several other configuration items but as they are not of interest for the question asked I simply left them out in my example)
Some additional information to complement the answer by #Cyril. If you want to upvote use his answer, not mine.
I was a little bit puzzled how easy it is to set additional connection properties that in the end get used while creating the database connection. So I did a little bit of research.
spring.datasource.connectionProperties is not mentioned in the reference. I created an issue because of this.
If I had used the Spring Boot YML editor, I would have seen which properties are supported. Here is what STS suggests when you create an application.yml and hit Ctrl+Space:
The dash does not matter because of relaxed binding but if you interpret it literally the propertys name is spring.datasource.connection-properties.
The correct setup in application.yml looks like this:
spring:
datasource:
connection-properties: defaultBatchValue=1000;defaultRowPrefetch=1000
...
This gets honored which is proven by my perf4j measurements of mass SELECTs.
Before:
2016-01-19 08:58:32.604 INFO 15108 --- [ main]
org.perf4j.TimingLogger : start[1453190311227]
time[1377] tag[get elements]
After:
2016-01-19 08:09:18.214 INFO 9152 --- [ main]
org.perf4j.TimingLogger : start[1453187358066]
time[147] tag[get elements]
The time taken to complete the SQL statement drops from 1377ms to 147, which is an enormous gain in performance.
After digging around in the Tomcat code for a bit, I found that the dataSource.getPoolProperties().getDbProperties() is the Properties object that will actually get used to generate connections for the pool.
If you use the BeanPostProcessor approach mentioned by #m-deinum, but instead use it to populate the dbProperties like so, you should be able to add the properties in a way that makes them stick and get passed to the Oracle driver.
import java.util.Properties;
import org.apache.tomcat.jdbc.pool.DataSource;
import org.apache.tomcat.jdbc.pool.PoolConfiguration;
#Component
public class OracleConfigurer implements BeanPostProcessor {
#Override
public Object postProcessBeforeInitialization(Object bean, String name) throws BeansException {
if (bean instanceof DataSource) {
DataSource dataSource = (DataSource)bean;
PoolConfiguration configuration = dataSource.getPoolProperties();
Properties properties = configuration.getDbProperties();
if (null == properties) properties = new Properties();
properties.put("defaultRowPrefetch", 1000);
properties.put("defaultBatchValue", 1000);
configuration.setDbProperties(properties);
}
return bean;
}
#Override
public Object postProcessAfterInitialization(Object bean, String name) throws BeansException {
return bean;
}
}