I'm trying to read YAML properties file using the #PropertySource annotation in Spring Boot.
configuration.yaml file has about 7.5K lines of data in the below format:
# Configuration for privilege management
---
role-configuration:
roles:
- name: Agent
groups:
-name: privilege-group
group-configuration:
groups:
- name: privilege-group
privilege-configuration:
privileges:
- name: admin-dashboard-view
description: View Admin dashboard
groups:
- name: privilege-group
- name: admin-dashboard-edit
description: Edit Admin dashboard
groups:
- name: privilege-group
....
...
..
Configuration bean and YAML specific PropertySourceFactory has been implemented by following this link
PrivilegeProvider.java
#Configuration
#ConfigurationProperties(prefix = "privilege-configuration")
#PropertySource(value = "classpath:security/configuration.yaml", factory = YamlPropertySourceFactory.class)
#Data # lombok annotation for generating getter/setter and other helper functions
public class PrivilegeProvider {
private List<Privilege> privileges;
}
YamlPropertySourceFactory.java
public class YamlPropertySourceFactory implements PropertySourceFactory {
#Override
public PropertySource<?> createPropertySource(String name, EncodedResource encodedResource) throws IOException {
YamlPropertiesFactoryBean factory = new YamlPropertiesFactoryBean();
factory.setResources(encodedResource.getResource());
Properties properties = factory.getObject();
return new PropertiesPropertySource(encodedResource.getResource().getFilename(), properties);
}
}
All the data is successfully loaded from YAML file but it is taking ~5-7 minutes to load, which is quite a lot of time given the file size.
Can this be optimized? or is there any other way in which I can implement the same?
Related
I am writing a integration test for rest api entry.The api require to initialize database before the test.However it gives an error which showing it cant find any path to my sql file.
#SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
#AutoConfigureMockMvc
#TestExecutionListeners(listeners = { SqlScriptsTestExecutionListener.class })
class OrderServiceApplicationTests {
#Autowired
private MockMvc mockMvc;
#Autowired
private ObjectMapper objectMapper;
#Autowired
private JobRepository jobRepository;
//#Sql("INSERT INTO user_info(name) VALUES \"alex\" ")
#Test
#Sql("{/resources/schema.sql}")
void shouldCreatePost() throws Exception {
JobOrder job=jobRepository.save(createjob());
String request=objectMapper.writeValueAsString(job);
mockMvc.perform(MockMvcRequestBuilders.post("/Job/Joborder")
.contentType(MediaType.APPLICATION_JSON)
.content(request))
.andExpect(status().is2xxSuccessful());
}
JobOrder createjob(){
JobOrder job=new JobOrder();
job.setTitle("Hiring software engineer");
job.setDescription("responsible for developing and maintaining mobile app");
job.setRequirement("Need to know basic sql springboot,2 years exp");
job.setSalary(234);
job.setOrder_id(1);
return job;
}
}
schema.sql:
INSERT INTO user_info (name) VALUES ('India');
and i got an error:
org.springframework.jdbc.datasource.init.CannotReadScriptException: Cannot read SQL script from class path resource [com/example/OrderService/{/resources/schema.sql}]
at org.springframework.jdbc.datasource.init.ScriptUtils.executeSqlScript(ScriptUtils.java:239)
at org.springframework.jdbc.datasource.init.ResourceDatabasePopulator.populate(ResourceDatabasePopulator.java:254)
at org.springframework.jdbc.datasource.init.DatabasePopulatorUtils.execute(DatabasePopulatorUtils.java:54)
at org.springframework.jdbc.datasource.init.ResourceDatabasePopulator.execute(ResourceDatabasePopulator.java:269)
at org.springframewo
My properties:
spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver
spring.datasource.url=jdbc:mysql://localhost:3306/test
spring.datasource.username=***
spring.datasource.password=*******
spring.jpa.show-sql=true
spring.jpa.hibernate.ddl-auto=create-drop
spring.jpa.defer-datasource-initialization=true
spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.MySQL5Dialect
spring.sql.init.mode=always
My Path:
I dont know what is the problem on my path.And i am wondering if there are problem on my test script or strategy
Several things are wrong with your #Sql("{/resources/schema.sql}").
the { and } do not belong into the path, why did you add it in the first place?
You don't need to include the /resources directory in the path. To find out what the path of your file is, open the target/test-classes and have a look. Everything that resides in the resources directory will be placed in the root directory. If your file was in resources/sql/test.sql it would be available under sql/test.sql.
Solution
#Sql("schema.sql")
A short rant at the beginning, just because it has to be said:
I'm getting tired of reading the terrible documentation of Log4j2 for the umpteenth time and still not finding any solutions for my problems. The documentation is completely outdated, sample code is torn uselessly out of a context that is needed but not explained further and the explanations are consistently insufficient. It shouldn't be that only Log4j2 developers can use Log4j2 in-depth. Frameworks should make the work of other developers easier, which is definitely not the case here. Period and thanks.
Now to my actual problem:
I have a Spring Boot application that is primarily configured with yaml files. The DataSource however is set programmatically so that we have a handle to its bean. Log4j2 is initially set up using yaml configuration as well.
log4j2-spring.yaml:
Configuration:
name: Default
status: warn
Appenders:
Console:
name: Console
target: SYSTEM_OUT
PatternLayout:
pattern: "%d{yyyy-MM-dd HH:mm:ss} %-5level [%t] %c: %msg%n"
Loggers:
Root:
level: warn
AppenderRef:
- ref: Console
Logger:
- name: com.example
level: debug
additivity: false
AppenderRef:
- ref: Console
What I want to do now is to extend this initial configuration programmatically with a JDBC Appender using the already existing connection-pool. According to the documentation, the following should be done:
The recommended approach for customizing a configuration is to extend one of the standard Configuration classes, override the setup method to first do super.setup() and then add the custom Appenders, Filters and LoggerConfigs to the configuration before it is registered for use.
So here is my custom Log4j2Configuration which extends YamlConfiguration:
public class Log4j2Configuration extends YamlConfiguration {
/* private final Log4j2ConnectionSource connectionSource; */ // <-- needs to get somehow injected
public Log4j2Configuration(LoggerContext loggerContext, ConfigurationSource configSource) {
super(loggerContext, configSource);
}
#Override
public void setup() {
super.setup();
}
#Override
protected void doConfigure() {
super.doConfigure();
LoggerContext context = (LoggerContext) LogManager.getContext(false);
Configuration config = context.getConfiguration();
ColumnConfig[] columns = new ColumnConfig[]{
//...
};
Appender jdbcAppender = JdbcAppender.newBuilder()
.setName("DataBase")
.setTableName("application_log")
// .setConnectionSource(connectionSource)
.setColumnConfigs(columns)
.build();
jdbcAppender.start();
config.addAppender(jdbcAppender);
AppenderRef ref = AppenderRef.createAppenderRef("DataBase", null, null);
AppenderRef[] refs = new AppenderRef[]{ref};
/* Deprecated, but still in the Log4j2 documentation */
LoggerConfig loggerConfig = LoggerConfig.createLogger(
false,
Level.TRACE,
"com.example",
"true",
refs,
null,
config,
null);
loggerConfig.addAppender(jdbcAppender, null, null);
config.addLogger("com.example", loggerConfig);
context.updateLoggers();
}
}
The ConnectionSource exists as an implementation of AbstractConnectionSource in the Spring context and still needs to be injected into the Log4j2Configuration class. Once I know how the configuration process works I can try to find a solution for this.
Log4j2ConnectionSource:
#Configuration
public class Log4j2ConnectionSource extends AbstractConnectionSource {
private final DataSource dataSource;
public Log4j2ConnectionSource(#Autowired #NotNull DataSource dataSource) {
this.dataSource = dataSource;
}
#Override
public Connection getConnection() throws SQLException {
return dataSource.getConnection();
}
}
And finally the ConfigurationFactory as described here in the documentation (It is interesting that the method getConfiguration calls with new MyXMLConfiguration(source, configFile) a constructor that doesn't exist. Is witchcraft at play here?).
Log4j2ConfigurationFactory:
#Order(50)
#Plugin(name = "Log4j2ConfigurationFactory", category = ConfigurationFactory.CATEGORY)
public class Log4j2ConfigurationFactory extends YamlConfigurationFactory {
#Override
public Configuration getConfiguration(LoggerContext context, ConfigurationSource configSource) {
return new Log4j2Configuration(context, configSource);
}
#Override
public String[] getSupportedTypes() {
return new String[]{".yml", "*"};
}
}
Now that the set up is more or less done, the running Log4j2 configuration needs somehow to be updated. So somebody should call doConfigure() within Log4j2Configuration. Log4j2 doesn't seem to do anything here on its own. Spring Boot doesn't do anything either. And I unfortunately don't have any plan what do at all.
Therefore my request:
Can anyone please explain to me how to get Log4j2 to update its configuration?
Many thanks for any help.
Any Help how can I add state store on Spring cloud
I always receive this error "nested exception is org.springframework.kafka.KafkaException: Could not start stream: ; nested exception is org.apache.kafka.streams.errors.TopologyException: Invalid topology: StateStore myStore is not added yet."
Here is the bean definition however it never works
#Bean
public StoreBuilder storeBuilder() {
KeyValueBytesStoreSupplier storeSupplier = Stores.persistentKeyValueStore("mystore");
StoreBuilder<KeyValueStore<String, MyData>> storeBuilder = Stores.keyValueStoreBuilder(storeSupplier, Serdes.String(), StreamsSerde.MyDataSerde());
return storeBuilder;
}
Here is the Serde
public static final class MyDataSerde extends Serdes.WrapperSerde<MyData> {
public MyDataSerde() {
super(new JsonSerializer<>(), new JsonDeserializer<>(MyData.class));
}
}
Here is the data class
public class MyData {
private String name;
private String course;
}
Here is the spring cloud dependencies
springBootVersion = "2.2.5.RELEASE"
set('springCloudVersion', "Hoxton.SR3")
implementation group:"org.springframework.cloud", name: "spring-cloud-stream"
implementation group: "org.springframework.cloud", name: "spring-cloud-stream-binder-kafka-streams"
implementation group: "org.springframework.cloud", name: "spring-cloud-starter-stream-kafka"
You need to add state stores like this when you have to use the lower level processor or transformer API. Did you try to add the state store to your process or transform method call? Here is a test that works. Take a look at the process call and the way the state stores are passed along.
I found a solution to add the store programmatically on this article
public void initializeStateStores() throws Exception {
StreamsBuilderFactoryBean streamsBuilderFactoryBean =
applicationContext.getBean("&stream-builder-requestListener", StreamsBuilderFactoryBean.class);
StreamsBuilder streamsBuilder = streamsBuilderFactoryBean.getObject();
StoreBuilder<KeyValueStore<String, Long>> keyValueStoreBuilder = Stores.keyValueStoreBuilder(Stores.persistentKeyValueStore(stateStoreName), Serdes.String(), Serdes.Long());
streamsBuilder.addStateStore(keyValueStoreBuilder);
}
https://medium.com/#daniyaryeralin/utilizing-kafka-streams-processor-api-and-implementing-custom-aggregator-6cb23d00eaa7
I have custom.yaml file with content:
refill-interval-millis: 1000
endpoints:
- path: /account/all
rate-limit: 10
- path: /account/create
rate-limit: 20
and my class to read that file:
#Component
#PropertySource("classpath:custom.yaml")
#ConfigurationProperties
public class Properties {
private int refillIntervalMillis;
private List<Endpoint> endpoints = new ArrayList<>();
// getters/setters
public static class Endpoint {
private String path;
private int rateLimit;
// getters/setters
}
}
So, when I run my code, refillIntervalMillis is set properly, but endpoints list is empty. Can not get why?
You cannot use #PropertySource directly for YAML-files: YAML Shortcomings
You have 2 simple options:
use application.yml and Spring Boot loads if by default (and you can use application-xxx.yml and set #ActiveProfiles(value = "xxx");
load YAML-file manually:
#Bean
public static PropertySourcesPlaceholderConfigurer properties() {
var propertySourcesPlaceholderConfigurer = new PropertySourcesPlaceholderConfigurer();
var yamlPropertiesFactoryBean = new YamlPropertiesFactoryBean();
yamlPropertiesFactoryBean.setResources(new ClassPathResource("custom.yaml"));
propertySourcesPlaceholderConfigurer.setProperties(yamlPropertiesFactoryBean.getObject());
return propertySourcesPlaceholderConfigurer;
}
I have this specific problem with my yml configuration file.
I have a multi-module maven project as follows:
app
|-- core
|-- web
|-- app
I have this configuration file in core project
#Configuration
#PropertySource("core-properties.yml")
public class CoreConfig {
}
And this mapping:
#Component
#ConfigurationProperties(prefix = "some.key.providers.by")
#Getter
#Setter
public class ProvidersByMarket {
private Map<String, List<String>> market;
}
Here are my core-properties.yml
some.key.providers:
p1: 'NAME1'
p2: 'NAME2'
some.key.providers.by.market:
de:
- ${some.key.providers.p1}
- ${some.key.providers.p2}
gb:
- ${some.key.providers.p1}
When I load the file via profile activation, for example, rename the file to application-core-properties.yml and then -Dspring.profiles.active=core-propertiesit does work however if when I try to load the file via #PropertySource("core-properties.yml") it does not and I get the following error:
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2019-03-27 10:07:36.397 -ERROR 13474|| --- [ restartedMain] o.s.b.d.LoggingFailureAnalysisReporter :
***************************
APPLICATION FAILED TO START
***************************
Description:
Failed to bind properties under 'some.key.providers.by.market' to java.util.Map<java.lang.String, java.util.List<java.lang.String>>:
Reason: No converter found capable of converting from type [java.lang.String] to type [java.util.Map<java.lang.String, java.util.List<java.lang.String>>]
Action:
Update your application's configuration
Process finished with exit code 1
Bacouse you don't have equivalent properties stracture,
example
spring:
profiles: test
name: test-YAML
environment: test
servers:
- www.abc.test.com
- www.xyz.test.com
---
spring:
profiles: prod
name: prod-YAML
environment: production
servers:
- www.abc.com
- www.xyz.com
And config class should be
#Configuration
#EnableConfigurationProperties
#ConfigurationProperties
public class YAMLConfig {
private String name;
private String environment;
private List<String> servers = new ArrayList<>();
// standard getters and setters
I have resolved the issue implementing the following PropertySourceFactory detailed described in here
import java.io.FileNotFoundException;
import java.io.IOException;
import java.util.Properties;
import org.springframework.beans.factory.config.YamlPropertiesFactoryBean;
import org.springframework.core.env.PropertiesPropertySource;
import org.springframework.core.env.PropertySource;
import org.springframework.core.io.support.EncodedResource;
import org.springframework.core.io.support.PropertySourceFactory;
import org.springframework.lang.Nullable;
public class YamlPropertySourceFactory implements PropertySourceFactory {
#Override
public PropertySource<?> createPropertySource(#Nullable String name, EncodedResource resource) throws IOException {
Properties propertiesFromYaml = loadYamlIntoProperties(resource);
String sourceName = name != null ? name : resource.getResource().getFilename();
return new PropertiesPropertySource(sourceName, propertiesFromYaml);
}
private Properties loadYamlIntoProperties(EncodedResource resource) throws FileNotFoundException {
try {
YamlPropertiesFactoryBean factory = new YamlPropertiesFactoryBean();
factory.setResources(resource.getResource());
factory.afterPropertiesSet();
return factory.getObject();
} catch (IllegalStateException e) {
// for ignoreResourceNotFound
Throwable cause = e.getCause();
if (cause instanceof FileNotFoundException)
throw (FileNotFoundException) e.getCause();
throw e;
}
}
}
I had a similar problem and found a workaround like this:
diacritic:
isEnabled: true
chars: -> I wanted this to be parsed to map but it didn't work
ą: a
ł: l
ę: e
And my solution so far:
diacritic:
isEnabled: true
chars[ą]: a -> these ones could be parsed to Map<String, String>
chars[ł]: l
chars[ę]: e