I run spring boot and kafka with auto configuration(via annotattions only) and having props defined in .yaml file, ie:
spring:
kafka:
bootstrap-servers: someserver:9999
consumer:
group-id: mygroup
....
#KafkaListener()
public void receive(ConsumerRecord<?, ?> consumerRecord) {
....
}
And it works fine, spring maps i.e. the field group-id properly.
But when i try configure kafka manually(with ConsumerFactory and ConsumerConfig) using the same yaml file then i run into problem.
In class ConsumerConfig kafka properties are named with . in name, not _ i.e.:
public static final String GROUP_ID_CONFIG = "group.id";
So I cant just load them into map and pass the map to ConsumerFactory because keys are with _ not .
I dont want to do it as ugly as shown in example provided by spring kafka team, when they map props from yaml into config class and then manually assign props in the map to be passed to factory:
#ConfigurationProperties(prefix = "kafka")
public class ConfigProperties {
private String brokerAddress;
private String topic;
private String fooTopic;
public String getBrokerAddress() {
return this.brokerAddress;
}
.....
#Bean
public ProducerFactory<String, String> producerFactory() {
Map<String, Object> props = new HashMap<>();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, this.configProperties.getBrokerAddress());
.....
return new DefaultKafkaProducerFactory<>(props);
}
If you want to use the dotted properties in the yml file, you need to do it like boot does for arbitrary properties that are not directly supported as boot properties:
spring:
kafka:
consumer:
properties:
group.id: foo
i.e. populate a Properties property.
The reason boot provides first class support for some properties is so that IDE editors can provide content-assist, which is not possible with arbitrary properties.
Related
My goal is to read some property from application.yml file and ignore other properties. This is need for testing purposes - I want to read part of config and validate it. I want to ignore other properties because Spring Boot tring to handle it but this is no need for my test.
Example.
Config:
first:
property:
{cipher}value #for example Spring trying to handle cipher and throws exception
second:
property:
value
Class (using String for simplification):
#Configuration
public class Configuration {
#Bean
#ConfigurationProperties(prefix = "second.property")
public String secondProperty() {
return new String();
}
}
Test class:
#SpringBootTest
public class ConfigPropertiesCheckerIntegrationTest {
#Autowired
String secondProperty;
#Test
void checkAutoConfigEndpointProperties() {
assertThat(secondProperty, is(notNullValue()));
}
}
So question: how to ignore first.property and say Spring just skip it?
You can disable {cipher}ed properties decryption with spring.cloud.decrypt-environment-post-processor.enabled=false. E.g:
#SpringBootTest(properties = "spring.cloud.decrypt-environment-post-processor.enabled=false")
I am very new to spring boot and here is how my yaml looks like
configs:
-
collection: col1
groupId: groupId1
topic: topic1
-
collection: col2
groupId: groupId2
topic: topic2
I would like to have 3 classes with their scope defined as a prototype and use the property from the yaml something like
#Scope(value = ConfigurableBeanFactory.SCOPE_PROTOTYPE)
public class mongoListener {
public mongoListener(#Value("${config.collection}") String collectionName) {
//do something
}
}
Similarly, have 2 more classes that would use groupID and topic as well. I am stuck here on how to design.
It will be of great help if anyone can explain how to achieve this.
Please try:
Put your application.yaml that consists of your yaml configuration into /src/main/resources/.
Use the property by #Value annotation like:
public mongoListener(#Value("${configs[0].collection}") String collectionName) {
// collectionName is 'col1'
//do something
}
[EDIT 1]
You can create singleton beans different from each other in name like:
#Configuration
public class MyConfiguration {
#Bean("mongoListener_A")
public mongoListener mongoListener_A(#Value("${configs[0].collection}") String collectionName) {
return new mongoListener(collectionName);
}
#Bean("mongoListener_B")
public mongoListener mongoListener_B(#Value("${configs[1].collection}") String collectionName) {
return new mongoListener(collectionName);
}
// ... and so on
}
[EDIT 2]
You can read your yaml configurations into java objects by using #ConfigurationProperties annotation.
#Data //lombok annotation.
#Component("myConfigs")
#ConfigurationProperties
public class Configs {
private final List<Config> configs;
#Data //lombok annotation.
public static class Config {
private String collection;
private String groupId;
private String topic;
}
}
You can create your mongoListeners from that objects like:
#Configuration
public class MyConfiguration {
#Bean
public List<mongoListener> mongoListeners(#Qualifier("myConfigs") Configs configs) {
return configs.getConfigs().stream()
.map(Config::getCollection)
.map(mongoListener::new)
.collect(Collectors.toList());
}
// ... and so on
}
The beans are still singleton but you can create an arbitrary number of instances depending on your yaml file.
See Also
Use YAML for External Properties - Spring Boot Reference Documentation “How-to” Guides
Spring #Value with arraylist split and obtain the first value
I am injecting properties w/ EnvironmentPostProcessor, they are legacy properties that come from another properties xml file,
how do I access the properties list w/ a REST URL?
#Component
public class SpringCloudConfigRuntimeEnvironmentPostProcessor implements
EnvironmentPostProcessor {
#Override
public void postProcessEnvironment(ConfigurableEnvironment environment, SpringApplication application) {
LinkedHashMap<String, Object> map = new LinkedHashMap<>();
map.put("com.xyz.jdbcUrl", "x.y.z");
MapPropertySource propertySource = new MapPropertySource("defaultProperties", map);
environment.getPropertySources().addLast(propertySource);
and I added it to spring.factories
org.springframework.boot.env.EnvironmentPostProcessor=com.xyz.configservice.config.SpringCloudConfigRuntimeEnvironmentPostProcessor
when I print the environment I do see them being loaded
in application.yml
server:
port: 8888
spring:
application:
name: config-service
I can't find the right syntax to get those properties I loaded, a couple of the many I tried:
http://x.x.x.x:8888/config-service/defaultProperties-default.properties
http://x.x.x.x:8888/defaultProperties-default.properties
I have successfully integrated kafka outbound channle adapter with fixed topic name. Now, i want to make the topic name configurable and hence, want to expose it via application properties.
application.properties contain one of the following entry:
kafkaTopic:testNewTopic
My configuration class looks like below:
#Configuration
#Component
public class KafkaConfig {
#Value("${kafkaTopic}")
private String kafkaTopicName;
#Bean
public String getTopic(){
return kafkaTopicName;
}
#Bean
public KafkaTemplate<String, String> kafkaTemplate() {
return new KafkaTemplate<>(producerFactory());
}
#Bean
public ProducerFactory<String, String> producerFactory() {
Map<String, Object> props = new HashMap<>();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");//this.brokerAddress);
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
// set more properties
return new DefaultKafkaProducerFactory<>(props);
}
}
and in my si-config.xml, i have used the following (ex: topic="getTopic") :
<int-kafka:outbound-channel-adapter
id="kafkaOutboundChannelAdapter" kafka-template="kafkaTemplate"
auto-startup="true" sync="true" channel="inputToKafka" topic="getTopic">
</int-kafka:outbound-channel-adapter>
However, the configuration is unable to pick up the topic name when exposed via bean. But it works fine when i hard code the value of the topic name.
Can someone please suggest what i am doing wrong here?
Does topic within kafka outbound channel accept the value referred as bean?
How do i externalize it as every application using my utility will supply different kafka topic names
The topic attribute is for string value.
However it supports property placeholder resolution:
topic="${kafkaTopic}"
and also SpEL evaluation for aforementioned bean:
topic="#{getTopic}"
Just because this is allowed by the XML parser configuration.
However you may pay attention that KafkaTemplate, which you inject into the <int-kafka:outbound-channel-adapter> has defaultTopic property. Therefore you won't need to worry about that XML.
And one more option available for you is Spring Integration Annotations configuration. Where you can define a #ServiceActivator for the KafkaProducerMessageHandler #Bean:
#ServiceActivator(inputChannel = "inputToKafka")
#Bean
KafkaProducerMessageHandler kafkaOutboundChannelAdapter() {
kafkaOutboundChannelAdapter adapter = new kafkaOutboundChannelAdapter( kafkaTemplate());
adapter.setSync(true);
adapter.setTopicExpression(new LiteralExpression(this.kafkaTopicName));
return adapter;
}
In my spring boot application, I have the main application.yml file. I have a lot of properties, and therefore I would like to have another yml files, which contains the specified properties, grouped by their logic or something.
How can I configure a Bean, to load and work all the properties from one new yml file, and another Bean from another new yml? What is the best practice for it?
I found examples using YamlPropertiesFactoryBean, and this bean can read several resources (yml files), but in an another Bean, when I autowire this YamlPropertiesFactoryBean, I cannot get that specific yml, because the getObject() of this YamlPropertiesFactoryBean will have all the yml resources I added to it.
Finally I have it! This is how it works:
I have a properties configuration class, which loads the yml files:
#Configuration
public class PropertiesConfig {
public static final String PERSONS_FILE_NAME = "persons.yml";
public static final String FOODS_FILE_NAME = "foods.yml";
#Bean
public PropertySourcesPlaceholderConfigurer properties() {
final PropertySourcesPlaceholderConfigurer propertySourcesPlaceholderConfigurer = new PropertySourcesPlaceholderConfigurer();
final YamlPropertiesFactoryBean personsYaml = personsPropertiesFromYamlFile();
final YamlPropertiesFactoryBean foodsYaml = foodsPropertiesFromYamlFile();
propertySourcesPlaceholderConfigurer.setPropertiesArray(personsYaml.getObject(), foodsYaml.getObject());
return propertySourcesPlaceholderConfigurer;
}
#Bean
#Qualifier(PersonsManager.QUALIFIER_NAME)
public YamlPropertiesFactoryBean personsPropertiesFromYamlFile() {
final YamlPropertiesFactoryBean yaml = new YamlPropertiesFactoryBean();
yaml.setResources(new ClassPathResource(PERSONS_FILE_NAME));
return yaml;
}
#Bean
#Qualifier(FoodsManager.QUALIFIER_NAME)
public YamlPropertiesFactoryBean foodsPropertiesFromYamlFile() {
final YamlPropertiesFactoryBean yaml = new YamlPropertiesFactoryBean();
yaml.setResources(new ClassPathResource(FOODS_FILE_NAME));
return yaml;
}
}
And finally, I have two beans (managers), which hold only the corresponding yml properties:
#Component
public class PersonsManager extends YmlPropertiesManager {
public static final String QUALIFIER_NAME = "personsYaml";
#Autowired
public PersonsManager(#Qualifier(QUALIFIER_NAME) YamlPropertiesFactoryBean yamlObject) {
super(yamlObject);
}
...
}
and:
#Component
public class FoodsManager extends YmlPropertiesManager {
public static final String QUALIFIER_NAME = "personsYaml";
#Autowired
public FoodsManager(#Qualifier(QUALIFIER_NAME) YamlPropertiesFactoryBean yamlObject) {
super(yamlObject);
}
...
}
So the main thing here is the #Qualifier annotation.
Beans shouldn't be aware of yaml files. The yaml files are just sources that use used to build up the Spring Environment instance.
If you want specific properties for specific beans, the best way is to prefix those properties in application.yaml, and then use the #ConfigurationProperties with an argument of the prefix you want to use, to bind those properties to the bean in question.
See here:
https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-external-config.html