I have this Spring boot application.properties
list1=valueA,valueB
list2=valueC
list3=valueD,valueE
topics=list1,list2,list3
What I'm trying to do is to use in the topics element of #KafkaListener annotation the values of the values of topics property
Using the expression
#KafkaListener(topics={"#{'${topics}'.split(',')}"})
I get list1,list2,list3 as separated string
How can I loop on this list in order to get valueA,valueB,valueC,valueD,valueE?
Edit: I must parse topics properties in order that #KafkaListener registers for consuming message from topics valueA,valueB,valueC, etc.
I read that is possible call a method in this way:
#KafkaListener(topics="#parse(${topics})")
So, I wrote this method:
public String[] parse(String s) {
ExpressionParser parser = new SpelExpressionParser();
return Arrays.stream(s.split(",").map(key -> (String)(parser.parse(key).getValue())).toArray(String[]::new);
}
But the parse method is not invoked
So, I tried directly to do this into annotations
in this way:
#KafkaListener(topics="#{Arrays.stream('${topics}'.split(',')).map(key->${key}).toArray(String[]::new)}")
But also this solution give me errors.
Edit 2:
Modifying in this way the method is invoked
#KafkaListener(topics="parse()")
#Bean
public String[] parse(String s) {
...
}
The problems is how to get "topics" props inside the method
You can't invoke arbitrary methods like that; you need to reference a bean #someBean.parse(...); using #parse requires registering a static method as a function.
However, this works for me and is much simpler:
list1=valueA,valueB
list2=valueC
list3=valueD,valueE
topics=${list1},${list2},${list3}
and
#KafkaListener(id = "so64390079", topics = "#{'${topics}'.split(',')}")
EDIT
If you can't use placeholders in topics, this works...
#SpringBootApplication
public class So64390079Application {
public static void main(String[] args) {
SpringApplication.run(So64390079Application.class, args);
}
#KafkaListener(id = "so64390079", topics = "#{#parser.parse('${topics}')}")
public void listen(String in) {
System.out.println(in);
}
}
#Component
class Parser implements EnvironmentAware {
private Environment environmment;
#Override
public void setEnvironment(Environment environment) {
this.environmment = environment;
}
public String[] parse(String[] topics) {
StringBuilder sb = new StringBuilder();
for (String topic : topics) {
sb.append(this.environmment.getProperty(topic));
sb.append(',');
}
return StringUtils.commaDelimitedListToStringArray(sb.toString().substring(0, sb.length() - 1));
}
}
Related
I would like to create a spring boot application that reads from several Kafka topics. I realise I can create a comma separated list of topics on my appliation.properties, however I would like the topic names to be listed separately for readability and so I can use each topic name to work out how to process the message.
I've found the following questions, but they all have the topics listed as a comma separated array:
Consume multiple topics in one listener in spring boot kafka
Using multiple topic names with KafkaListener annotation
Enabling #KafkaListener to take in variable topic names from application.yml file
Pass array list of topic names to #KafkaListener
The closest I've come is with the following:
application.properties
kafka.topic1=topic1
kafka.topic2=topic2
KafkaConsumer
#KafkaListener(topics = "#{'${kafka.topic1}'},#{'${kafka.topic2}'}")
public void receive(#Header(KafkaHeaders.RECEIVED_TOPIC) String topic,
#Header(required = false, name= KafkaHeaders.RECEIVED_MESSAGE_KEY) String key,
#Payload(required = false) String payload) throws IOException {
}
This gives the error:
Caused by: org.apache.kafka.common.errors.InvalidTopicException: Invalid topics: [topic1,topic2]
I realise I need it to be {"topic1", "topic2} but I can't work out how.
Having the annotation #KafkaListener(topics = "#{'${kafka.topic1}'}") correctly subscribes to the first topic. And if I change it to #KafkaListener(topics = "#{'${kafka.topic2}'}") I can correctly subscribe to the second topic.
It's just the creating of the array of topics in the annotation that I can't fathom.
Any help would be wonderful
#KafkaListener(id = "so71497475", topics = { "${kafka.topic1}", "${kafka.topic2}" })
EDIT
And this is a more sophisticated technique which would allow you to add more topics without changing any code:
#SpringBootApplication
#EnableConfigurationProperties
public class So71497475Application {
public static void main(String[] args) {
SpringApplication.run(So71497475Application.class, args);
}
#KafkaListener(id = "so71497475", topics = "#{#myProps.kafkaTopics}")
void listen(String in) {
System.out.println(in);
}
#Bean // This will add the topics to the broker if not present
KafkaAdmin.NewTopics topics(MyProps props) {
return new KafkaAdmin.NewTopics(props.getTopics().stream()
.map(t -> TopicBuilder.name(t).partitions(1).replicas(1).build())
.toArray(size -> new NewTopic[size]));
}
}
#ConfigurationProperties("my.kafka")
#Component
class MyProps {
private List<String> topics = new ArrayList<>();
public List<String> getTopics() {
return this.topics;
}
public void setTopics(List<String> topics) {
this.topics = topics;
}
public String[] getKafkaTopics() {
return this.topics.toArray(new String[0]);
}
}
my.kafka.topics[0]=topic1
my.kafka.topics[1]=topic2
my.kafka.topics[2]=topic3
so71497475: partitions assigned: [topic1-0, topic2-0, topic3-0]
If you have your topics configured as comma seperated like:
kafka.topics = topic1,topic2
In this case you can simply use:
#KafkaListener(topics = "#{'${kafka.topics}'.split(',')}")
void listen(){}
I am working on project where I need to validate consumer group is created on topic or not. Is there any way in boldSpring Kafkabold to validate it
Currently, I haven't seen describeConsumerGroups supported in Spring-Kafka KafkaAdmin. So, you may need to create a Kafka AdminClient and call the method by yourself.
E.g: Here, I took advantage of the auto-configuration property class KafkaProperties and autowired it to the service.
#Service
public class KafkaBrokerService implements BrokerService {
private Map<String, Object> configs;
public KafkaBrokerService(KafkaProperties kafkaProperties) { // Autowired
this.configs = kafkaProperties.buildAdminProperties();
}
private AdminClient createAdmin() {
Map<String, Object> configs2 = new HashMap<>(this.configs);
return AdminClient.create(configs2);
}
public SomeDto consumerGroupDescription(String groupId) {
try (AdminClient adminClient = createAdmin()) {
// ConsumerGroup's members
ConsumerGroupDescription consumerGroupDescription = adminClient.describeConsumerGroups(Collections.singletonList(groupId))
.describedGroups().get(groupId).get();
// ConsumerGroup's partitions and the committed offset in each partition
Map<TopicPartition, OffsetAndMetadata> offsets = adminClient.listConsumerGroupOffsets(groupId).partitionsToOffsetAndMetadata().get();
// When you get the information, you can validate it here.
...
} catch (ExecutionException | InterruptedException e) {
//
}
}
}
I own a spring application and want to add camel routes dynamically during my application startup.End points are configured in property file and are loaded at run time.
Using Java DSL, i am using for loop to create all routes,
for(int i=0;i<allEndPoints;i++)
{
DynamcRouteBuilder route = new
DynamcRouteBuilder(context,fromUri,toUri)
camelContext.addRoutes(route)
}
private class DynamcRouteBuilder extends RouteBuilder {
private final String from;
private final String to;
private MyDynamcRouteBuilder(CamelContext context, String from, String to) {
super(context);
this.from = from;
this.to = to;
}
#Override
public void configure() throws Exception {
from(from).to(to);
}
}
but getting below exception while creating first route itself
Failed to create route file_routedirect: at: >>> OnException[[class org.apache.camel.component.file.GenericFileOperationFailedException] -> [Log[Exception trapped ${exception.class}], process[Processor#0x0]]] <<< in route: Route(file_routedirect:)[[From[direct:... because of ref must be specified on: process[Processor#0x0]\n\ta
Not sure about it- what is the issue ? Can someone has any suggestion or fix for this. Thanks
Well, to create routes in an iteration it is nice to have some object that holds the different values for one route. Let's call this RouteConfiguration, a simple POJO with String fields for from, to and routeId.
We are using YAML files to configure such things because you have a real List format instead of using "flat lists" in property files (route[0].from, route[0].to).
If you use Spring you can directly transform such a "list of object configurations" into a Collection of objects using #ConfigurationProperties
When you are able to create such a Collection of value objects, you can simply iterate over it. Here is a strongly simplified example.
#Override
public void configure() {
createConfiguredRoutes();
}
void createConfiguredRoutes() {
configuration.getRoutes().forEach(this::addRouteToContext);
}
// Implement route that is added in an iteration
private void addRouteToContext(final RouteConfiguration routeConfiguration) throws Exception {
this.camelContext.addRoutes(new RouteBuilder() {
#Override
public void configure() throws Exception {
from(routeConfiguration.getFrom())
.routeId(routeConfiguration.getRouteId())
...
.to(routeConfiguration.getTo());
}
});
}
I have been evaluating to adopt spring-data-mongodb for a project. In summary, my aim is:
Using existing XML schema files to generate Java classes.
This is achieved using JAXB xjc
The root class is TSDProductDataType and is further modeled as below:
The thing to note here is that ExtensionType contains protected List<Object> any; allowing it to store Objects of any class. In my case, it is amongst the classes named TSDModule_Name_HereModuleType and can be browsed here
Use spring-data-mongodb as persistence store
This is achieved using a simple ProductDataRepository
#RepositoryRestResource(collectionResourceRel = "product", path = "product")
public interface ProductDataRepository extends MongoRepository<TSDProductDataType, String> {
TSDProductDataType queryByGtin(#Param("gtin") String gtin);
}
The unmarshalled TSDProductDataType, however, contains JAXBElement which spring-data-mongodb doesn't seem to handle by itself and throws a CodecConfigurationException org.bson.codecs.configuration.CodecConfigurationException: Can't find a codec for class java.lang.Class.
Here is the faulty statement:
TSDProductDataType tsdProductDataType = jaxbElement.getValue();
repository.save(tsdProductDataType);
I tried playing around with Converters for spring-data-mongodb as explained here, however, it seems I am missing something since the exception is about "Codecs" and not "Converters".
Any help is appreciated.
EDIT:
Adding converters for JAXBElement
Note: Works with version 1.5.6.RELEASE of org.springframework.boot::spring-boot-starter-parent. With version 2.0.0.M3, hell breaks loose
It seems that I missed something while trying to add converter earlier. So, I added it like below for testing:
#Component
#ReadingConverter
public class JAXBElementReadConverter implements Converter<DBObject, JAXBElement> {
//#Autowired
//MongoConverter converter;
#Override
public JAXBElement convert(DBObject dbObject) {
Class declaredType, scope;
QName name = qNameFromString((String)dbObject.get("name"));
Object rawValue = dbObject.get("value");
try {
declaredType = Class.forName((String)dbObject.get("declaredType"));
} catch (ClassNotFoundException e) {
if (rawValue.getClass().isArray()) declaredType = List.class;
else declaredType = LinkedHashMap.class;
}
try {
scope = Class.forName((String) dbObject.get("scope"));
} catch (ClassNotFoundException e) {
scope = JAXBElement.GlobalScope.class;
}
//Object value = rawValue instanceof DBObject ? converter.read(declaredType, (DBObject) rawValue) : rawValue;
Object value = "TODO";
return new JAXBElement(name, declaredType, scope, value);
}
QName qNameFromString(String s) {
String[] parts = s.split("[{}]");
if (parts.length > 2) return new QName(parts[1], parts[2], parts[0]);
if (parts.length == 1) return new QName(parts[0]);
return new QName("undef");
}
}
#Component
#WritingConverter
public class JAXBElementWriteConverter implements Converter<JAXBElement, DBObject> {
//#Autowired
//MongoConverter converter;
#Override
public DBObject convert(JAXBElement jaxbElement) {
DBObject dbObject = new BasicDBObject();
dbObject.put("name", qNameToString(jaxbElement.getName()));
dbObject.put("declaredType", jaxbElement.getDeclaredType().getName());
dbObject.put("scope", jaxbElement.getScope().getCanonicalName());
//dbObject.put("value", converter.convertToMongoType(jaxbElement.getValue()));
dbObject.put("value", "TODO");
dbObject.put("_class", JAXBElement.class.getName());
return dbObject;
}
public String qNameToString(QName name) {
if (name.getNamespaceURI() == XMLConstants.NULL_NS_URI) return name.getLocalPart();
return name.getPrefix() + '{' + name.getNamespaceURI() + '}' + name.getLocalPart();
}
}
#SpringBootApplication
public class TsdApplication {
public static void main(String[] args) {
SpringApplication.run(TsdApplication.class, args);
}
#Bean
public CustomConversions customConversions() {
return new CustomConversions(Arrays.asList(
new JAXBElementReadConverter(),
new JAXBElementWriteConverter()
));
}
}
So far so good. However, how do I instantiate MongoConverter converter;?
MongoConverter is an interface so I guess I need an instantiable class adhering to this interface. Any suggestions?
I understand the desire for convenience in being able to just map an existing domain object to the database layer with no boilerplate, but even if you weren't having the JAXB class structure issue, I would still be recommending away from using it verbatim. Unless this is a simple one-off project, you almost definitely will hit a point where your domain models will need to change but your persisted data need to remain in an existing state. If you are just straight persisting the data, you have no mechanism to convert between a newer domain schema and an older persisted data scheme. Versioning of the persisted data scheme would be wise too.
The link you posted for writing the customer converters is one way to achieve this and fits in nicely with the Spring ecosystem. That method should also solve the issue you are experiencing (about the underlying messy JAXB data structure not converting cleanly).
Are you unable to get that method working? Ensure you are loading them into the Spring context with #Component plus auto-class scanning or manually via some Configuration class.
EDIT to address your EDIT:
Add the following to each of your converters:
private final MongoConverter converter;
public JAXBElement____Converter(MongoConverter converter) {
this.converter = converter;
}
Try changing your bean definition to:
#Bean
public CustomConversions customConversions(#Lazy MongoConverter converter) {
return new CustomConversions(Arrays.asList(
new JAXBElementReadConverter(converter),
new JAXBElementWriteConverter(converter)
));
}
I am working with Spring-websocket and I have the following problem:
I am trying to put a placeholder inside a #MessageMapping annotation in order to get the url from properties. It works with #RequestMapping but not with #MessageMapping.
If I use this placeholder, the URL is null. Any idea or suggestion?
Example:
#RequestMapping(value= "${myProperty}")
#MessageMapping("${myProperty}")
Rossen Stoyanchev added placeholder support for #MessageMapping and #SubscribeMapping methods.
See Jira issue: https://jira.spring.io/browse/SPR-13271
Spring allows you to use property placeholders in #RequestMapping, but not in #MessageMapping. This is 'cause the MessageHandler. So, we need to override the default MessageHandler to do this.
WebSocketAnnotationMethodMessageHandler does not support placeholders and you need add this support yourself.
For simplicity I just created another WebSocketAnnotationMethodMessageHandler class in my project at the same package of the original, org.springframework.web.socket.messaging, and override getMappingForMethod method from SimpAnnotationMethodMessageHandler with same content, changing only how SimpMessageMappingInfo is contructed using this with this methods (private in WebSocketAnnotationMethodMessageHandler):
private SimpMessageMappingInfo createMessageMappingCondition(final MessageMapping annotation) {
return new SimpMessageMappingInfo(SimpMessageTypeMessageCondition.MESSAGE, new DestinationPatternsMessageCondition(
this.resolveAnnotationValues(annotation.value()), this.getPathMatcher()));
}
private SimpMessageMappingInfo createSubscribeCondition(final SubscribeMapping annotation) {
final SimpMessageTypeMessageCondition messageTypeMessageCondition = SimpMessageTypeMessageCondition.SUBSCRIBE;
return new SimpMessageMappingInfo(messageTypeMessageCondition, new DestinationPatternsMessageCondition(
this.resolveAnnotationValues(annotation.value()), this.getPathMatcher()));
}
These methods now will resolve value considering properties (calling resolveAnnotationValues method), so we need use something like this:
private String[] resolveAnnotationValues(final String[] destinationNames) {
final int length = destinationNames.length;
final String[] result = new String[length];
for (int i = 0; i < length; i++) {
result[i] = this.resolveAnnotationValue(destinationNames[i]);
}
return result;
}
private String resolveAnnotationValue(final String name) {
if (!(this.getApplicationContext() instanceof ConfigurableApplicationContext)) {
return name;
}
final ConfigurableApplicationContext applicationContext = (ConfigurableApplicationContext) this.getApplicationContext();
final ConfigurableBeanFactory configurableBeanFactory = applicationContext.getBeanFactory();
final String placeholdersResolved = configurableBeanFactory.resolveEmbeddedValue(name);
final BeanExpressionResolver exprResolver = configurableBeanFactory.getBeanExpressionResolver();
if (exprResolver == null) {
return name;
}
final Object result = exprResolver.evaluate(placeholdersResolved, new BeanExpressionContext(configurableBeanFactory, null));
return result != null ? result.toString() : name;
}
You still need to define a PropertySourcesPlaceholderConfigurer bean in your configuration.
If you are using XML based configuration, include something like this:
<context:property-placeholder location="classpath:/META-INF/spring/url-mapping-config.properties" />
If you are using Java based configuration, you can try in this way:
#Configuration
#PropertySources(value = #PropertySource("classpath:/META-INF/spring/url-mapping-config.properties"))
public class URLMappingConfig {
#Bean
public static PropertySourcesPlaceholderConfigurer propertySourcesPlaceholderConfigurer() {
return new PropertySourcesPlaceholderConfigurer();
}
}
Obs.: in this case, url-mapping-config.properties file are in a gradle/maven project in src\main\resources\META-INF\spring folder and content look like this:
myPropertyWS=urlvaluews
This is my sample controller:
#Controller
public class WebSocketController {
#SendTo("/topic/test")
#MessageMapping("${myPropertyWS}")
public String test() throws Exception {
Thread.sleep(4000); // simulated delay
return "OK";
}
}
With default MessageHandler startup log will print something like this:
INFO: Mapped "{[/${myPropertyWS}],messageType=[MESSAGE]}" onto public java.lang.String com.brunocesar.controller.WebSocketController.test() throws java.lang.Exception
And with our MessageHandler now print this:
INFO: Mapped "{[/urlvaluews],messageType=[MESSAGE]}" onto public java.lang.String com.brunocesar.controller.WebSocketController.test() throws java.lang.Exception
See in this gist the full WebSocketAnnotationMethodMessageHandler implementation.
EDIT: this solution resolves the problem for versions before 4.2 GA. For more information, see this jira.
Update :
Now I understood what you mean, but I think that is not possible(yet).
Documentation does not mention anything related to Path mapping URIs.
Old answer
Use
#MessageMapping("/handler/{myProperty}")
instead of
#MessageMapping("/handler/${myProperty}")
And use it like this:
#MessageMapping("/myHandler/{username}")
public void handleTextMessage(#DestinationVariable String username,Message message) {
//do something
}
#MessageMapping("/chat/{roomId}")
public Message handleMessages(#DestinationVariable("roomId") String roomId, #Payload Message message, Traveler traveler) throws Exception {
System.out.println("Message received for room: " + roomId);
System.out.println("User: " + traveler.toString());
// store message in database
message.setAuthor(traveler);
message.setChatRoomId(Integer.parseInt(roomId));
int id = MessageRepository.getInstance().save(message);
message.setId(id);
return message;
}