I'm trying to find a way to configure the topic that a spring cloud stream #StreamListener will listen to. My first attempt was to try to use SPeL to get this, eg.
#StreamListener("#{ systemProperties['topic.name'] }")
but the expression is not replaced and I end up with
java.lang.IllegalArgumentException: Topic name can only have ASCII alphanumerics, '.', '_' and '-'
Is there any way to control what the topic name is when launching the application, rather than just at compile time with a constant?
The BeanPostProcessor that handles #StreamListener does not support SpEL or property placeholders; it can only contain a bean name for the object (message channel) that is bound to the binder destination.
Exactly what are you trying to achieve? There is already an indirection between the destination and the channel via application.properties/yml.
In my case, if the destination contains any space between values - for example destination=foo, bar, then it causes the error:
Topic name can only have ASCII alphanumerics.
After I removed spaces, it worked for me. Just wanted to share it.
Related
I'm currently trying to change the functionality of the QuerySpecUrlMapper for only one specific endpoint. In particular, we want to disable the behavior that splits a string parameter by commas. I've figured out I can do this by getting the default URL mapper and calling setAllowCommaSeparatedValue, but I'm not sure how to do this for the specific endpoint I want. We're using Spring, and I tried Autowiring CrnkBoot into my Repository object so I could get the UrlMapper that way, but this breaks the whole thing in strange ways I don't yet understand. Some exception about "Cipher not initialized"
Anyone have ideas about how to handle this?
I've an entity with property name qYear. I tried creating a findByIdAndQYear method in repository but that did not work. I ran into IllegalArgumentException: Unable to locate Attribute with the the given name [QYear] on this ManagedType).
However findByIdAndqYear works. Any idea how single letter abbreviations like this are expanded please?
Spring Data (not just the JPA module) base this on the Java Bean Specification.
In order to avoid misinterpretation of the specification this is actually implemented using [java.beans.Introspector][1].
See also https://jira.spring.io/browse/DATACMNS-1589
Customized grouping Tasks by topic name instead of partition Id.
How to refer my custom partition grouper class in my Kafka stream application ?
Thanks
You can set a custom partition grouper class using the StreamsConfig.PARTITION_GROUPER_CLASS_CONFIG option in your streams config.
However, as Matthias says, this is unadvisable unless you know what you're doing or want to learn :). Perhaps what you are trying to do can be accomplished some other way?
As a result of a bug in Kafka-Streams source code (version 2.1.0), you'll need in addition to add this configuration with a consumer prefix as follows:
Properties props = new Properties();
props.put(StreamsConfig.consumerPrefix(PARTITION_GROUPER_CLASS_CONFIG), CustomPartitionGrouper.class.getName());
props.put(StreamsConfig.PARTITION_GROUPER_CLASS_CONFIG, CustomPartitionGrouper.class.getName());
The reason for adding the consumer config prefix is that the StickyTaskAssignor and the PartitionGrouper instances are being initialized in the consumer initialization flow. Without the prefix, the consumer will ignore the PARTITION_GROUPER_CLASS_CONFIG and will use the default which is the DefaultPartitionGrouper class.
I am using spring-cloud-stream kafka binder to consume messages from a kafka topic. The source system is sending the json message in ascii. When My consumer listens to the topic it throws
o.s.c.s.b.k.KafkaMessageChannelBinder : Could not convert message: 7B22736..
Is there any property that I can set in my .yml file to deserialize it? or is there an example that I can look into?
I am not sure what you mean by json in hexadecimal-binary data if you mean it's ascii data in a byte[], try adding spring.cloud.stream.bindings.input.content-type=text/plain (or application/json).
You can look for the configuration property here: http://docs.spring.io/spring-cloud-stream/docs/Brooklyn.SR2/reference/htmlsingle/#_kafka_consumer_properties
In your case, you can set this by doing the following:
spring.cloud.stream.kafka.bindings.<channelName>.consumer.configuration.value.deserializer=<Deserialzier class>
Kafka binder takes all properties from the configuration map. Thus you can use any generic Kafka consumer properties and pass them this way.
When I added content-type:plain/text and spring.cloud.stream.bindings.<subscriptionChannel>.consumer.headerMode:raw it worked.
Thank you!
Is there an easy way to convert a JSON payload to a Java object using a custom ObjectMapper (Jackson) or do I have to provide a custom type converter. I know that I could use a processor, but somehow it would be nice to use input and output types of the stream definition.
In the second case: Am I even able to provide a custom type converter for application/json to Java?
The documentation states: "The customMessageConverters are added after the standard converters in the order defined. So it is generally easier to add converters for new media types than to replace existing converters."
I bet that there is an existing "application/json" converter - but at a first glance I could not find further information if it is even possible to replace existing converters.
Thanks!
Peter
If you look at streams.xml You can see the relevant configuration. The configured lists are used to construct a CompositeMessageConverter which visits every MessageConverter in list order until it finds one that can do the conversion and returns a non-null result. A CompositeConverter instance is created for each module instance that is configured for conversion (i.e., defines an inputType or outputType value) by filtering the list of candidate message converters, which all inherit AbstractFromMessageConverter. The list is paired down to those which respond true to public boolean supportsTargetMimeType(MimeType mimeType) (where mimeType is the value of the input/outputType). The CompositeMessageConverter is injected into the corresponding MessageChannel and converts the payload.
There are a couple of things you can do. You can override the xd.messageConverters bean definition. For example, you can replace JsonToPojoMessageConverter and PojoToJsonMessageConverter with your own subclasses. You can also insert your own implementations in the list before the above converters and have your implementation match only specific domain objects for which you need a custom JSON mapper.
Another possibility is to define your own mime type and provide converters for that mime type as customMessageConverters. In any case, follow these guidelines forextending Spring XD