Customized grouping Tasks by topic name instead of partition Id.
How to refer my custom partition grouper class in my Kafka stream application ?
Thanks
You can set a custom partition grouper class using the StreamsConfig.PARTITION_GROUPER_CLASS_CONFIG option in your streams config.
However, as Matthias says, this is unadvisable unless you know what you're doing or want to learn :). Perhaps what you are trying to do can be accomplished some other way?
As a result of a bug in Kafka-Streams source code (version 2.1.0), you'll need in addition to add this configuration with a consumer prefix as follows:
Properties props = new Properties();
props.put(StreamsConfig.consumerPrefix(PARTITION_GROUPER_CLASS_CONFIG), CustomPartitionGrouper.class.getName());
props.put(StreamsConfig.PARTITION_GROUPER_CLASS_CONFIG, CustomPartitionGrouper.class.getName());
The reason for adding the consumer config prefix is that the StickyTaskAssignor and the PartitionGrouper instances are being initialized in the consumer initialization flow. Without the prefix, the consumer will ignore the PARTITION_GROUPER_CLASS_CONFIG and will use the default which is the DefaultPartitionGrouper class.
Related
I am working on a spring boot application, i want to create a separate logging file for each user of the application, is this possible ?
For example: user1.log, user2.log, user3.log, ....
Thanks.
It's possible but it will create as many log files as users. Imagine if your user base increases to 20K. Unless you have very strong need don't go for it.
Instead go for application level and user level loggings. To achieve this refer here - https://stackoverflow.com/a/9652239
Although I agree with Kann's answer that the best approach is to filter after the fact, the answer to your question when using Log4j 2 would be to use the RoutingAppender or the SiftingAppender when using Logback. They both work similarly in that they will create new appenders for each unique item which can cause a problem with file handles. Log4j2's RoutingAppender provides for a PurgePolicy to handle that while Logback provides a timeToLive attribute. Logback uses a Discriminator class to choose how to determine how to match the log event to an Appender while Log4j 2 uses either a pattern that should contain a Lookup (Log4j 2's variable substitution mechanism) or a script to perform the matching.
If you are using java.util.logging you will have to write your own mechanism.
My task is to read events from multiple different topics (class of all data in all topics is "Event"). This class contains field "data" (Map) which carries specific for each topic data, that can be deserialized to specific class (e.g. to "DeviceCreateEvent" or smth.). I can create consumers for each topic with #KafkaListener on methods with parameter type "Event". But in this case firstly i have to event.getData() and deserialize it into specific class, so I will get code duplication in all consumer methods. Is there any way to get in annotated consumer method already deserialized object to specific class?
It's not clear what you are asking.
If you have a different #KafkaListener for each topic/event type, and use JSON, the framework will automatically tell the message converter the type the data should be converted to; see the documentation.
Although the Serializer and Deserializer API is quite simple and flexible from the low-level Kafka Consumer and Producer perspective, you might need more flexibility at the Spring Messaging level, when using either #KafkaListener or Spring Integration. To let you easily convert to and from org.springframework.messaging.Message, Spring for Apache Kafka provides a MessageConverter abstraction with the MessagingMessageConverter implementation and its JsonMessageConverter (and subclasses) customization. You can inject the MessageConverter into a KafkaTemplate instance directly and by using AbstractKafkaListenerContainerFactory bean definition for the #KafkaListener.containerFactory() property. The following example shows how to do so: ...
On the consumer side, you can configure a JsonMessageConverter; it can handle ConsumerRecord values of type byte[], Bytes and String so should be used in conjunction with a ByteArrayDeserializer, BytesDeserializer or StringDeserializer. (byte[] and Bytes are more efficient because they avoid an unnecessary byte[] to String conversion). You can also configure the specific subclass of JsonMessageConverter corresponding to the deserializer, if you so wish.
I am using spring-cloud-stream kafka binder to consume messages from a kafka topic. The source system is sending the json message in ascii. When My consumer listens to the topic it throws
o.s.c.s.b.k.KafkaMessageChannelBinder : Could not convert message: 7B22736..
Is there any property that I can set in my .yml file to deserialize it? or is there an example that I can look into?
I am not sure what you mean by json in hexadecimal-binary data if you mean it's ascii data in a byte[], try adding spring.cloud.stream.bindings.input.content-type=text/plain (or application/json).
You can look for the configuration property here: http://docs.spring.io/spring-cloud-stream/docs/Brooklyn.SR2/reference/htmlsingle/#_kafka_consumer_properties
In your case, you can set this by doing the following:
spring.cloud.stream.kafka.bindings.<channelName>.consumer.configuration.value.deserializer=<Deserialzier class>
Kafka binder takes all properties from the configuration map. Thus you can use any generic Kafka consumer properties and pass them this way.
When I added content-type:plain/text and spring.cloud.stream.bindings.<subscriptionChannel>.consumer.headerMode:raw it worked.
Thank you!
I'm trying to find a way to configure the topic that a spring cloud stream #StreamListener will listen to. My first attempt was to try to use SPeL to get this, eg.
#StreamListener("#{ systemProperties['topic.name'] }")
but the expression is not replaced and I end up with
java.lang.IllegalArgumentException: Topic name can only have ASCII alphanumerics, '.', '_' and '-'
Is there any way to control what the topic name is when launching the application, rather than just at compile time with a constant?
The BeanPostProcessor that handles #StreamListener does not support SpEL or property placeholders; it can only contain a bean name for the object (message channel) that is bound to the binder destination.
Exactly what are you trying to achieve? There is already an indirection between the destination and the channel via application.properties/yml.
In my case, if the destination contains any space between values - for example destination=foo, bar, then it causes the error:
Topic name can only have ASCII alphanumerics.
After I removed spaces, it worked for me. Just wanted to share it.
Is there any way to use key-class and value-class parameters for the Gemfire sink in Spring xd?
Regarding to documentation i can use only keyExpression but nothing about its class type. Same for the key-class.
I have such command for the Gemfire,
put --key-class = java.lang.String --value-class = Employee --key = ('id': '998') --value = ('id': 186, 'firstName': 'James', 'lastName': 'Goslinga') --region = replicated2
So i use --key-class and --value-class parameters in Gemfire.
But i cannot use them from Spring xd since there is only keyExpression parameter in Gemfire Sink.
Any idea to solve?
As far as I know the syntax above is not supported by native GemFire. So you can't do it out of the box with Spring XD. The syntax looks vaguely SQL-like. Are you using Gemfire XD? Is this something you wrote yourself?
The gemfire sink uses spring-integration-gemfire, allowing you to declare the keyExpression using SpEL. The value, using the gemfire sink, is always the payload. The SI gemfire outbound adapter wraps Region.put(key, value). The GemFire API supports typing via generics, i.e. Region<K,V> but this is not enforced in this case. GemFire RegionFactory allows keyConstraint and valueConstraint attributes to constrain types but this is part of the Region configuration which is external to Spring XD. Furthermore, none of this addresses the data binding in your example, e.g.,
Person p = ('id': 186, 'firstName': 'James', 'lastName': 'Goslinga')
This capability would require a custom sink module. If your command can be executed as a shell script, you might be able to use a shell sink to invoke it.
Thank you for your answer,
Maybe basically i can explain my problem in this way.
if i write following command to gemfire console i can create new entry in region which contains object of Employee class.
put --key-class=java.lang.String --value-class=Employee --key=('id':'998') --value=('id':186,'firstName':'James','lastName':'Goslinga') --region=replicated2
The think that i want to do is i will send data from spring-xd. And i will have a new object of Employee class in Gemfire.
If i create such stream which will get data from rabbit MQ and send it to gemfire.
stream create --name reference-data-import --definition "rabbit --outputType=text/plain | gemfire-json-server --host=MC0WJ1BC --regionName=region10 --keyExpression=payload.getField('id')" --deploy
I can see that data in this type of "com.gemstone.gemfire.pdx.internal.PdxInstanceImpl".
Regarding to spring-xd documentation i can use such parametter outputType=application/x-java-object;type=com.bar.Foo but i never managed to work it out even though i deploy my class.
if i can see a simple working example it will be great for me.