As one would expect its common to want to have different Consumers deserializing in different ways off topics in Kafka. There is a known problem with spring boot autoconfig. It seems that as soon as other factories are defined Spring Kafka or the autoconfig complains about not being able to find a suitable consumer factory anymore. Some have pointed out that one solution is to include a ConsumerFactory of type (Object, Object) in the config. But no one has shown the source code for this or clarified if it needs to be named in any particular way. Or if simply adding this Consumer to the config removes the need to turn off autoconfig. All that remains very unclear.
If you are not familiar with this issue please read https://github.com/spring-projects/spring-boot/issues/19221
Where it was just stated ok, define the ConsumerFactory and add it somewhere in your config. Can someone be a bit more precise about this please.
Show exactly how to define the ConsumerFactory so that Spring boot autoconfig will not complain.
Explain if turning off autoconfig is or is not needed?
Explain if Consumer Factory needs to be named in any special way or not.
The simplest solution is to stick with Boot's auto-configuration and override the deserializer on the #KafkaListener itself...
#SpringBootApplication
public class So63108344Application {
public static void main(String[] args) {
SpringApplication.run(So63108344Application.class, args);
}
#KafkaListener(id = "so63108344-1", topics = "so63108344-1")
public void listen1(String in) {
System.out.println(in);
}
#KafkaListener(id = "so63108344-2", topics = "so63108344-2", properties =
ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG +
"=org.apache.kafka.common.serialization.ByteArrayDeserializer")
public void listen2(byte[] in) {
System.out.println(in);
}
#Bean
public NewTopic topic1() {
return TopicBuilder.name("so63108344-1").partitions(1).replicas(1).build();
}
#Bean
public NewTopic topic2() {
return TopicBuilder.name("so63108344-2").partitions(1).replicas(1).build();
}
}
For more advanced container customization (or if you don't want to pollute the #KafkaListener, you can use a ContainerCustomizer...
#Component
class Customizer {
public Customizer(ConcurrentKafkaListenerContainerFactory<?, ?> factory) {
factory.setContainerCustomizer(container -> {
if (container.getGroupId().equals("so63108344-2")) {
container.getContainerProperties().setAckMode(AckMode.MANUAL_IMMEDIATE);
container.getContainerProperties().getKafkaConsumerProperties()
.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.ByteArrayDeserializer");
}
});
}
}
Related
I am fairly new to spring and spring integration. What I'm trying to do: publish mqtt messages using spring integration.
Here is the code:
#Configuration
#IntegrationComponentScan
#Service
public class MQTTPublishAdapter {
private MqttConfiguration mqttConfiguration;
public MQTTPublishAdapter(MqttConfiguration mqttConfiguration) {
this.mqttConfiguration = mqttConfiguration;
}
#Bean
public MessageChannel mqttOutboundChannel() {
return new PublishSubscribeChannel();
}
#Bean
public MqttPahoClientFactory mqttClientFactory() {
DefaultMqttPahoClientFactory factory = new
DefaultMqttPahoClientFactory();
//... set factory details
return factory;
}
#Bean
#ServiceActivator(inputChannel = "mqttOutboundChannel")
public MQTTCustomMessageHandler mqttOutbound() {
String clientId = UUID.randomUUID().toString();
MQTTCustomMessageHandler messageHandler =
new MQTTCustomMessageHandler(clientId, mqttClientFactory());
//...set messagehandler details
return messageHandler;
}
//I extend this only because the publish method is protected and I want to
send messages to different topics
public class MQTTCustomMessageHandler extends MqttPahoMessageHandler {
//default constructors
public void sendMessage(String topic, String message){
MqttMessage mqttMessage = new MqttMessage();
mqttMessage.setPayload(message.getBytes());
try {
super.publish(topic, mqttMessage, null);
} catch (Exception e) {
log.error("Failure to publish message on topic " + topic,
e.getMessage());
}
}
}
This is the clase where I am trying to inject the Handler
#Service
public class MQTTMessagePublisher {
private MQTTCustomMessageHandler mqttCustomMessageHandler;
public MQTTMessagePublisher(#Lazy MQTTCustomMessageHandler
mqttCustomMessageHandler) {
this.mqttCustomMessageHandler = mqttCustomMessageHandler;
}
public void publishMessage(String topic, String message) {
mqttCustomMessageHandler.sendMessage(topic, message);
}
}
So my question is about how should I inject the bean I am trying to use because if I remove the #Lazy annotation it says that "Requested bean is currently in creation: Is there an unresolvable circular reference?". I do not have any circular dependencies as in the bean I only set some strings, so I'm guessing that I don't really understand how this should work.
Very sorry about the formating, it's one of my first questions around here.
Edit:
If I remove
#ServiceActivator(inputChannel = "mqttOutboundChannel")
and add
messageHandler.setChannelResolver((name) -> mqttOutboundChannel());
it works. I'm still unclear why the code crashes.
You show a lot of custom code, but not all of them.
It's really hard to answer to questions where it is only a custom code. Would be great to share as much info as possible. For example an external project on GitHub to let us to play and reproduce would be fully helpful and would save some time.
Nevertheless, I wonder what is your MQTTCustomMessageHandler. However I guess it is not a MessageHandler implementation. From here the #ServiceActivator annotation is not going to work properly since it is applied really for the mqttOutbound(), not whatever you expect. Or you need to move this annotation to your sendMessage() method in the MQTTCustomMessageHandler or have it as a MessageHandler.
On the other hand it is not clear why do you need that #ServiceActivator annotation at all since you call that method manually from the MQTTMessagePublisher.
Also it is not clear why you have so much custom code when Framework provides for your out-of-the-box channel adapter implementations.
Too many questions to your code, than possible answer...
See more info in the reference manual:
https://docs.spring.io/spring-integration/docs/current/reference/html/#annotations
https://docs.spring.io/spring-integration/docs/current/reference/html/#mqtt
I have code with named beans
#Bean
#Named("heimdall-uri-supplier")
public URISupplier heimdallEndpointSupplier(CredentialsClientConfig config, EnvInfo envInfo) {
....
}
#Named("vault-uri-supplier")
#Bean
public URISupplier vaultURISupplier(EnvInfo envInfo, CredentialsClientConfig config) {
....
}
They are explicitly Named because I want specific implementations injected into different consuming classes. These classes also use #Named.
In tests, prior to Spring Boot 2.1
#Bean
#Primary
#Named("heimdall-uri-supplier")
public URISupplier heimdallEndpointSupplier(CredentialsClientConfig config, EnvInfo envInfo) {
return mock of some sort
}
#Named("vault-uri-supplier")
#Bean
#Primary
public URISupplier vaultURISupplier(EnvInfo envInfo, CredentialsClientConfig config) {
return mock of some sort
}
worked great.
Now, of course spring boot 2.1 disables overriding. I know I can reenable it, but I'd rather not in theory do so.
But my "normal" workaround (do #Bean(name="testFoo") won't work here because the #Named injector in the consuming classes will now fail.
Are there any solutions?
I managed to do it, but it was quite painful. I had to use a BeanDefinitionRegistryPostProcessor. Setting up GenericBeanDefinition is ugly as sin. It wasn't too bad here, because the mocks are no-args. If you have to base them on injected or set constructor args, it becomes quite involved indeed.
/**
* This class basically removes the existing bean definition and substitutes in the mocks.
* Normally we don't need this - we just change the bean name. However because the
* classes use an #Named qualifier, more heroic efforts are needed
*/
public static class OverridePostProcessor implements BeanDefinitionRegistryPostProcessor {
#Override
public void postProcessBeanDefinitionRegistry(final BeanDefinitionRegistry registry) throws BeansException {
if (registry.isBeanNameInUse("vault-uri-supplier")) {
registry.removeBeanDefinition("vault-uri-supplier");
}
// Note: These are a bear to work with, but in theory you can build
// them from scratch based on injected beans etc. Fortunately these
// two examples just needed a no-args supplier
GenericBeanDefinition g = new GenericBeanDefinition();
g.setBeanClass(URISupplier.class);
g.setInstanceSupplier(this::vaultURISupplier);
registry.registerBeanDefinition("vault-uri-supplier", g);
if (registry.isBeanNameInUse("taskRequestValidator")) {
registry.removeBeanDefinition("taskRequestValidator");
}
g = new GenericBeanDefinition();
g.setBeanClass(TaskRequestValidator.class);
g.setInstanceSupplier(this::taskRequestValidator);
registry.registerBeanDefinition("taskRequestValidator", g);
}
URISupplier vaultURISupplier() {
return new com.opentable.credentials.client.internal.TestVaultConfiguration.MockSupplier();
}
TaskRequestValidator taskRequestValidator() {
return (getTokenRequest, servicePolicy) -> ValidationResult.OK;
}
#Override
public void postProcessBeanFactory(final ConfigurableListableBeanFactory beanFactory) throws BeansException {
/* No op */
}
}
I need to bind single queue with multiple routing keys.
I have configuration in application.properties:
spring.cloud.stream.bindings.some-channel1.destination=exch
spring.cloud.stream.bindings.some-channel1.group=a-queue
spring.cloud.stream.rabbit.bindings.some-channel1.consumer.binding-routing-key=event.domain1
spring.cloud.stream.bindings.some-channel2.destination=exch
spring.cloud.stream.bindings.some-channel2.group=a-queue
spring.cloud.stream.rabbit.bindings.some-channel2.consumer.binding-routing-key=event.domain2
This creates queue and bindings properly in rabbit, but finally after running application I got:
org.springframework.cloud.stream.binder.BinderException: Exception thrown while starting consumer:
After all above configuration i still bad because I need single channel. But queue binded with list of routing keys.
Any Ideas how to configure it?
You can't do it with stream properties, but you can always add extra bindings with normal Spring AMQP declarations...
#SpringBootApplication
#EnableBinding(Sink.class)
public class So50526298Application {
public static void main(String[] args) {
SpringApplication.run(So50526298Application.class, args);
}
#StreamListener(Sink.INPUT)
public void listen(String in) {
System.out.println(in);
}
// extra bindings...
#Bean
public TopicExchange exch() {
return new TopicExchange("exch");
}
#Bean
public Queue queue() {
return new Queue("exch.a-queue");
}
#Bean
public Binding extraBinding1() {
return BindingBuilder.bind(queue()).to(exch()).with("event-domain2");
}
}
There is also a third party "advanced" boot starter that allows you to add declarations in a yaml file. I haven't tried it, but it looks interesting.
I'm trying to disable a Zuul route to a microservice registered with Eureka at runtime (I'm using spring boot).
This is an example:
localhost/hello
localhost/world
Those two are the registered microservices. I would like to disable the route to one of them at runtime without shutting it down.
Is there a way to do this?
Thank you,
Nano
Alternatively to using Cloud Config, custom ZuulFilter can be used. Something like (partial implementation to show the concept):
public class BlackListFilter extends ZuulFilter {
#Override
public String filterType() {
return "pre";
}
...
#Override
public Object run() {
RequestContext ctx = RequestContext.getCurrentContext();
String uri = ctx.getRequest().getRequestURI();
String appId = uri.split("/")[1];
if (blackList.contains(appId)) {
ctx.setSendZuulResponse(false);
LOG.info("Request '{}' from {}:{} is blocked",
uri, ctx.getRequest().getRemoteHost(), ctx.getRequest().getRemotePort());
}
return null;
}
}
where blackList contains list of application IDs (Spring Boot application name) managed for example via some RESTful API.
After a lot of efforts I came up with this solution. First, I used Netflix Archaius to watch a property file. Then I proceeded as follows:
public class ApplicationRouteLocator extends SimpleRouteLocator implements RefreshableRouteLocator {
public ApplicationRouteLocator(String servletPath, ZuulProperties properties) {
super(servletPath, properties );
}
#Override
public void refresh() {
doRefresh();
}
}
Made the doRefresh() method public by extending SimpleRouteLocator and calling its method in the overridden one of the interface RefreshableRouteLocator.
Then I redefined the bean RouteLocator with my custom implementation:
#Configuration
#EnableConfigurationProperties( { ZuulProperties.class } )
public class ZuulConfig {
public static ApplicationRouteLocator simpleRouteLocator;
#Autowired
private ZuulProperties zuulProperties;
#Autowired
private ServerProperties server;
#Bean
#Primary
public RouteLocator routeLocator() {
logger.info( "zuulProperties are: {}", zuulProperties );
simpleRouteLocator = new ApplicationRouteLocator( this.server.getServletPrefix(),
this.zuulProperties );
ConfigurationManager.getConfigInstance().addConfigurationListener( configurationListener );
return simpleRouteLocator;
}
private ConfigurationListener configurationListener =
new ConfigurationListener() {
#Override
public void configurationChanged( ConfigurationEvent ce ) {
// zuulProperties.getRoutes() do something
// zuulProperties.getIgnoredPatterns() do something
simpleRouteLocator.refresh();
}
}
}
Every time a property in the file was modified an event was triggered and the ConfigurationEvent was able to deal with it (getPropertyName() and getPropertyValue() to extract data from the event). Since I also Autowired the ZuulProperties I was able to get access to it. With the right rule I could find whether the property of Zuul
zuul.ignoredPatterns
was modified changing its value in the ZuulProperties accordingly.
Here refresh context should work (as long as you are not adding a new routing rule or removing a currently existing one), if you are adding or removing routing rules, you have to add a new bean for ZuulProperties and mark it with #RefreshScope, #Primary.
You can autowire refreshEndpoint bean for example and apply refreshEndpoint.refresh() on the listener.
Marking a custom RouteLocator as primary will cause problems as zuul already has bean of same type marked as primary.
I have a java DSL based spring integration (spring-integration-java-dsl:1.0.1.RELEASE) flow which puts messages through a Filter to filter out certain messages. The Filter component works okay in terms of filtering out unwanted messages.
Now, I would like to set either a discardChannel="discard.ch" but, when I set the discard channel, the filtered out messages never seem to actually go to the specified discardChannel. Any ideas why this might be?
My #Filter annotated class/method:
#Component
public class MessageFilter {
#Filter(discardChannel = "discard.ch")
public boolean filter(String payload) {
// force all messages to be discarded to test discardChannel
return false;
}
}
My Integration Flow class:
#Configuration
#EnableIntegration
public class IntegrationConfig {
#Autowired
private MessageFilter messageFilter;
#Bean(name = "discard.ch")
public DirectChannel discardCh() {
return new DirectChannel();
}
#Bean
public IntegrationFlow inFlow() {
return IntegrationFlows
.from(Jms.messageDriverChannelAdapter(mlc)
.filter("#messageFilter.filter('payload')")
...
.get();
}
#Bean
public IntegrationFlow discardFlow() {
return IntegrationFlows
.from("discard.ch")
...
.get();
}
}
I have turned on spring debugging on and, I can't see where discarded messages are actually going. It is as though the discardChannel I have set on the #Filter is not being picked up at all. Any ideas why this might be?
The annotation configuration is for when using annotation-based configuration.
When using the dsl, the annotation is not relevant; you need to configure the .filter within the DSL itself...
.filter("#messageFilter.filter('payload')", e -> e.discardChannel(discardCh())