"Dispatcher has no subscribers" during startup - spring

Question

I am currently facing a problem which only happens sometimes and is very hard to reproduce. Therefore I have problems actually creating a compelling test case.
Our setup looks like this:
spring integration with spring boot
rabbitmq listener
custom bus which manages transaction aware messages
The setup is not the newest anymore and probably a lot of code could be replaced by more idiomatic spring wiring.
Though this is the exception I am getting when the service is started:
Dispatcher has no subscribers, failedMessage=GenericMessage [payload=com.foobar.service.greencheckout.message.GreenOrderPropertyAddedMessage#2ba0efb2, headers={id=f750a792-6b01-16d3-8206-6e553a03f8fa, type=com.foobar.service.greencheckout.message.GreenOrderPropertyAddedMessage, amqp_deliveryMode=PERSISTENT, timestamp=1505797680967}], failedMessage=GenericMessage [payload=com.foobar.service.greencheckout.message.GreenOrderPropertyAddedMessage#2ba0efb2, headers={id=f750a792-6b01-16d3-8206-6e553a03f8fa, type=com.foobar.service.greencheckout.message.GreenOrderPropertyAddedMessage, amqp_deliveryMode=PERSISTENT, timestamp=1505797680967}]
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:93)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:423)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:373)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:105)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.convertAndSend(AbstractMessageSendingTemplate.java:143)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.convertAndSend(AbstractMessageSendingTemplate.java:135)
at org.springframework.integration.gateway.MessagingGatewaySupport.send(MessagingGatewaySupport.java:392)
at org.springframework.integration.gateway.GatewayProxyFactoryBean.invokeGatewayMethod(GatewayProxyFactoryBean.java:477)
at org.springframework.integration.gateway.GatewayProxyFactoryBean.doInvoke(GatewayProxyFactoryBean.java:429)
at org.springframework.integration.gateway.GatewayProxyFactoryBean.invoke(GatewayProxyFactoryBean.java:420)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy223.send(Unknown Source)
at com.foobar.library.messaging.bus.TransactionAwareBus.afterCommit(TransactionAwareBus.java:50)
at org.springframework.transaction.support.TransactionSynchronizationUtils.invokeAfterCommit(TransactionSynchronizationUtils.java:133)
at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerAfterCommit(TransactionSynchronizationUtils.java:121)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerAfterCommit(AbstractPlatformTransactionManager.java:958)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:803)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:730)
at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:504)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:292)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
at com.foobar.service.greencheckout.silo.domain.modifyingservice.TransitionService$$EnhancerBySpringCGLIB$$de36730c.execute(<generated>)
at com.foobar.service.greencheckout.silo.domain.statemachine.OrderFsm.invokeCreationalCommandMethod(OrderFsm.java:380)
at com.foobar.service.greencheckout.silo.domain.statemachine.OrderFsm.lambda$allowCreational$1(OrderFsm.java:345)
at akka.japi.pf.FSMStateFunctionBuilder$2.apply(FSMStateFunctionBuilder.java:80)
at akka.japi.pf.FSMStateFunctionBuilder$2.apply(FSMStateFunctionBuilder.java:77)
at akka.japi.pf.CaseStatement.apply(CaseStatements.scala:18)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
at akka.japi.pf.CaseStatement.applyOrElse(CaseStatements.scala:13)
at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
at akka.japi.pf.CaseStatement.applyOrElse(CaseStatements.scala:13)
at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
at akka.japi.pf.CaseStatement.applyOrElse(CaseStatements.scala:13)
at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
at akka.japi.pf.CaseStatement.applyOrElse(CaseStatements.scala:13)
at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)
at akka.actor.FSM$class.processEvent(FSM.scala:663)
at akka.actor.AbstractFSM.processEvent(AbstractFSM.scala:36)
at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:657)
at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:651)
at akka.actor.Actor$class.aroundReceive(Actor.scala:497)
at akka.actor.AbstractFSM.aroundReceive(AbstractFSM.scala:36)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers, failedMessage=GenericMessage [payload=com.foobar.service.greencheckout.message.GreenOrderPropertyAddedMessage#2ba0efb2, headers={id=f750a792-6b01-16d3-8206-6e553a03f8fa, type=com.foobar.service.greencheckout.message.GreenOrderPropertyAddedMessage, amqp_deliveryMode=PERSISTENT, timestamp=1505797680967}]
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:154)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:121)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:89)
... 58 more
The code to set this up looks like:
import org.springframework.amqp.core.AmqpAdmin;
import org.springframework.amqp.core.AmqpTemplate;
import org.springframework.amqp.core.BindingBuilder;
import org.springframework.amqp.core.Exchange;
import org.springframework.amqp.core.Queue;
import org.springframework.beans.factory.BeanFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.DependsOn;
import org.springframework.context.annotation.Scope;
import org.springframework.context.annotation.ScopedProxyMode;
import org.springframework.expression.ExpressionParser;
import org.springframework.integration.amqp.outbound.AmqpOutboundEndpoint;
import org.springframework.integration.amqp.support.DefaultAmqpHeaderMapper;
import org.springframework.integration.channel.DirectChannel;
import org.springframework.integration.channel.ExecutorChannel;
import org.springframework.integration.channel.NullChannel;
import org.springframework.integration.endpoint.EventDrivenConsumer;
import org.springframework.integration.gateway.GatewayProxyFactoryBean;
import org.springframework.messaging.SubscribableChannel;
import java.util.concurrent.Executors;
#Configuration
public class MessagingOutboundConfiguration
{
#Autowired
private AmqpTemplate amqpTemplate;
#Autowired
private Exchange publisherExchange;
#Autowired
private AmqpAdmin amqpAdmin;
#Autowired
private Exchange errorExchange;
#Autowired
private Queue errorQueue;
#Autowired
private BeanFactory beanFactory;
#Autowired
private ExpressionParser expressionParser;
#Autowired
private MessagingSettings messagingSettings;
#Bean
#DependsOn({"connectionFactory", "consumer"})
public AsynchronousBus asyncBus(SubscribableChannel amqpOutboundChannel) throws Exception
{
GatewayProxyFactoryBean factoryBean = new GatewayProxyFactoryBean(AsynchronousBus.class);
factoryBean.setBeanFactory(beanFactory);
factoryBean.setDefaultRequestChannel(amqpOutboundChannel);
factoryBean.afterPropertiesSet();
return (AsynchronousBus) factoryBean.getObject();
}
#Bean
#Scope(value = "request", proxyMode = ScopedProxyMode.TARGET_CLASS)
public TransactionAwareBus transactionAwareBus()
{
return new TransactionAwareBus();
}
/**
* Channel from message bus to the outbound channel adapter
*/
#Bean
public SubscribableChannel amqpOutboundChannel(HeaderChannelInterceptor headerChannelInterceptor)
{
DirectChannel channel = new DirectChannel();
channel.setComponentName("amqp-outbound-channel");
channel.addInterceptor(headerChannelInterceptor);
return channel;
}
/**
* Outbound Channel Adapter
*/
#Bean
public AmqpOutboundEndpoint endpoint(SubscribableChannel confirmationChannel)
{
DefaultAmqpHeaderMapper headerMapper = DefaultAmqpHeaderMapper.outboundMapper();
String[] allowedHeaders = new String[]{"*"};
headerMapper.setRequestHeaderNames(allowedHeaders);
headerMapper.setReplyHeaderNames(allowedHeaders);
AmqpOutboundEndpoint endpoint = new PhasedAmqpOutboundEndpoint(amqpTemplate);
endpoint.setHeaderMapper(headerMapper);
endpoint.setExchangeName(publisherExchange.getName());
endpoint.setRoutingKeyExpression(expressionParser.parseExpression("headers.type"));
endpoint.setConfirmCorrelationExpression(expressionParser.parseExpression("#this"));
if (messagingSettings.getPublisherConfirmations()) {
endpoint.setConfirmNackChannel(confirmationChannel);
endpoint.setConfirmAckChannel(new NullChannel());
}
return endpoint;
}
#Bean
public HeaderChannelInterceptor headerChannelInterceptor()
{
return new HeaderChannelInterceptor();
}
/**
* Shovels messages from channel to outbound channel adapter
*/
#Bean
public EventDrivenConsumer consumer(SubscribableChannel amqpOutboundChannel, AmqpOutboundEndpoint endpoint)
{
final EventDrivenConsumer consumer = new EventDrivenConsumer(amqpOutboundChannel, endpoint);
consumer.setBeanName("amqp-outbound-consumer");
return consumer;
}
/**
* Outbound Channel Adapter
*/
#Bean
public AmqpOutboundEndpoint errorEndpoint()
{
amqpAdmin.declareBinding(BindingBuilder.bind(errorQueue).to(errorExchange).with("#").noargs());
DefaultAmqpHeaderMapper headerMapper = DefaultAmqpHeaderMapper.outboundMapper();
String[] allowedHeaders = new String[1];
allowedHeaders[0] = "*";
headerMapper.setRequestHeaderNames(allowedHeaders);
AmqpOutboundEndpoint endpoint = new PhasedAmqpOutboundEndpoint(amqpTemplate);
endpoint.setHeaderMapper(headerMapper);
endpoint.setExchangeName(errorExchange.getName());
endpoint.setRoutingKeyExpression(expressionParser.parseExpression("headers.routing"));
return endpoint;
}
#Bean
public EventDrivenConsumer errorConsumer(SubscribableChannel errorChannel, AmqpOutboundEndpoint errorEndpoint)
{
final EventDrivenConsumer consumer = new EventDrivenConsumer(errorChannel, errorEndpoint);
consumer.setBeanName("amqp-error-consumer");
return consumer;
}
#Bean
public SubscribableChannel errorChannel()
{
DirectChannel channel = new DirectChannel();
channel.setComponentName("amqp-error-channel");
return channel;
}
#Bean
public SubscribableChannel confirmationChannel()
{
ExecutorChannel channel = new ExecutorChannel(Executors.newSingleThreadExecutor());
channel.setComponentName("amqp-confirmation-channel");
return channel;
}
}
From my understanding:
This message normally means (especially in the shutdown case) that beans are not correctly wired or that the context does not know the best order for stopping them.
What I do not understand: What's wrong with my setup? :(
UPDATE:
As you can see the code is called from inside an actor. The actor system is configured this way:
#Bean
#DependsOn({"asyncBus", "prototypedTransactionAwareBus", "transactionAwareBus", "syncBus"})
public SpringActorSystem actorSystem() throws Exception
{
String profile = environment.getActiveProfiles()[0];
ActorSystem system = ActorSystem.create(akkaSettings.getSystemName(), akkaConfiguration(profile));
if ("testing".equals(profile)) {
Cluster.get(system).joinSeedNodes(Lists.newArrayList(Cluster.get(system).selfAddress()));
}
if ("kubernetes".equals(profile)) {
joinKubernetesSeedNodes(system);
}
SpringExtension.SpringExtProvider.get(system).initialize(context);
return new SpringActorSystem(system);
}
And the SpringActorSystem:
public class SpringActorSystem implements ApplicationListener<ContextClosedEvent>
{
private static final Logger LOGGER = LoggerFactory.getLogger(SpringActorSystem.class);
private final ActorSystem actorSystem;
public SpringActorSystem(ActorSystem actorSystem)
{
this.actorSystem = actorSystem;
}
public ActorSystem system()
{
return actorSystem;
}
#Override
public void onApplicationEvent(ContextClosedEvent event)
{
final Cluster cluster = Cluster.get(this.actorSystem);
cluster.leave(cluster.selfAddress());
LOGGER.info("SpringActorSystem shutdown initiated");
this.actorSystem.terminate();
try {
Await.result(actorSystem.whenTerminated(), Duration.create(10, TimeUnit.SECONDS));
} catch (Exception e) {
LOGGER.info("Exception while waiting for termination", e);
}
LOGGER.info("SpringActorSystem shutdown finished");
}
}

Answer 1

Well, I think
SpringExtension.SpringExtProvider.get(system).initialize(context);
in the initializing phase is really too early.
Consider to implement SmartLifecycle for your SpringActorSystem to mover that initialize(context) to the start() method.