ActiveMQ Artemis - Queue published to but not consumed - spring-boot

Question

I have a test application (Spring Boot 2.7.8) that uses ActiveMQ Artemis 2.27.1 as a messaging system. I have a 6 node cluster split into 3 live/backup pairs. Load balanced using ON_DEMAND with a redistribution delay of 2000.
The application creates a connection factory specifying all 3 live nodes and creates a withHA connection factory.
I have a generator class that publishes messages to a single queue. There is a consumer of that queue which replicates this message to 3 different queues. I am aware of topics and wish to move there eventually, but I am modeling an existing solution which does this sort of thing now.
Testing shows that I publish a message and consume it, it publishes to the other 3 queues but only consumes from 2 of them despite all having listeners. Checking the queues after execution shows that it has sent messages to the queue. This is consistent over several runs, the same queue is never consumed whilst I am generating 'new' events.
If I disable the initial generation of new messages and just rerun, the missing queue is then drained by its listener.
This feels like when the connections are made, this queue has a publisher on one node and the consumer on another and the redistribution is not happening. Not sure how I can prove this or why, if the publishing node does not have consumers, it is not redistributing to the consumer.
Connection factory bean
#Bean
public ActiveMQConnectionFactory jmsConnectionFactory() throws Exception {
HashMap<String, Object> map1 = new HashMap<>();
map1.put("host", "192.168.0.10");
map1.put("port", "61616");
HashMap<String, Object> map2 = new HashMap<>();
map2.put("host", "192.168.0.11");
map2.put("port", "61617");
HashMap<String, Object> map3 = new HashMap<>();
map3.put(TransportConstants.HOST_PROP_NAME, "192.168.0.12");
map3.put(TransportConstants.PORT_PROP_NAME, "61618");
TransportConfiguration server1 = new TransportConfiguration(NettyConnectorFactory.class.getName(), map1);
TransportConfiguration server2 = new TransportConfiguration(NettyConnectorFactory.class.getName(), map2);
TransportConfiguration server3 = new TransportConfiguration(NettyConnectorFactory.class.getName(), map3);
ActiveMQConnectionFactory connectionFactory = ActiveMQJMSClient.createConnectionFactoryWithHA(JMSFactoryType.CF, server1, server2, server3);
ActiveMQJMSClient.createConnectionFactoryWithoutHA(JMSFactoryType.QUEUE_CF, server1);
connectionFactory.setPassword(brokerPassword);
connectionFactory.setUser(brokerUsername);
return connectionFactory;
}
Listener factory bean
#Bean
public DefaultJmsListenerContainerFactory jmsQueueListenerContainerFactory() throws Exception {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(jmsConnectionFactory());
//factory.setConcurrency("4-10");
factory.setSessionAcknowledgeMode(Session.CLIENT_ACKNOWLEDGE);
factory.setSessionTransacted(true);
return factory;
}
This handler listens to the initial published queue and splits
#Slf4j
#Component
#RequiredArgsConstructor
public class TransactionManagerListener {
private final JmsTemplate jmsTemplate;
/**
*
* Handle the ItemStatsUpdate event
*
* #param data - Event details wrapper object
* #throws RuntimeException that triggers a retry for that item following the backoff rules in the retryable
*/
#JmsListener(destination = "NewItem", containerFactory = "jmsQueueListenerContainerFactory")
public void podA(Session session, Message message, String data) throws RuntimeException {
log.info("TML {}!", data);
sendItemOn(data);
}
private void sendItemOn(String data) {
jmsTemplate.convertAndSend("Stash", data);
jmsTemplate.convertAndSend("PCE", data);
jmsTemplate.convertAndSend("ACD", data);
}
}
Extract from broker.xml. All nodes are slightly different to hook up the differnt live servers and their backup
<connectors>
<connector name="live1-connector">tcp://192.168.0.10:61616</connector>
<connector name="live2-connector">tcp://192.168.0.11:61617</connector>
<connector name="live3-connector">tcp://192.168.0.12:61618</connector>
<connector name="back1-connector">tcp://192.168.0.13:61619</connector>
<connector name="back2-connector">tcp://192.168.0.10:61620</connector>
<connector name="back3-connector">tcp://192.168.0.11:61621</connector>
</connectors>
<cluster-user>my-cluster-user</cluster-user>
<cluster-password>my-cluster-password</cluster-password>
<cluster-connections>
<cluster-connection name="my-cluster">
<connector-ref>live2-connector</connector-ref>
<message-load-balancing>ON_DEMAND</message-load-balancing>
<static-connectors>
<connector-ref>live1-connector</connector-ref>
<connector-ref>live3-connector</connector-ref>
<connector-ref>back2-connector</connector-ref>
<!--
<connector-ref>back1-connector</connector-ref>
<connector-ref>back3-connector</connector-ref>
-->
</static-connectors>
</cluster-connection>
</cluster-connections>
<ha-policy>
<replication>
<master>
<group-name>gloucester</group-name>
<check-for-live-server>true</check-for-live-server>
</master>
</replication>
</ha-policy>
As you can see form the commented out concurrency setting I have tried to tweak the threads and consumers available in the listener factory but it made no difference.

Answer 1

This came down to network configuration. It is necessary to run the cluster on the local swarm network but provide ingress connectors for the JMS client.
<connectors>
<connector name="live1-connector">tcp://192.168.0.10:61616</connector>
<connector name="live2-connector">tcp://192.168.0.11:61617</connector>
<connector name="live3-connector">tcp://192.168.0.12:61618</connector>
<connector name="back1-connector">tcp://192.168.0.13:61619</connector>
<connector name="back2-connector">tcp://192.168.0.10:61620</connector>
<connector name="back3-connector">tcp://192.168.0.11:61621</connector>
<connector name="live1-connector">tcp://live1:61616</connector>
<connector name="live2-connector">tcp://live2:61616</connector>
<connector name="live3-connector">tcp://live3:61616</connector>
<connector name="back1-connector">tcp://back1:61616</connector>
<connector name="back2-connector">tcp://back2:61616</connector>
<connector name="back3-connector">tcp://back3:61616</connector>
</connectors>
<cluster-user>my-cluster-user</cluster-user>
<cluster-password>my-cluster-password</cluster-password>
<cluster-connections>
<cluster-connection name="my-cluster">
<connector-ref>live3-connector</connector-ref>
<static-connectors>
<connector-ref>back3-connector</connector-ref>
<connector-ref>live1-connector</connector-ref>
<connector-ref>back1-connector</connector-ref>
<connector-ref>live2-connector</connector-ref>
<connector-ref>back2-connector</connector-ref>
</static-connectors>
</cluster-connection>
</cluster-connections>