I want to build an ActiveMQ Artemis system that can communicate with another cross-region server and perform load balancing and high availability at the same time.
Therefore, I have two virtual machines (172.16.212.32 & 172.16.212.33). Each virtual machine has two replication servers for high availabilit, and I've set up federation in each server to achieve load balancing.
I know that federation can set the ha option, but I think the replication server will perform a better failover effect locally.
There is the setting below.
Master (172.16.212.20) in VM (172.16.212.32) :
<connectors>
<connector name="netty-connector">tcp://172.16.212.20:61616</connector>
<connector name="master-connector33">tcp://172.16.212.22:61616</connector>
<connector name="slave-connector33">tcp://172.16.212.23:61616</connector>
</connectors>
<broadcast-groups>
<broadcast-group name="bg-group32">
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<broadcast-period>5000</broadcast-period>
<connector-ref>netty-connector</connector-ref>
</broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="dg-group32">
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<refresh-timeout>10000</refresh-timeout>
</discovery-group>
</discovery-groups>
<cluster-user>root</cluster-user>
<cluster-password>syscom#1</cluster-password>
<cluster-connections>
<cluster-connection name="cluster32">
<connector-ref>netty-connector</connector-ref>
<message-load-balancing>ON_DEMAND</message-load-balancing>
<max-hops>1</max-hops>
<discovery-group-ref discovery-group-name="dg-group32"/>
</cluster-connection>
</cluster-connections>
<ha-policy>
<replication>
<master>
<check-for-live-server>true</check-for-live-server>
</master>
</replication>
</ha-policy>
<federations>
<federation name="master-federation32">
<upstream name="master-upstream33">
<circuit-breaker-timeout>1000</circuit-breaker-timeout>
<static-connectors>
<connector-ref>master-connector33</connector-ref>
</static-connectors>
<policy ref="policySetA"/>
</upstream>
<upstream name="slave-upstream33">
<circuit-breaker-timeout>1000</circuit-breaker-timeout>
<static-connectors>
<connector-ref>slave-connector33</connector-ref>
</static-connectors>
<policy ref="policySetA"/>
</upstream>
<policy-set name="policySetA">
<policy ref="queue-federation" />
</policy-set>
<queue-policy name="queue-federation" >
<include queue-match="#" address-match="#" />
</queue-policy>
</federation>
</federations>
Slave (172.16.212.21) in VM (172.16.212.32):
<connectors>
<connector name="netty-connector">tcp://172.16.212.21:61616</connector>
<connector name="master-connector33">tcp://172.16.212.22:61616</connector>
<connector name="slave-connector33">tcp://172.16.212.23:61616</connector>
</connectors>
<broadcast-groups>
<broadcast-group name="bg-group32">
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<broadcast-period>5000</broadcast-period>
<connector-ref>netty-connector</connector-ref>
</broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="dg-group32">
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<refresh-timeout>10000</refresh-timeout>
</discovery-group>
</discovery-groups>
<cluster-user>root</cluster-user>
<cluster-password>syscom#1</cluster-password>
<cluster-connections>
<cluster-connection name="cluster32">
<connector-ref>netty-connector</connector-ref>
<message-load-balancing>ON_DEMAND</message-load-balancing>
<max-hops>1</max-hops>
<discovery-group-ref discovery-group-name="dg-group32"/>
</cluster-connection>
</cluster-connections>
<ha-policy>
<replication>
<slave>
<allow-failback>true</allow-failback>
</slave>
</replication>
</ha-policy>
<federations>
<federation name="slave-federation32">
<upstream name="master-upstream33" priority-adjustment="0">
<circuit-breaker-timeout>1000</circuit-breaker-timeout>
<static-connectors>
<connector-ref>master-connector33</connector-ref>
</static-connectors>
<policy ref="policySetA"/>
</upstream>
<upstream name="slave-upstream33" priority-adjustment="0">
<circuit-breaker-timeout>1000</circuit-breaker-timeout>
<static-connectors>
<connector-ref>slave-connector33</connector-ref>
</static-connectors>
<policy ref="policySetA"/>
</upstream>
<policy-set name="policySetA">
<policy ref="queue-federation" />
</policy-set>
<queue-policy name="queue-federation" >
<include queue-match="#" address-match="#" />
</queue-policy>
</federation>
</federations>
172.16.212.22 and 172.16.212.23 are master and slave servers in VM 172.16.212.33.
After set up like that the master server in different VM can communicate with each other, and slave servers announce backup successfully. But the load balancing does not work.
Does the idea of federation plus ha replication policy not work?
I would appreciate it if you have any advice for me.
Related
I'm trying to configure HA on ActiveMQ Artemis 2.13. I'm trying to start with a simple primary and backup. I've read the documentation on clusters and HA a few times how, but I'm still not sure what I'm doing. I've also studied the replicated-failback java example.
From the client, will I have to specify connection information for both the primary and backup nodes? The example has me confused because it looks like the URLs/connection is passed into the java program via input parameters, and I'm not sure where they come from.
In the console for the primary, everything looks normal, but I now have a "broadcast-groups" and "cluster-connections". The secondary only have these two.
On the primary, for Attribute "Failover on server shutdown" is has false...
Here are the HA configurations I made:
Primary (192.168.56.105) broker.xml:
<configuration xmlns="urn:activemq"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xi="http://www.w3.org/2001/XInclude"
xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
<core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:activemq:core ">
<name>0.0.0.0</name>
<persistence-enabled>true</persistence-enabled>
<journal-type>NIO</journal-type>
<paging-directory>data/paging</paging-directory>
<bindings-directory>data/bindings</bindings-directory>
<journal-directory>data/journal</journal-directory>
<large-messages-directory>data/large-messages</large-messages-directory>
<journal-datasync>true</journal-datasync>
<journal-min-files>2</journal-min-files>
<journal-pool-files>10</journal-pool-files>
<journal-device-block-size>4096</journal-device-block-size>
<journal-file-size>10M</journal-file-size>
<journal-buffer-timeout>2884000</journal-buffer-timeout>
<journal-max-io>1</journal-max-io>
<disk-scan-period>5000</disk-scan-period>
<max-disk-usage>90</max-disk-usage>
<critical-analyzer>true</critical-analyzer>
<critical-analyzer-timeout>120000</critical-analyzer-timeout>
<critical-analyzer-check-period>60000</critical-analyzer-check-period>
<critical-analyzer-policy>HALT</critical-analyzer-policy>
<page-sync-timeout>2884000</page-sync-timeout>
<acceptors>
<!-- Acceptor for every supported protocol -->
<acceptor name="artemis">tcp://0.0.0.0:61616?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;amqpMinLargeMessageSize=102400;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpDuplicateDetection=true</acceptor>
<!-- AMQP Acceptor. Listens on default AMQP port for AMQP traffic.-->
<acceptor name="amqp">tcp://0.0.0.0:5672?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpMinLargeMessageSize=102400;amqpDuplicateDetection=true</acceptor>
<!-- STOMP Acceptor. -->
<acceptor name="stomp">tcp://0.0.0.0:61613?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=STOMP;useEpoll=true</acceptor>
<!-- HornetQ Compatibility Acceptor. Enables HornetQ Core and STOMP for legacy HornetQ clients. -->
<acceptor name="hornetq">tcp://0.0.0.0:5445?anycastPrefix=jms.queue.;multicastPrefix=jms.topic.;protocols=HORNETQ,STOMP;useEpoll=true</acceptor>
<!-- MQTT Acceptor -->
<acceptor name="mqtt">tcp://0.0.0.0:1883?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=MQTT;useEpoll=true</acceptor>
</acceptors>
<connectors>
<connector name="artemis">tcp://192.168.56.105:61616</connector>
</connectors>
<broadcast-groups>
<broadcast-group name="broadcast-group-1">
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<connector-ref>artemis</connector-ref>
</broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="discovery-group-1">
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
</discovery-group>
</discovery-groups>
<cluster-user>cluster.user</cluster-user>
<cluster-password>password</cluster-password>
<ha-policy>
<replication>
<master>
<check-for-live-server>true</check-for-live-server>
</master>
</replication>
</ha-policy>
<cluster-connections>
<cluster-connection name="cluster-1">
<connector-ref>artemis</connector-ref>
<discovery-group-ref discovery-group-name="discovery-group-1"/>
</cluster-connection>
</cluster-connections>
<security-settings>
<security-setting match="#">
<permission type="createNonDurableQueue" roles="amq"/>
<permission type="deleteNonDurableQueue" roles="amq"/>
<permission type="createDurableQueue" roles="amq"/>
<permission type="deleteDurableQueue" roles="amq"/>
<permission type="createAddress" roles="amq"/>
<permission type="deleteAddress" roles="amq"/>
<permission type="consume" roles="amq"/>
<permission type="browse" roles="amq"/>
<permission type="send" roles="amq"/>
<!-- we need this otherwise ./artemis data imp wouldn't work -->
<permission type="manage" roles="amq"/>
</security-setting>
</security-settings>
<address-settings>
<!-- if you define auto-create on certain queues, management has to be auto-create -->
<address-setting match="activemq.management#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
<!--default for catch all-->
<address-setting match="#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
</address-settings>
<addresses>
<address name="DLQ">
<anycast>
<queue name="DLQ" />
</anycast>
</address>
<address name="ExpiryQueue">
<anycast>
<queue name="ExpiryQueue" />
</anycast>
</address>
</addresses>
</core>
</configuration>
Backup (192.168.56.106) broker.xml:
<configuration xmlns="urn:activemq"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xi="http://www.w3.org/2001/XInclude"
xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
<core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:activemq:core ">
<name>0.0.0.0</name>
<persistence-enabled>true</persistence-enabled>
<journal-type>NIO</journal-type>
<paging-directory>data/paging</paging-directory>
<bindings-directory>data/bindings</bindings-directory>
<journal-directory>data/journal</journal-directory>
<large-messages-directory>data/large-messages</large-messages-directory>
<journal-datasync>true</journal-datasync>
<journal-min-files>2</journal-min-files>
<journal-pool-files>10</journal-pool-files>
<journal-device-block-size>4096</journal-device-block-size>
<journal-file-size>10M</journal-file-size>
<journal-buffer-timeout>2868000</journal-buffer-timeout>
<journal-max-io>1</journal-max-io>
<disk-scan-period>5000</disk-scan-period>
<max-disk-usage>90</max-disk-usage>
<critical-analyzer>true</critical-analyzer>
<critical-analyzer-timeout>120000</critical-analyzer-timeout>
<critical-analyzer-check-period>60000</critical-analyzer-check-period>
<critical-analyzer-policy>HALT</critical-analyzer-policy>
<page-sync-timeout>2868000</page-sync-timeout>
<acceptors>
<!-- Acceptor for every supported protocol -->
<acceptor name="artemis">tcp://0.0.0.0:61616?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;amqpMinLargeMessageSize=102400;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpDuplicateDetection=true</acceptor>
<!-- AMQP Acceptor. Listens on default AMQP port for AMQP traffic.-->
<acceptor name="amqp">tcp://0.0.0.0:5672?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpMinLargeMessageSize=102400;amqpDuplicateDetection=true</acceptor>
<!-- STOMP Acceptor. -->
<acceptor name="stomp">tcp://0.0.0.0:61613?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=STOMP;useEpoll=true</acceptor>
<!-- HornetQ Compatibility Acceptor. Enables HornetQ Core and STOMP for legacy HornetQ clients. -->
<acceptor name="hornetq">tcp://0.0.0.0:5445?anycastPrefix=jms.queue.;multicastPrefix=jms.topic.;protocols=HORNETQ,STOMP;useEpoll=true</acceptor>
<!-- MQTT Acceptor -->
<acceptor name="mqtt">tcp://0.0.0.0:1883?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=MQTT;useEpoll=true</acceptor>
</acceptors>
<connectors>
<connector name="artemis">tcp://192.168.56.106:61616</connector>
</connectors>
<security-settings>
<security-setting match="#">
<permission type="createNonDurableQueue" roles="amq"/>
<permission type="deleteNonDurableQueue" roles="amq"/>
<permission type="createDurableQueue" roles="amq"/>
<permission type="deleteDurableQueue" roles="amq"/>
<permission type="createAddress" roles="amq"/>
<permission type="deleteAddress" roles="amq"/>
<permission type="consume" roles="amq"/>
<permission type="browse" roles="amq"/>
<permission type="send" roles="amq"/>
<!-- we need this otherwise ./artemis data imp wouldn't work -->
<permission type="manage" roles="amq"/>
</security-setting>
</security-settings>
<address-settings>
<!-- if you define auto-create on certain queues, management has to be auto-create -->
<address-setting match="activemq.management#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
<!--default for catch all-->
<address-setting match="#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
</address-settings>
<addresses>
<address name="DLQ">
<anycast>
<queue name="DLQ" />
</anycast>
</address>
<address name="ExpiryQueue">
<anycast>
<queue name="ExpiryQueue" />
</anycast>
</address>
</addresses>
<broadcast-groups>
<broadcast-group name="broadcast-group-1">
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<connector-ref>artemis</connector-ref>
</broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="discovery-group-1">
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
</discovery-group>
</discovery-groups>
<cluster-user>cluster.user</cluster-user>
<cluster-password>password</cluster-password>
<ha-policy>
<replication>
<slave>
<allow-failback>true</allow-failback>
</slave>
</replication>
</ha-policy>
<cluster-connections>
<cluster-connection name="cluster-1">
<connector-ref>artemis</connector-ref>
<discovery-group-ref discovery-group-name="discovery-group-1"/>
</cluster-connection>
</cluster-connections>
</core>
</configuration>
I only have the default addresses in the broker.xml files - the DLQ and ExpiryQueue address and queue.
Also, here is a screenshot of what is displaying on the console. A lot is missing from the backup server.
Primary:
Backup:
The main thing you need to configure a cluster is a cluster-connection. The cluster-connection references a connector which specifies how other nodes can connect to it. In other words, the connector-ref configured on the cluster-connection should match an acceptor configured on the broker. One problem you have is that both nodes have connectors using 0.0.0.0. This is a meta-address exclusively for use in an acceptor. It will be meaningless for a remote client. Your connectors should the real IP address or hostname of the server where it's running.
Here's what your configuration should probably be:
Primary (192.168.56.105) broker.xml:
<acceptors>
<acceptor name="netty-acceptor">tcp://0.0.0.0:61616</acceptor>
</acceptors>
<connectors>
<connector name="netty-connector">tcp://192.168.56.105:61616</connector>
</connectors>
<broadcast-groups>
<broadcast-group name="broadcast-group-1">
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<connector-ref>netty-connector</connector-ref>
</broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="discovery-group-1">
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
</discovery-group>
</discovery-groups>
<cluster-user>cluster.user</cluster-user>
<cluster-password>password</cluster-password>
<ha-policy>
<replication>
<master>
<check-for-live-server>true</check-for-live-server>
</master>
</replication>
</ha-policy>
<cluster-connections>
<cluster-connection name="cluster-1">
<connector-ref>netty-connector</connector-ref>
<discovery-group-ref discovery-group-name="discovery-group-1"/>
</cluster-connection>
</cluster-connections>
Backup (192.168.56.106) broker.xml:
<acceptor>
<acceptor name="netty-acceptor">tcp://0.0.0.0:61616</acceptor>
</acceptors>
<connectors>
<connector name="netty-connector">tcp://192.168.56.106:61616</connector>
</connectors>
<broadcast-groups>
<broadcast-group name="broadcast-group-1">
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<connector-ref>netty-connector</connector-ref>
</broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="discovery-group-1">
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
</discovery-group>
</discovery-groups>
<cluster-user>cluster.user</cluster-user>
<cluster-password>password</cluster-password>
<ha-policy>
<replication>
<slave>
<allow-failback>true</allow-failback>
</slave>
</replication>
</ha-policy>
<cluster-connections>
<cluster-connection name="cluster-1">
<connector-ref>netty-connector</connector-ref>
<discovery-group-ref discovery-group-name="discovery-group-1"/>
</cluster-connection>
</cluster-connections>
Clients using any supported protocol can use the acceptor listening on 61616. There's no strict need to add any additional acceptor elements.
It's normal for the backup to display limited information on the web console because the backup is passive while the master is active. Once the master goes down the backup will activate and the web console will be fully functional.
I have messages not getting delivered, where I would hope for somebody being able to provide some insights.
Only at times I end up in the following state on my broker - observed via the management console on some specific queue:
countMessages() -> 17
browse() -> empty list
listScheduledMessages() -> shows 17 messages
retryMessage("with some message id") -> the amount of scheduled is then reduced to 16
My application does not send any scheduled messages.
The JMSXDeliveryCount of those scheduled messages is always 0 - so it seems they are not in the process of being redelivered due to some exception in the consumer.
broker.xml:
<configuration xmlns="urn:activemq" xmlns:xi="http://www.w3.org/2001/XInclude"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
<core xmlns="urn:activemq:core" xsi:schemaLocation="urn:activemq:core ">
<name>xxx-artemis-activemq-artemis-master-0</name>
<persistence-enabled>true</persistence-enabled>
<journal-type>ASYNCIO</journal-type>
<paging-directory>data/paging</paging-directory>
<bindings-directory>data/bindings</bindings-directory>
<journal-directory>data/journal</journal-directory>
<large-messages-directory>data/large-messages</large-messages-directory>
<journal-datasync>true</journal-datasync>
<journal-min-files>2</journal-min-files>
<journal-pool-files>10</journal-pool-files>
<journal-device-block-size>4096</journal-device-block-size>
<journal-file-size>10M</journal-file-size>
<journal-buffer-timeout>12000</journal-buffer-timeout>
<journal-max-io>4096</journal-max-io>
<disk-scan-period>5000</disk-scan-period>
<max-disk-usage>90</max-disk-usage>
<acceptors>
<acceptor name="artemis">tcp://0.0.0.0:61616?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpDuplicateDetection=true</acceptor>
<acceptor name="amqp">tcp://0.0.0.0:5672?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpDuplicateDetection=true</acceptor>
<acceptor name="stomp">tcp://0.0.0.0:61613?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=STOMP;useEpoll=true</acceptor>
<acceptor name="hornetq">tcp://0.0.0.0:5445?anycastPrefix=jms.queue.;multicastPrefix=jms.topic.;protocols=HORNETQ,STOMP;useEpoll=true</acceptor>
<acceptor name="mqtt">tcp://0.0.0.0:1883?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=MQTT;useEpoll=true</acceptor>
</acceptors><security-settings>
<security-setting match="#">
<permission type="createNonDurableQueue" roles="amq"/>
<permission type="deleteNonDurableQueue" roles="amq"/>
<permission type="createDurableQueue" roles="amq"/>
<permission type="deleteDurableQueue" roles="amq"/>
<permission type="createAddress" roles="amq"/>
<permission type="deleteAddress" roles="amq"/>
<permission type="consume" roles="amq"/>
<permission type="browse" roles="amq"/>
<permission type="send" roles="amq"/>
<permission type="manage" roles="amq"/>
</security-setting>
</security-settings>
<cluster-user>exampleUser</cluster-user>
<cluster-password>secret</cluster-password>
<address-settings>
<address-setting match="activemq.management#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
<address-setting match="#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>1000</redelivery-delay>
<redelivery-delay-multiplier>5</redelivery-delay-multiplier>
<max-redelivery-delay>360000001</max-redelivery-delay>
<redelivery-collision-avoidance-factor>0.15</redelivery-collision-avoidance-factor>
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
</address-settings><addresses>
<address name="DLQ">
<anycast>
<queue name="DLQ"/>
</anycast>
</address>
<address name="ExpiryQueue">
<anycast>
<queue name="ExpiryQueue"/>
</anycast>
</address>
</addresses><connectors>
<connector name="xxx-artemis-activemq-artemis-master-0">tcp://xxx-artemis-activemq-artemis-master-0.xxx-artemis-activemq-artemis-master.artemis.svc.cluster.local:61616</connector>
<connector name="xxx-artemis-activemq-artemis-slave-0">tcp://xxx-artemis-activemq-artemis-slave-0.xxx-artemis-activemq-artemis-slave.artemis.svc.cluster.local:61616</connector>
</connectors><cluster-connections>
<cluster-connection name="xxx-artemis-activemq-artemis">
<address/>
<connector-ref>xxx-artemis-activemq-artemis-master-0</connector-ref>
<retry-interval>500</retry-interval>
<retry-interval-multiplier>1.1</retry-interval-multiplier>
<max-retry-interval>5000</max-retry-interval>
<initial-connect-attempts>-1</initial-connect-attempts>
<reconnect-attempts>-1</reconnect-attempts>
<message-load-balancing>ON_DEMAND</message-load-balancing>
<max-hops>1</max-hops>
<static-connectors>
<connector-ref>xxx-artemis-activemq-artemis-master-0</connector-ref>
<connector-ref>xxx-artemis-activemq-artemis-slave-0</connector-ref>
</static-connectors>
</cluster-connection>
</cluster-connections><ha-policy>
<replication>
<master>
<group-name>xxx-artemis-activemq-artemis-0</group-name>
<check-for-live-server>false</check-for-live-server>
</master>
</replication>
</ha-policy>
</core><core xmlns="urn:activemq:core">
<jmx-management-enabled>true</jmx-management-enabled>
</core>
</configuration>
ยดยดยด
I have ActiveMQ Artemis configured as a high available cluster (a master at 192.168.2.111 and two slaves at 192.168.2.112 and 192.168.2.113) with replication. I would like to use SSL (with self-signed certificates) between the brokers themselves as well as between the clients and the brokers.
I generated keyStore and trustore keys and included in broker.xml config on each of the cluster like this:
<connectors>
<connector name="netty-connector">tcp://192.168.2.111:61616?sslEnabled=true;keyStorePath=broker_ks.p12;keyStorePassword=artemis;trustStorePath=broker_ts.p12;trustStorePassword=artemis</connector>
</connectors>
<acceptors>
<acceptor name="netty-acceptor">tcp://192.168.2.111:61616?sslEnabled=true;keyStorePath=broker_ks.p12;keyStorePassword=artemis;trustStorePath=broker_ts.p12;trustStorePassword=artemis</acceptor>
</acceptors>
The other part of broker.xml is unchanged.
When I start now master and slave I can reach them through "Management console" but in the log of master I see error:
2020-01-29 16:49:19,395 ERROR [org.apache.activemq.artemis.core.server] AMQ224088: Timeout (10 seconds) on acceptor "netty-acceptor" during protocol handshake with /192.168.2.113:55118 has occurred.
Can somebody please explain how to fix this error?
This is how I'm generating keystore and truststore on each of the cluster nodes:
# Create a broker key and cert - import the keypair and cert into the broker keystore
openssl req -newkey rsa:2048 -nodes -keyout broker_keypair.pem -x509 -days 65000 -out broker_cert.pem
openssl pkcs12 -inkey broker_keypair.pem -in broker_cert.pem -export -out broker_ks.p12
# Create a client key and cert - import the keypair and cert into the client keystore
openssl req -newkey rsa:2048 -nodes -keyout client_keypair.pem -x509 -days 65000 -out client_cert.pem
openssl pkcs12 -inkey client_keypair.pem -in client_cert.pem -export -out client_ks.p12
# Create a truststore for the broker, and import the client's certificate. This establishes that the broker "trusts" the client:
keytool -import -alias client -keystore broker_ts.p12 -file client_cert.pem -deststoretype pkcs12
# Create a truststore for the client, and import the broker's certificate. This establishes that the client "trusts" the broker:
keytool -import -alias broker -keystore client_ts.p12 -file broker_cert.pem -deststoretype pkcs12
my broker.xml for master
<?xml version='1.0'?>
<configuration xmlns="urn:activemq"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xi="http://www.w3.org/2001/XInclude"
xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
<core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:activemq:core ">
<name>0.0.0.0</name>
<persistence-enabled>true</persistence-enabled>
<journal-type>ASYNCIO</journal-type>
<paging-directory>data/paging</paging-directory>
<bindings-directory>data/bindings</bindings-directory>
<journal-directory>data/journal</journal-directory>
<large-messages-directory>data/large-messages</large-messages-directory>
<journal-datasync>true</journal-datasync>
<journal-min-files>2</journal-min-files>
<journal-pool-files>10</journal-pool-files>
<journal-device-block-size>4096</journal-device-block-size>
<journal-file-size>10M</journal-file-size>
<journal-buffer-timeout>28000</journal-buffer-timeout>
<journal-max-io>4096</journal-max-io>
<!-- how often we are looking for how many bytes are being used on the disk in ms -->
<disk-scan-period>5000</disk-scan-period>
<max-disk-usage>100</max-disk-usage>
<!-- should the broker detect dead locks and other issues -->
<critical-analyzer>true</critical-analyzer>
<critical-analyzer-timeout>120000</critical-analyzer-timeout>
<critical-analyzer-check-period>60000</critical-analyzer-check-period>
<critical-analyzer-policy>HALT</critical-analyzer-policy>
<page-sync-timeout>1628000</page-sync-timeout>
<global-max-size>204Mb</global-max-size>
<!-- Connectors -->
<connectors>
<connector name="netty-connector">tcp://artemis01:61616</connector>
</connectors>
<acceptors>
<acceptor name="netty-ssl-acceptor">tcp://artemis01:61616?sslEnabled=true;needClientAuth=true;keyStorePath=/home/vagrant/ssl/server-side-keystore.jks;keyStorePassword=secureexample;trustStorePath=/home/vagrant/ssl/server-side-truststore.jks;trustStorePassword=secureexample</acceptor>
</acceptors>
<cluster-connections>
<cluster-connection name="my-cluster">
<address>amq</address>
<connector-ref>netty-connector</connector-ref>
<retry-interval>1000</retry-interval>
<retry-interval-multiplier>3</retry-interval-multiplier>
<use-duplicate-detection>true</use-duplicate-detection>
<message-load-balancing>STRICT</message-load-balancing>
<discovery-group-ref discovery-group-name="my-discovery-group"/>
</cluster-connection>
</cluster-connections>
<broadcast-groups>
<broadcast-group name="my-broadcast-group">
<local-bind-address>artemis01</local-bind-address>
<local-bind-port>9876</local-bind-port>
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<broadcast-period>2000</broadcast-period>
<connector-ref>netty-connector</connector-ref>
</broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="my-discovery-group">
<local-bind-address>artemis01</local-bind-address>
<local-bind-port>9876</local-bind-port>
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<refresh-timeout>10000</refresh-timeout>
</discovery-group>
</discovery-groups>
<network-check-list>artemis01,artemis02,artemis03</network-check-list>
<network-check-period>5000</network-check-period>
<network-check-timeout>1000</network-check-timeout>
<network-check-ping-command>ping -c 1 -t %d %s</network-check-ping-command>
<network-check-ping6-command>ping6 -c 1 %2$s</network-check-ping6-command>
<!-- Other config -->
<ha-policy>
<replication>
<master>
<check-for-live-server>true</check-for-live-server>
</master>
</replication>
</ha-policy>
<security-settings>
<security-setting match="#">
<permission type="createNonDurableQueue" roles="amq"/>
<permission type="deleteNonDurableQueue" roles="amq"/>
<permission type="createDurableQueue" roles="amq"/>
<permission type="deleteDurableQueue" roles="amq"/>
<permission type="createAddress" roles="amq"/>
<permission type="deleteAddress" roles="amq"/>
<permission type="consume" roles="amq"/>
<permission type="browse" roles="amq"/>
<permission type="send" roles="amq"/>
<!-- we need this otherwise ./artemis data imp wouldn't work -->
<permission type="manage" roles="amq"/>
</security-setting>
</security-settings>
<addresses>
<address name="exampleQueue">
<anycast>
<queue name="exampleQueue"/>
</anycast>
</address>
<address name="DLQ">
<anycast>
<queue name="DLQ" />
</anycast>
</address>
<address name="ExpiryQueue">
<anycast>
<queue name="ExpiryQueue" />
</anycast>
</address>
</addresses>
<address-settings>
<!-- if you define auto-create on certain queues, management has to be auto-create -->
<address-setting match="activemq.management#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
<!--default for catch all-->
<address-setting match="#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
<address-setting match="exampleQueue">
<dead-letter-address>DLQ</dead-letter-address>
<redelivery-delay>1000</redelivery-delay>
<max-delivery-attempts>3</max-delivery-attempts>
<max-size-bytes>-1</max-size-bytes>
<page-size-bytes>1048576</page-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
</address-setting>
</address-settings>
</core>
</configuration>
and broker.xml for slave:
<?xml version='1.0'?>
<configuration xmlns="urn:activemq"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xi="http://www.w3.org/2001/XInclude"
xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
<core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:activemq:core ">
<name>0.0.0.0</name>
<persistence-enabled>true</persistence-enabled>
<journal-type>ASYNCIO</journal-type>
<paging-directory>data/paging</paging-directory>
<bindings-directory>data/bindings</bindings-directory>
<journal-directory>data/journal</journal-directory>
<large-messages-directory>data/large-messages</large-messages-directory>
<journal-datasync>true</journal-datasync>
<journal-min-files>2</journal-min-files>
<journal-pool-files>10</journal-pool-files>
<journal-device-block-size>4096</journal-device-block-size>
<journal-file-size>10M</journal-file-size>
<journal-buffer-timeout>28000</journal-buffer-timeout>
<journal-max-io>4096</journal-max-io>
<disk-scan-period>5000</disk-scan-period>
<max-disk-usage>100</max-disk-usage>
<!-- should the broker detect dead locks and other issues -->
<critical-analyzer>true</critical-analyzer>
<critical-analyzer-timeout>120000</critical-analyzer-timeout>
<critical-analyzer-check-period>60000</critical-analyzer-check-period>
<critical-analyzer-policy>HALT</critical-analyzer-policy>
<page-sync-timeout>1628000</page-sync-timeout>
<global-max-size>204Mb</global-max-size>
<!-- Connectors -->
<connectors>
<connector name="netty-connector">tcp://artemis02:61616</connector>
</connectors>
<acceptors>
<acceptor name="netty-ssl-acceptor">tcp://artemis02:61616?sslEnabled=true;needClientAuth=true;keyStorePath=/home/vagrant/ssl/server-side-keystore.jks;keyStorePassword=secureexample;trustStorePath=/home/vagrant/ssl/server-side-truststore.jks;trustStorePassword=secureexample</acceptor>
</acceptors>
<cluster-connections>
<cluster-connection name="my-cluster">
<address>amq</address>
<connector-ref>netty-connector</connector-ref>
<retry-interval>1000</retry-interval>
<retry-interval-multiplier>3</retry-interval-multiplier>
<use-duplicate-detection>true</use-duplicate-detection>
<message-load-balancing>STRICT</message-load-balancing>
<discovery-group-ref discovery-group-name="my-discovery-group"/>
</cluster-connection>
</cluster-connections>
<broadcast-groups>
<broadcast-group name="my-broadcast-group">
<local-bind-address>artemis02</local-bind-address>
<local-bind-port>9876</local-bind-port>
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<broadcast-period>2000</broadcast-period>
<connector-ref>netty-connector</connector-ref>
</broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="my-discovery-group">
<local-bind-address>artemis02</local-bind-address>
<local-bind-port>9876</local-bind-port>
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<refresh-timeout>10000</refresh-timeout>
</discovery-group>
</discovery-groups>
<network-check-list>artemis01,artemis02,artemis03</network-check-list>
<network-check-period>5000</network-check-period>
<network-check-timeout>1000</network-check-timeout>
<network-check-ping-command>ping -c 1 -t %d %s</network-check-ping-command>
<network-check-ping6-command>ping6 -c 1 %2$s</network-check-ping6-command>
<!-- Other config -->
<ha-policy>
<replication>
<slave>
<allow-failback>true</allow-failback>
<failback-delay>5000</failback-delay>
</slave>
</replication>
</ha-policy>
<security-settings>
<security-setting match="#">
<permission type="createNonDurableQueue" roles="amq"/>
<permission type="deleteNonDurableQueue" roles="amq"/>
<permission type="createDurableQueue" roles="amq"/>
<permission type="deleteDurableQueue" roles="amq"/>
<permission type="createAddress" roles="amq"/>
<permission type="deleteAddress" roles="amq"/>
<permission type="consume" roles="amq"/>
<permission type="browse" roles="amq"/>
<permission type="send" roles="amq"/>
<!-- we need this otherwise ./artemis data imp wouldn't work -->
<permission type="manage" roles="amq"/>
</security-setting>
</security-settings>
<addresses>
<address name="exampleQueue">
<anycast>
<queue name="exampleQueue"/>
</anycast>
</address>
<address name="DLQ">
<anycast>
<queue name="DLQ" />
</anycast>
</address>
<address name="ExpiryQueue">
<anycast>
<queue name="ExpiryQueue" />
</anycast>
</address>
</addresses>
<address-settings>
<!-- if you define auto-create on certain queues, management has to be auto-create -->
<address-setting match="activemq.management#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
<!--default for catch all-->
<address-setting match="#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
<address-setting match="exampleQueue">
<dead-letter-address>DLQ</dead-letter-address>
<redelivery-delay>1000</redelivery-delay>
<max-delivery-attempts>3</max-delivery-attempts>
<max-size-bytes>-1</max-size-bytes>
<page-size-bytes>1048576</page-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
</address-setting>
</address-settings>
</core>
</configuration>
Connectors have two functions. Defining how to connect to other AMQ brokers and the connector parameters are send to a client when it connects the first time. If a client is going to use these parameters the keystore/truststore path, name and password must exist and be the same on all clients. Best is not to set the key/truststore parameters on a connector then a proper client will use the values from org.apache.activemq.ssl.* or javax.net.ssl.*.
See https://access.redhat.com/documentation/en-us/red_hat_amq/7.4/html/configuring_amq_broker/acceptor_connector_params
I am still struggling to get this to work.
AMQ broker HA cluster works without ssl
AMQ broker works with ssl
AMQ broker HA cluster still fails.
Truststore was a JKS and PKCS12 was expected. Now it works.
I'm using Wildfly 9.1 in domain mode.
I have three hosts: A , B and C.
All nodes have a test server with the same test.ear.
After starting the three nodes, all ClusterConnectionBridge are created correctly between them.
Each node send messages on commonTopic, and an MDB logs the received messages.
The problem is that what I see logged on the node A is only the messages sent by himself. On the other nodes i see logged all the messages sent by A , B , C . This is weird, since using tcpdump on node A i see the incoming messages on port 8080 from the other nodes.
Using cli to get metrics on node A:
[domain#MASTER:19990 /] /host=SLAVE.A/server=test/subsystem=messaging/hornetq-server=default/jms-topic=commonTopic:read-resource(recursive=true,proxies=false,include-runtime=true,include-defaults=true)
{
"outcome" => "success",
"result" => {
"delivering-count" => 0,
"durable-message-count" => 0,
"durable-subscription-count" => 1,
"entries" => [
"java:/jms/commonTopic",
"java:jboss/exported/jms/commonTopic"
],
"message-count" => 0L,
"messages-added" => 150L,
"non-durable-message-count" => 0,
"non-durable-subscription-count" => 3,
"subscription-count" => 4,
"temporary" => false,
"topic-address" => "jms.topic.commonTopic"
}
}
I see that the number of added messages is 3 times the messages sent by one node. So it seems messages are received through the bridge but not consumed by the MDB.
Could someone give me an idea where to look for this issue.
The hosts are on virtual machines , on the same LAN.
The same happens if I start a node D on my PC and join it to the cluster. Node D logs all messages from the other 3 nodes, but on the other nodes no D message are logged.
The following is a snippet from my domain.xml regarding HornetQ configuration:
<subsystem xmlns="urn:jboss:domain:messaging:3.0">
<hornetq-server>
<security-enabled>false</security-enabled>
<jmx-management-enabled>true</jmx-management-enabled>
<journal-file-size>102400</journal-file-size>
<connectors>
<http-connector name="http-connector" socket-binding="http">
<param key="http-upgrade-endpoint" value="http-acceptor"/>
</http-connector>
<http-connector name="http-connector-throughput" socket-binding="http">
<param key="http-upgrade-endpoint" value="http-acceptor-throughput"/>
<param key="batch-delay" value="50"/>
</http-connector>
<netty-connector name="netty" socket-binding="messaging"/>
<netty-connector name="netty-throughput" socket-binding="messaging-throughput">
<param key="batch-delay" value="50"/>
</netty-connector>
<in-vm-connector name="in-vm" server-id="0"/>
</connectors>
<acceptors>
<http-acceptor http-listener="default" name="http-acceptor"/>
<http-acceptor http-listener="default" name="http-acceptor-throughput">
<param key="batch-delay" value="50"/>
<param key="direct-deliver" value="false"/>
</http-acceptor>
<netty-acceptor name="netty" socket-binding="messaging"/>
<netty-acceptor name="netty-throughput" socket-binding="messaging-throughput">
<param key="batch-delay" value="50"/>
<param key="direct-deliver" value="false"/>
</netty-acceptor>
<in-vm-acceptor name="in-vm" server-id="0"/>
</acceptors>
<broadcast-groups>
<broadcast-group name="bg-group">
<jgroups-stack>udp</jgroups-stack>
<jgroups-channel>hq-cluster</jgroups-channel>
<connector-ref>
http-connector
</connector-ref>
</broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="dg-group">
<jgroups-stack>udp</jgroups-stack>
<jgroups-channel>hq-cluster</jgroups-channel>
</discovery-group>
</discovery-groups>
<cluster-connections>
<cluster-connection name="my-cluster">
<address>jms</address>
<connector-ref>http-connector</connector-ref>
<use-duplicate-detection>true</use-duplicate-detection>
<forward-when-no-consumers>false</forward-when-no-consumers>
<max-hops>1</max-hops>
<discovery-group-ref discovery-group-name="dg-group"/>
</cluster-connection>
</cluster-connections>
<security-settings>
<security-setting match="#">
<permission type="send" roles="guest"/>
<permission type="consume" roles="guest"/>
<permission type="createNonDurableQueue" roles="guest"/>
<permission type="deleteNonDurableQueue" roles="guest"/>
</security-setting>
</security-settings>
<address-settings>
<address-setting match="#">
<dead-letter-address>jms.queue.DLQ</dead-letter-address>
<expiry-address>jms.queue.ExpiryQueue</expiry-address>
<max-size-bytes>10485760</max-size-bytes>
<page-size-bytes>2097152</page-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<redistribution-delay>1000</redistribution-delay>
</address-setting>
</address-settings>
<jms-connection-factories>
<connection-factory name="InVmConnectionFactory">
<connectors>
<connector-ref connector-name="in-vm"/>
</connectors>
<entries>
<entry name="java:/jms/MessageConnectionFactory"/>
<entry name="java:jboss/jms/MessageConnectionFactory"/>
<entry name="java:/ConnectionFactory"/>
</entries>
<reconnect-attempts>-1</reconnect-attempts>
</connection-factory>
<connection-factory name="RemoteConnectionFactory">
<connectors>
<connector-ref connector-name="http-connector"/>
</connectors>
<entries>
<entry name="java:jboss/exported/jms/MessageConnectionFactory"/>
<entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
</entries>
<ha>true</ha>
<block-on-acknowledge>true</block-on-acknowledge>
<reconnect-attempts>-1</reconnect-attempts>
</connection-factory>
<pooled-connection-factory name="hornetq-ra">
<transaction mode="xa"/>
<connectors>
<connector-ref connector-name="in-vm"/>
</connectors>
<entries>
<entry name="java:/JmsXA"/>
<entry name="java:jboss/DefaultJMSConnectionFactory"/>
</entries>
</pooled-connection-factory>
</jms-connection-factories>
<jms-destinations>
<jms-queue name="ExpiryQueue">
<entry name="jms/queue/ExpiryQueue"/>
<entry name="java:jboss/exported/jms/queue/ExpiryQueue"/>
<durable>true</durable>
</jms-queue>
<jms-queue name="DLQ">
<entry name="jms/queue/DLQ"/>
<entry name="java:jboss/exported/jms/queue/DLQ"/>
<durable>true</durable>
</jms-queue>
<jms-topic name="commonTopic">
<entry name="java:/jms/commonTopic"/>
<entry name="java:jboss/exported/jms/commonTopic"/>
</jms-topic>
</jms-topic>
</jms-destinations>
</hornetq-server>
</subsystem>
I found out the problem was caused by message persistence.
A 5 seconds time to live was not enough for node A to consume the message.
I found the expired messages on expiredQueue.
Each node sends one message every 10 seconds, so i don't understand the reason why the messages expire on node A.
Setup:
Two Linux machines
Start Server in standalone mode
configuration used is standalone-full-ha.xml
cluster-demo.war deployed to both nodes with tag
Cluster successfully formed between the two nodes. I can see the jgroups messages and having their cluster views updated correctly.
Scenario:
Call node1/cluster-demo/put.jsp
Call node1/cluster-demo/get.jsp (Session lives)
Restart node1. Session persists
Call node2/cluster-demo/get.jsp
Session is null (why?)
<subsystem xmlns="urn:jboss:domain:infinispan:3.0">
<cache-container name="server" default-cache="default" module="org.wildfly.clustering.server" aliases="singleton cluster">
<transport lock-timeout="60000"/>
<distributed-cache name="default" mode="SYNC">
<transaction mode="BATCH"/>
</distributed-cache>
</cache-container>
<cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">
<transport lock-timeout="60000"/>
<distributed-cache name="dist" mode="ASYNC" owners="2" l1-lifespan="0">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
<cache-container name="ejb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan" aliases="sfsb">
<transport lock-timeout="60000"/>
<distributed-cache name="dist" mode="ASYNC" owners="2" l1-lifespan="0">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
<cache-container name="hibernate" default-cache="local-query" module="org.hibernate.infinispan">
<transport lock-timeout="60000"/>
<local-cache name="local-query">
<eviction strategy="LRU" max-entries="10000"/>
<expiration max-idle="100000"/>
</local-cache>
<invalidation-cache name="entity" mode="SYNC">
<transaction mode="NON_XA"/>
<eviction strategy="LRU" max-entries="10000"/>
<expiration max-idle="100000"/>
</invalidation-cache>
<replicated-cache name="timestamps" mode="ASYNC"/>
</cache-container>
</subsystem>