I'm trying to configure HA on ActiveMQ Artemis 2.13. I'm trying to start with a simple primary and backup. I've read the documentation on clusters and HA a few times how, but I'm still not sure what I'm doing. I've also studied the replicated-failback java example.
From the client, will I have to specify connection information for both the primary and backup nodes? The example has me confused because it looks like the URLs/connection is passed into the java program via input parameters, and I'm not sure where they come from.
In the console for the primary, everything looks normal, but I now have a "broadcast-groups" and "cluster-connections". The secondary only have these two.
On the primary, for Attribute "Failover on server shutdown" is has false...
Here are the HA configurations I made:
Primary (192.168.56.105) broker.xml:
<configuration xmlns="urn:activemq"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xi="http://www.w3.org/2001/XInclude"
xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
<core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:activemq:core ">
<name>0.0.0.0</name>
<persistence-enabled>true</persistence-enabled>
<journal-type>NIO</journal-type>
<paging-directory>data/paging</paging-directory>
<bindings-directory>data/bindings</bindings-directory>
<journal-directory>data/journal</journal-directory>
<large-messages-directory>data/large-messages</large-messages-directory>
<journal-datasync>true</journal-datasync>
<journal-min-files>2</journal-min-files>
<journal-pool-files>10</journal-pool-files>
<journal-device-block-size>4096</journal-device-block-size>
<journal-file-size>10M</journal-file-size>
<journal-buffer-timeout>2884000</journal-buffer-timeout>
<journal-max-io>1</journal-max-io>
<disk-scan-period>5000</disk-scan-period>
<max-disk-usage>90</max-disk-usage>
<critical-analyzer>true</critical-analyzer>
<critical-analyzer-timeout>120000</critical-analyzer-timeout>
<critical-analyzer-check-period>60000</critical-analyzer-check-period>
<critical-analyzer-policy>HALT</critical-analyzer-policy>
<page-sync-timeout>2884000</page-sync-timeout>
<acceptors>
<!-- Acceptor for every supported protocol -->
<acceptor name="artemis">tcp://0.0.0.0:61616?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;amqpMinLargeMessageSize=102400;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpDuplicateDetection=true</acceptor>
<!-- AMQP Acceptor. Listens on default AMQP port for AMQP traffic.-->
<acceptor name="amqp">tcp://0.0.0.0:5672?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpMinLargeMessageSize=102400;amqpDuplicateDetection=true</acceptor>
<!-- STOMP Acceptor. -->
<acceptor name="stomp">tcp://0.0.0.0:61613?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=STOMP;useEpoll=true</acceptor>
<!-- HornetQ Compatibility Acceptor. Enables HornetQ Core and STOMP for legacy HornetQ clients. -->
<acceptor name="hornetq">tcp://0.0.0.0:5445?anycastPrefix=jms.queue.;multicastPrefix=jms.topic.;protocols=HORNETQ,STOMP;useEpoll=true</acceptor>
<!-- MQTT Acceptor -->
<acceptor name="mqtt">tcp://0.0.0.0:1883?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=MQTT;useEpoll=true</acceptor>
</acceptors>
<connectors>
<connector name="artemis">tcp://192.168.56.105:61616</connector>
</connectors>
<broadcast-groups>
<broadcast-group name="broadcast-group-1">
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<connector-ref>artemis</connector-ref>
</broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="discovery-group-1">
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
</discovery-group>
</discovery-groups>
<cluster-user>cluster.user</cluster-user>
<cluster-password>password</cluster-password>
<ha-policy>
<replication>
<master>
<check-for-live-server>true</check-for-live-server>
</master>
</replication>
</ha-policy>
<cluster-connections>
<cluster-connection name="cluster-1">
<connector-ref>artemis</connector-ref>
<discovery-group-ref discovery-group-name="discovery-group-1"/>
</cluster-connection>
</cluster-connections>
<security-settings>
<security-setting match="#">
<permission type="createNonDurableQueue" roles="amq"/>
<permission type="deleteNonDurableQueue" roles="amq"/>
<permission type="createDurableQueue" roles="amq"/>
<permission type="deleteDurableQueue" roles="amq"/>
<permission type="createAddress" roles="amq"/>
<permission type="deleteAddress" roles="amq"/>
<permission type="consume" roles="amq"/>
<permission type="browse" roles="amq"/>
<permission type="send" roles="amq"/>
<!-- we need this otherwise ./artemis data imp wouldn't work -->
<permission type="manage" roles="amq"/>
</security-setting>
</security-settings>
<address-settings>
<!-- if you define auto-create on certain queues, management has to be auto-create -->
<address-setting match="activemq.management#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
<!--default for catch all-->
<address-setting match="#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
</address-settings>
<addresses>
<address name="DLQ">
<anycast>
<queue name="DLQ" />
</anycast>
</address>
<address name="ExpiryQueue">
<anycast>
<queue name="ExpiryQueue" />
</anycast>
</address>
</addresses>
</core>
</configuration>
Backup (192.168.56.106) broker.xml:
<configuration xmlns="urn:activemq"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xi="http://www.w3.org/2001/XInclude"
xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
<core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:activemq:core ">
<name>0.0.0.0</name>
<persistence-enabled>true</persistence-enabled>
<journal-type>NIO</journal-type>
<paging-directory>data/paging</paging-directory>
<bindings-directory>data/bindings</bindings-directory>
<journal-directory>data/journal</journal-directory>
<large-messages-directory>data/large-messages</large-messages-directory>
<journal-datasync>true</journal-datasync>
<journal-min-files>2</journal-min-files>
<journal-pool-files>10</journal-pool-files>
<journal-device-block-size>4096</journal-device-block-size>
<journal-file-size>10M</journal-file-size>
<journal-buffer-timeout>2868000</journal-buffer-timeout>
<journal-max-io>1</journal-max-io>
<disk-scan-period>5000</disk-scan-period>
<max-disk-usage>90</max-disk-usage>
<critical-analyzer>true</critical-analyzer>
<critical-analyzer-timeout>120000</critical-analyzer-timeout>
<critical-analyzer-check-period>60000</critical-analyzer-check-period>
<critical-analyzer-policy>HALT</critical-analyzer-policy>
<page-sync-timeout>2868000</page-sync-timeout>
<acceptors>
<!-- Acceptor for every supported protocol -->
<acceptor name="artemis">tcp://0.0.0.0:61616?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;amqpMinLargeMessageSize=102400;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpDuplicateDetection=true</acceptor>
<!-- AMQP Acceptor. Listens on default AMQP port for AMQP traffic.-->
<acceptor name="amqp">tcp://0.0.0.0:5672?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpMinLargeMessageSize=102400;amqpDuplicateDetection=true</acceptor>
<!-- STOMP Acceptor. -->
<acceptor name="stomp">tcp://0.0.0.0:61613?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=STOMP;useEpoll=true</acceptor>
<!-- HornetQ Compatibility Acceptor. Enables HornetQ Core and STOMP for legacy HornetQ clients. -->
<acceptor name="hornetq">tcp://0.0.0.0:5445?anycastPrefix=jms.queue.;multicastPrefix=jms.topic.;protocols=HORNETQ,STOMP;useEpoll=true</acceptor>
<!-- MQTT Acceptor -->
<acceptor name="mqtt">tcp://0.0.0.0:1883?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=MQTT;useEpoll=true</acceptor>
</acceptors>
<connectors>
<connector name="artemis">tcp://192.168.56.106:61616</connector>
</connectors>
<security-settings>
<security-setting match="#">
<permission type="createNonDurableQueue" roles="amq"/>
<permission type="deleteNonDurableQueue" roles="amq"/>
<permission type="createDurableQueue" roles="amq"/>
<permission type="deleteDurableQueue" roles="amq"/>
<permission type="createAddress" roles="amq"/>
<permission type="deleteAddress" roles="amq"/>
<permission type="consume" roles="amq"/>
<permission type="browse" roles="amq"/>
<permission type="send" roles="amq"/>
<!-- we need this otherwise ./artemis data imp wouldn't work -->
<permission type="manage" roles="amq"/>
</security-setting>
</security-settings>
<address-settings>
<!-- if you define auto-create on certain queues, management has to be auto-create -->
<address-setting match="activemq.management#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
<!--default for catch all-->
<address-setting match="#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
</address-settings>
<addresses>
<address name="DLQ">
<anycast>
<queue name="DLQ" />
</anycast>
</address>
<address name="ExpiryQueue">
<anycast>
<queue name="ExpiryQueue" />
</anycast>
</address>
</addresses>
<broadcast-groups>
<broadcast-group name="broadcast-group-1">
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<connector-ref>artemis</connector-ref>
</broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="discovery-group-1">
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
</discovery-group>
</discovery-groups>
<cluster-user>cluster.user</cluster-user>
<cluster-password>password</cluster-password>
<ha-policy>
<replication>
<slave>
<allow-failback>true</allow-failback>
</slave>
</replication>
</ha-policy>
<cluster-connections>
<cluster-connection name="cluster-1">
<connector-ref>artemis</connector-ref>
<discovery-group-ref discovery-group-name="discovery-group-1"/>
</cluster-connection>
</cluster-connections>
</core>
</configuration>
I only have the default addresses in the broker.xml files - the DLQ and ExpiryQueue address and queue.
Also, here is a screenshot of what is displaying on the console. A lot is missing from the backup server.
Primary:
Backup:
The main thing you need to configure a cluster is a cluster-connection. The cluster-connection references a connector which specifies how other nodes can connect to it. In other words, the connector-ref configured on the cluster-connection should match an acceptor configured on the broker. One problem you have is that both nodes have connectors using 0.0.0.0. This is a meta-address exclusively for use in an acceptor. It will be meaningless for a remote client. Your connectors should the real IP address or hostname of the server where it's running.
Here's what your configuration should probably be:
Primary (192.168.56.105) broker.xml:
<acceptors>
<acceptor name="netty-acceptor">tcp://0.0.0.0:61616</acceptor>
</acceptors>
<connectors>
<connector name="netty-connector">tcp://192.168.56.105:61616</connector>
</connectors>
<broadcast-groups>
<broadcast-group name="broadcast-group-1">
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<connector-ref>netty-connector</connector-ref>
</broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="discovery-group-1">
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
</discovery-group>
</discovery-groups>
<cluster-user>cluster.user</cluster-user>
<cluster-password>password</cluster-password>
<ha-policy>
<replication>
<master>
<check-for-live-server>true</check-for-live-server>
</master>
</replication>
</ha-policy>
<cluster-connections>
<cluster-connection name="cluster-1">
<connector-ref>netty-connector</connector-ref>
<discovery-group-ref discovery-group-name="discovery-group-1"/>
</cluster-connection>
</cluster-connections>
Backup (192.168.56.106) broker.xml:
<acceptor>
<acceptor name="netty-acceptor">tcp://0.0.0.0:61616</acceptor>
</acceptors>
<connectors>
<connector name="netty-connector">tcp://192.168.56.106:61616</connector>
</connectors>
<broadcast-groups>
<broadcast-group name="broadcast-group-1">
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<connector-ref>netty-connector</connector-ref>
</broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="discovery-group-1">
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
</discovery-group>
</discovery-groups>
<cluster-user>cluster.user</cluster-user>
<cluster-password>password</cluster-password>
<ha-policy>
<replication>
<slave>
<allow-failback>true</allow-failback>
</slave>
</replication>
</ha-policy>
<cluster-connections>
<cluster-connection name="cluster-1">
<connector-ref>netty-connector</connector-ref>
<discovery-group-ref discovery-group-name="discovery-group-1"/>
</cluster-connection>
</cluster-connections>
Clients using any supported protocol can use the acceptor listening on 61616. There's no strict need to add any additional acceptor elements.
It's normal for the backup to display limited information on the web console because the backup is passive while the master is active. Once the master goes down the backup will activate and the web console will be fully functional.
Related
I want to build an ActiveMQ Artemis system that can communicate with another cross-region server and perform load balancing and high availability at the same time.
Therefore, I have two virtual machines (172.16.212.32 & 172.16.212.33). Each virtual machine has two replication servers for high availabilit, and I've set up federation in each server to achieve load balancing.
I know that federation can set the ha option, but I think the replication server will perform a better failover effect locally.
There is the setting below.
Master (172.16.212.20) in VM (172.16.212.32) :
<connectors>
<connector name="netty-connector">tcp://172.16.212.20:61616</connector>
<connector name="master-connector33">tcp://172.16.212.22:61616</connector>
<connector name="slave-connector33">tcp://172.16.212.23:61616</connector>
</connectors>
<broadcast-groups>
<broadcast-group name="bg-group32">
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<broadcast-period>5000</broadcast-period>
<connector-ref>netty-connector</connector-ref>
</broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="dg-group32">
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<refresh-timeout>10000</refresh-timeout>
</discovery-group>
</discovery-groups>
<cluster-user>root</cluster-user>
<cluster-password>syscom#1</cluster-password>
<cluster-connections>
<cluster-connection name="cluster32">
<connector-ref>netty-connector</connector-ref>
<message-load-balancing>ON_DEMAND</message-load-balancing>
<max-hops>1</max-hops>
<discovery-group-ref discovery-group-name="dg-group32"/>
</cluster-connection>
</cluster-connections>
<ha-policy>
<replication>
<master>
<check-for-live-server>true</check-for-live-server>
</master>
</replication>
</ha-policy>
<federations>
<federation name="master-federation32">
<upstream name="master-upstream33">
<circuit-breaker-timeout>1000</circuit-breaker-timeout>
<static-connectors>
<connector-ref>master-connector33</connector-ref>
</static-connectors>
<policy ref="policySetA"/>
</upstream>
<upstream name="slave-upstream33">
<circuit-breaker-timeout>1000</circuit-breaker-timeout>
<static-connectors>
<connector-ref>slave-connector33</connector-ref>
</static-connectors>
<policy ref="policySetA"/>
</upstream>
<policy-set name="policySetA">
<policy ref="queue-federation" />
</policy-set>
<queue-policy name="queue-federation" >
<include queue-match="#" address-match="#" />
</queue-policy>
</federation>
</federations>
Slave (172.16.212.21) in VM (172.16.212.32):
<connectors>
<connector name="netty-connector">tcp://172.16.212.21:61616</connector>
<connector name="master-connector33">tcp://172.16.212.22:61616</connector>
<connector name="slave-connector33">tcp://172.16.212.23:61616</connector>
</connectors>
<broadcast-groups>
<broadcast-group name="bg-group32">
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<broadcast-period>5000</broadcast-period>
<connector-ref>netty-connector</connector-ref>
</broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="dg-group32">
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<refresh-timeout>10000</refresh-timeout>
</discovery-group>
</discovery-groups>
<cluster-user>root</cluster-user>
<cluster-password>syscom#1</cluster-password>
<cluster-connections>
<cluster-connection name="cluster32">
<connector-ref>netty-connector</connector-ref>
<message-load-balancing>ON_DEMAND</message-load-balancing>
<max-hops>1</max-hops>
<discovery-group-ref discovery-group-name="dg-group32"/>
</cluster-connection>
</cluster-connections>
<ha-policy>
<replication>
<slave>
<allow-failback>true</allow-failback>
</slave>
</replication>
</ha-policy>
<federations>
<federation name="slave-federation32">
<upstream name="master-upstream33" priority-adjustment="0">
<circuit-breaker-timeout>1000</circuit-breaker-timeout>
<static-connectors>
<connector-ref>master-connector33</connector-ref>
</static-connectors>
<policy ref="policySetA"/>
</upstream>
<upstream name="slave-upstream33" priority-adjustment="0">
<circuit-breaker-timeout>1000</circuit-breaker-timeout>
<static-connectors>
<connector-ref>slave-connector33</connector-ref>
</static-connectors>
<policy ref="policySetA"/>
</upstream>
<policy-set name="policySetA">
<policy ref="queue-federation" />
</policy-set>
<queue-policy name="queue-federation" >
<include queue-match="#" address-match="#" />
</queue-policy>
</federation>
</federations>
172.16.212.22 and 172.16.212.23 are master and slave servers in VM 172.16.212.33.
After set up like that the master server in different VM can communicate with each other, and slave servers announce backup successfully. But the load balancing does not work.
Does the idea of federation plus ha replication policy not work?
I would appreciate it if you have any advice for me.
I have messages not getting delivered, where I would hope for somebody being able to provide some insights.
Only at times I end up in the following state on my broker - observed via the management console on some specific queue:
countMessages() -> 17
browse() -> empty list
listScheduledMessages() -> shows 17 messages
retryMessage("with some message id") -> the amount of scheduled is then reduced to 16
My application does not send any scheduled messages.
The JMSXDeliveryCount of those scheduled messages is always 0 - so it seems they are not in the process of being redelivered due to some exception in the consumer.
broker.xml:
<configuration xmlns="urn:activemq" xmlns:xi="http://www.w3.org/2001/XInclude"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
<core xmlns="urn:activemq:core" xsi:schemaLocation="urn:activemq:core ">
<name>xxx-artemis-activemq-artemis-master-0</name>
<persistence-enabled>true</persistence-enabled>
<journal-type>ASYNCIO</journal-type>
<paging-directory>data/paging</paging-directory>
<bindings-directory>data/bindings</bindings-directory>
<journal-directory>data/journal</journal-directory>
<large-messages-directory>data/large-messages</large-messages-directory>
<journal-datasync>true</journal-datasync>
<journal-min-files>2</journal-min-files>
<journal-pool-files>10</journal-pool-files>
<journal-device-block-size>4096</journal-device-block-size>
<journal-file-size>10M</journal-file-size>
<journal-buffer-timeout>12000</journal-buffer-timeout>
<journal-max-io>4096</journal-max-io>
<disk-scan-period>5000</disk-scan-period>
<max-disk-usage>90</max-disk-usage>
<acceptors>
<acceptor name="artemis">tcp://0.0.0.0:61616?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpDuplicateDetection=true</acceptor>
<acceptor name="amqp">tcp://0.0.0.0:5672?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpDuplicateDetection=true</acceptor>
<acceptor name="stomp">tcp://0.0.0.0:61613?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=STOMP;useEpoll=true</acceptor>
<acceptor name="hornetq">tcp://0.0.0.0:5445?anycastPrefix=jms.queue.;multicastPrefix=jms.topic.;protocols=HORNETQ,STOMP;useEpoll=true</acceptor>
<acceptor name="mqtt">tcp://0.0.0.0:1883?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=MQTT;useEpoll=true</acceptor>
</acceptors><security-settings>
<security-setting match="#">
<permission type="createNonDurableQueue" roles="amq"/>
<permission type="deleteNonDurableQueue" roles="amq"/>
<permission type="createDurableQueue" roles="amq"/>
<permission type="deleteDurableQueue" roles="amq"/>
<permission type="createAddress" roles="amq"/>
<permission type="deleteAddress" roles="amq"/>
<permission type="consume" roles="amq"/>
<permission type="browse" roles="amq"/>
<permission type="send" roles="amq"/>
<permission type="manage" roles="amq"/>
</security-setting>
</security-settings>
<cluster-user>exampleUser</cluster-user>
<cluster-password>secret</cluster-password>
<address-settings>
<address-setting match="activemq.management#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
<address-setting match="#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>1000</redelivery-delay>
<redelivery-delay-multiplier>5</redelivery-delay-multiplier>
<max-redelivery-delay>360000001</max-redelivery-delay>
<redelivery-collision-avoidance-factor>0.15</redelivery-collision-avoidance-factor>
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
</address-settings><addresses>
<address name="DLQ">
<anycast>
<queue name="DLQ"/>
</anycast>
</address>
<address name="ExpiryQueue">
<anycast>
<queue name="ExpiryQueue"/>
</anycast>
</address>
</addresses><connectors>
<connector name="xxx-artemis-activemq-artemis-master-0">tcp://xxx-artemis-activemq-artemis-master-0.xxx-artemis-activemq-artemis-master.artemis.svc.cluster.local:61616</connector>
<connector name="xxx-artemis-activemq-artemis-slave-0">tcp://xxx-artemis-activemq-artemis-slave-0.xxx-artemis-activemq-artemis-slave.artemis.svc.cluster.local:61616</connector>
</connectors><cluster-connections>
<cluster-connection name="xxx-artemis-activemq-artemis">
<address/>
<connector-ref>xxx-artemis-activemq-artemis-master-0</connector-ref>
<retry-interval>500</retry-interval>
<retry-interval-multiplier>1.1</retry-interval-multiplier>
<max-retry-interval>5000</max-retry-interval>
<initial-connect-attempts>-1</initial-connect-attempts>
<reconnect-attempts>-1</reconnect-attempts>
<message-load-balancing>ON_DEMAND</message-load-balancing>
<max-hops>1</max-hops>
<static-connectors>
<connector-ref>xxx-artemis-activemq-artemis-master-0</connector-ref>
<connector-ref>xxx-artemis-activemq-artemis-slave-0</connector-ref>
</static-connectors>
</cluster-connection>
</cluster-connections><ha-policy>
<replication>
<master>
<group-name>xxx-artemis-activemq-artemis-0</group-name>
<check-for-live-server>false</check-for-live-server>
</master>
</replication>
</ha-policy>
</core><core xmlns="urn:activemq:core">
<jmx-management-enabled>true</jmx-management-enabled>
</core>
</configuration>
ยดยดยด
I have ActiveMQ Artemis configured as a high available cluster (a master at 192.168.2.111 and two slaves at 192.168.2.112 and 192.168.2.113) with replication. I would like to use SSL (with self-signed certificates) between the brokers themselves as well as between the clients and the brokers.
I generated keyStore and trustore keys and included in broker.xml config on each of the cluster like this:
<connectors>
<connector name="netty-connector">tcp://192.168.2.111:61616?sslEnabled=true;keyStorePath=broker_ks.p12;keyStorePassword=artemis;trustStorePath=broker_ts.p12;trustStorePassword=artemis</connector>
</connectors>
<acceptors>
<acceptor name="netty-acceptor">tcp://192.168.2.111:61616?sslEnabled=true;keyStorePath=broker_ks.p12;keyStorePassword=artemis;trustStorePath=broker_ts.p12;trustStorePassword=artemis</acceptor>
</acceptors>
The other part of broker.xml is unchanged.
When I start now master and slave I can reach them through "Management console" but in the log of master I see error:
2020-01-29 16:49:19,395 ERROR [org.apache.activemq.artemis.core.server] AMQ224088: Timeout (10 seconds) on acceptor "netty-acceptor" during protocol handshake with /192.168.2.113:55118 has occurred.
Can somebody please explain how to fix this error?
This is how I'm generating keystore and truststore on each of the cluster nodes:
# Create a broker key and cert - import the keypair and cert into the broker keystore
openssl req -newkey rsa:2048 -nodes -keyout broker_keypair.pem -x509 -days 65000 -out broker_cert.pem
openssl pkcs12 -inkey broker_keypair.pem -in broker_cert.pem -export -out broker_ks.p12
# Create a client key and cert - import the keypair and cert into the client keystore
openssl req -newkey rsa:2048 -nodes -keyout client_keypair.pem -x509 -days 65000 -out client_cert.pem
openssl pkcs12 -inkey client_keypair.pem -in client_cert.pem -export -out client_ks.p12
# Create a truststore for the broker, and import the client's certificate. This establishes that the broker "trusts" the client:
keytool -import -alias client -keystore broker_ts.p12 -file client_cert.pem -deststoretype pkcs12
# Create a truststore for the client, and import the broker's certificate. This establishes that the client "trusts" the broker:
keytool -import -alias broker -keystore client_ts.p12 -file broker_cert.pem -deststoretype pkcs12
my broker.xml for master
<?xml version='1.0'?>
<configuration xmlns="urn:activemq"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xi="http://www.w3.org/2001/XInclude"
xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
<core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:activemq:core ">
<name>0.0.0.0</name>
<persistence-enabled>true</persistence-enabled>
<journal-type>ASYNCIO</journal-type>
<paging-directory>data/paging</paging-directory>
<bindings-directory>data/bindings</bindings-directory>
<journal-directory>data/journal</journal-directory>
<large-messages-directory>data/large-messages</large-messages-directory>
<journal-datasync>true</journal-datasync>
<journal-min-files>2</journal-min-files>
<journal-pool-files>10</journal-pool-files>
<journal-device-block-size>4096</journal-device-block-size>
<journal-file-size>10M</journal-file-size>
<journal-buffer-timeout>28000</journal-buffer-timeout>
<journal-max-io>4096</journal-max-io>
<!-- how often we are looking for how many bytes are being used on the disk in ms -->
<disk-scan-period>5000</disk-scan-period>
<max-disk-usage>100</max-disk-usage>
<!-- should the broker detect dead locks and other issues -->
<critical-analyzer>true</critical-analyzer>
<critical-analyzer-timeout>120000</critical-analyzer-timeout>
<critical-analyzer-check-period>60000</critical-analyzer-check-period>
<critical-analyzer-policy>HALT</critical-analyzer-policy>
<page-sync-timeout>1628000</page-sync-timeout>
<global-max-size>204Mb</global-max-size>
<!-- Connectors -->
<connectors>
<connector name="netty-connector">tcp://artemis01:61616</connector>
</connectors>
<acceptors>
<acceptor name="netty-ssl-acceptor">tcp://artemis01:61616?sslEnabled=true;needClientAuth=true;keyStorePath=/home/vagrant/ssl/server-side-keystore.jks;keyStorePassword=secureexample;trustStorePath=/home/vagrant/ssl/server-side-truststore.jks;trustStorePassword=secureexample</acceptor>
</acceptors>
<cluster-connections>
<cluster-connection name="my-cluster">
<address>amq</address>
<connector-ref>netty-connector</connector-ref>
<retry-interval>1000</retry-interval>
<retry-interval-multiplier>3</retry-interval-multiplier>
<use-duplicate-detection>true</use-duplicate-detection>
<message-load-balancing>STRICT</message-load-balancing>
<discovery-group-ref discovery-group-name="my-discovery-group"/>
</cluster-connection>
</cluster-connections>
<broadcast-groups>
<broadcast-group name="my-broadcast-group">
<local-bind-address>artemis01</local-bind-address>
<local-bind-port>9876</local-bind-port>
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<broadcast-period>2000</broadcast-period>
<connector-ref>netty-connector</connector-ref>
</broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="my-discovery-group">
<local-bind-address>artemis01</local-bind-address>
<local-bind-port>9876</local-bind-port>
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<refresh-timeout>10000</refresh-timeout>
</discovery-group>
</discovery-groups>
<network-check-list>artemis01,artemis02,artemis03</network-check-list>
<network-check-period>5000</network-check-period>
<network-check-timeout>1000</network-check-timeout>
<network-check-ping-command>ping -c 1 -t %d %s</network-check-ping-command>
<network-check-ping6-command>ping6 -c 1 %2$s</network-check-ping6-command>
<!-- Other config -->
<ha-policy>
<replication>
<master>
<check-for-live-server>true</check-for-live-server>
</master>
</replication>
</ha-policy>
<security-settings>
<security-setting match="#">
<permission type="createNonDurableQueue" roles="amq"/>
<permission type="deleteNonDurableQueue" roles="amq"/>
<permission type="createDurableQueue" roles="amq"/>
<permission type="deleteDurableQueue" roles="amq"/>
<permission type="createAddress" roles="amq"/>
<permission type="deleteAddress" roles="amq"/>
<permission type="consume" roles="amq"/>
<permission type="browse" roles="amq"/>
<permission type="send" roles="amq"/>
<!-- we need this otherwise ./artemis data imp wouldn't work -->
<permission type="manage" roles="amq"/>
</security-setting>
</security-settings>
<addresses>
<address name="exampleQueue">
<anycast>
<queue name="exampleQueue"/>
</anycast>
</address>
<address name="DLQ">
<anycast>
<queue name="DLQ" />
</anycast>
</address>
<address name="ExpiryQueue">
<anycast>
<queue name="ExpiryQueue" />
</anycast>
</address>
</addresses>
<address-settings>
<!-- if you define auto-create on certain queues, management has to be auto-create -->
<address-setting match="activemq.management#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
<!--default for catch all-->
<address-setting match="#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
<address-setting match="exampleQueue">
<dead-letter-address>DLQ</dead-letter-address>
<redelivery-delay>1000</redelivery-delay>
<max-delivery-attempts>3</max-delivery-attempts>
<max-size-bytes>-1</max-size-bytes>
<page-size-bytes>1048576</page-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
</address-setting>
</address-settings>
</core>
</configuration>
and broker.xml for slave:
<?xml version='1.0'?>
<configuration xmlns="urn:activemq"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xi="http://www.w3.org/2001/XInclude"
xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
<core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:activemq:core ">
<name>0.0.0.0</name>
<persistence-enabled>true</persistence-enabled>
<journal-type>ASYNCIO</journal-type>
<paging-directory>data/paging</paging-directory>
<bindings-directory>data/bindings</bindings-directory>
<journal-directory>data/journal</journal-directory>
<large-messages-directory>data/large-messages</large-messages-directory>
<journal-datasync>true</journal-datasync>
<journal-min-files>2</journal-min-files>
<journal-pool-files>10</journal-pool-files>
<journal-device-block-size>4096</journal-device-block-size>
<journal-file-size>10M</journal-file-size>
<journal-buffer-timeout>28000</journal-buffer-timeout>
<journal-max-io>4096</journal-max-io>
<disk-scan-period>5000</disk-scan-period>
<max-disk-usage>100</max-disk-usage>
<!-- should the broker detect dead locks and other issues -->
<critical-analyzer>true</critical-analyzer>
<critical-analyzer-timeout>120000</critical-analyzer-timeout>
<critical-analyzer-check-period>60000</critical-analyzer-check-period>
<critical-analyzer-policy>HALT</critical-analyzer-policy>
<page-sync-timeout>1628000</page-sync-timeout>
<global-max-size>204Mb</global-max-size>
<!-- Connectors -->
<connectors>
<connector name="netty-connector">tcp://artemis02:61616</connector>
</connectors>
<acceptors>
<acceptor name="netty-ssl-acceptor">tcp://artemis02:61616?sslEnabled=true;needClientAuth=true;keyStorePath=/home/vagrant/ssl/server-side-keystore.jks;keyStorePassword=secureexample;trustStorePath=/home/vagrant/ssl/server-side-truststore.jks;trustStorePassword=secureexample</acceptor>
</acceptors>
<cluster-connections>
<cluster-connection name="my-cluster">
<address>amq</address>
<connector-ref>netty-connector</connector-ref>
<retry-interval>1000</retry-interval>
<retry-interval-multiplier>3</retry-interval-multiplier>
<use-duplicate-detection>true</use-duplicate-detection>
<message-load-balancing>STRICT</message-load-balancing>
<discovery-group-ref discovery-group-name="my-discovery-group"/>
</cluster-connection>
</cluster-connections>
<broadcast-groups>
<broadcast-group name="my-broadcast-group">
<local-bind-address>artemis02</local-bind-address>
<local-bind-port>9876</local-bind-port>
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<broadcast-period>2000</broadcast-period>
<connector-ref>netty-connector</connector-ref>
</broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="my-discovery-group">
<local-bind-address>artemis02</local-bind-address>
<local-bind-port>9876</local-bind-port>
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<refresh-timeout>10000</refresh-timeout>
</discovery-group>
</discovery-groups>
<network-check-list>artemis01,artemis02,artemis03</network-check-list>
<network-check-period>5000</network-check-period>
<network-check-timeout>1000</network-check-timeout>
<network-check-ping-command>ping -c 1 -t %d %s</network-check-ping-command>
<network-check-ping6-command>ping6 -c 1 %2$s</network-check-ping6-command>
<!-- Other config -->
<ha-policy>
<replication>
<slave>
<allow-failback>true</allow-failback>
<failback-delay>5000</failback-delay>
</slave>
</replication>
</ha-policy>
<security-settings>
<security-setting match="#">
<permission type="createNonDurableQueue" roles="amq"/>
<permission type="deleteNonDurableQueue" roles="amq"/>
<permission type="createDurableQueue" roles="amq"/>
<permission type="deleteDurableQueue" roles="amq"/>
<permission type="createAddress" roles="amq"/>
<permission type="deleteAddress" roles="amq"/>
<permission type="consume" roles="amq"/>
<permission type="browse" roles="amq"/>
<permission type="send" roles="amq"/>
<!-- we need this otherwise ./artemis data imp wouldn't work -->
<permission type="manage" roles="amq"/>
</security-setting>
</security-settings>
<addresses>
<address name="exampleQueue">
<anycast>
<queue name="exampleQueue"/>
</anycast>
</address>
<address name="DLQ">
<anycast>
<queue name="DLQ" />
</anycast>
</address>
<address name="ExpiryQueue">
<anycast>
<queue name="ExpiryQueue" />
</anycast>
</address>
</addresses>
<address-settings>
<!-- if you define auto-create on certain queues, management has to be auto-create -->
<address-setting match="activemq.management#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
<!--default for catch all-->
<address-setting match="#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
<address-setting match="exampleQueue">
<dead-letter-address>DLQ</dead-letter-address>
<redelivery-delay>1000</redelivery-delay>
<max-delivery-attempts>3</max-delivery-attempts>
<max-size-bytes>-1</max-size-bytes>
<page-size-bytes>1048576</page-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
</address-setting>
</address-settings>
</core>
</configuration>
Connectors have two functions. Defining how to connect to other AMQ brokers and the connector parameters are send to a client when it connects the first time. If a client is going to use these parameters the keystore/truststore path, name and password must exist and be the same on all clients. Best is not to set the key/truststore parameters on a connector then a proper client will use the values from org.apache.activemq.ssl.* or javax.net.ssl.*.
See https://access.redhat.com/documentation/en-us/red_hat_amq/7.4/html/configuring_amq_broker/acceptor_connector_params
I am still struggling to get this to work.
AMQ broker HA cluster works without ssl
AMQ broker works with ssl
AMQ broker HA cluster still fails.
Truststore was a JKS and PKCS12 was expected. Now it works.
I'm using Wildfly 9.1 in domain mode.
I have three hosts: A , B and C.
All nodes have a test server with the same test.ear.
After starting the three nodes, all ClusterConnectionBridge are created correctly between them.
Each node send messages on commonTopic, and an MDB logs the received messages.
The problem is that what I see logged on the node A is only the messages sent by himself. On the other nodes i see logged all the messages sent by A , B , C . This is weird, since using tcpdump on node A i see the incoming messages on port 8080 from the other nodes.
Using cli to get metrics on node A:
[domain#MASTER:19990 /] /host=SLAVE.A/server=test/subsystem=messaging/hornetq-server=default/jms-topic=commonTopic:read-resource(recursive=true,proxies=false,include-runtime=true,include-defaults=true)
{
"outcome" => "success",
"result" => {
"delivering-count" => 0,
"durable-message-count" => 0,
"durable-subscription-count" => 1,
"entries" => [
"java:/jms/commonTopic",
"java:jboss/exported/jms/commonTopic"
],
"message-count" => 0L,
"messages-added" => 150L,
"non-durable-message-count" => 0,
"non-durable-subscription-count" => 3,
"subscription-count" => 4,
"temporary" => false,
"topic-address" => "jms.topic.commonTopic"
}
}
I see that the number of added messages is 3 times the messages sent by one node. So it seems messages are received through the bridge but not consumed by the MDB.
Could someone give me an idea where to look for this issue.
The hosts are on virtual machines , on the same LAN.
The same happens if I start a node D on my PC and join it to the cluster. Node D logs all messages from the other 3 nodes, but on the other nodes no D message are logged.
The following is a snippet from my domain.xml regarding HornetQ configuration:
<subsystem xmlns="urn:jboss:domain:messaging:3.0">
<hornetq-server>
<security-enabled>false</security-enabled>
<jmx-management-enabled>true</jmx-management-enabled>
<journal-file-size>102400</journal-file-size>
<connectors>
<http-connector name="http-connector" socket-binding="http">
<param key="http-upgrade-endpoint" value="http-acceptor"/>
</http-connector>
<http-connector name="http-connector-throughput" socket-binding="http">
<param key="http-upgrade-endpoint" value="http-acceptor-throughput"/>
<param key="batch-delay" value="50"/>
</http-connector>
<netty-connector name="netty" socket-binding="messaging"/>
<netty-connector name="netty-throughput" socket-binding="messaging-throughput">
<param key="batch-delay" value="50"/>
</netty-connector>
<in-vm-connector name="in-vm" server-id="0"/>
</connectors>
<acceptors>
<http-acceptor http-listener="default" name="http-acceptor"/>
<http-acceptor http-listener="default" name="http-acceptor-throughput">
<param key="batch-delay" value="50"/>
<param key="direct-deliver" value="false"/>
</http-acceptor>
<netty-acceptor name="netty" socket-binding="messaging"/>
<netty-acceptor name="netty-throughput" socket-binding="messaging-throughput">
<param key="batch-delay" value="50"/>
<param key="direct-deliver" value="false"/>
</netty-acceptor>
<in-vm-acceptor name="in-vm" server-id="0"/>
</acceptors>
<broadcast-groups>
<broadcast-group name="bg-group">
<jgroups-stack>udp</jgroups-stack>
<jgroups-channel>hq-cluster</jgroups-channel>
<connector-ref>
http-connector
</connector-ref>
</broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="dg-group">
<jgroups-stack>udp</jgroups-stack>
<jgroups-channel>hq-cluster</jgroups-channel>
</discovery-group>
</discovery-groups>
<cluster-connections>
<cluster-connection name="my-cluster">
<address>jms</address>
<connector-ref>http-connector</connector-ref>
<use-duplicate-detection>true</use-duplicate-detection>
<forward-when-no-consumers>false</forward-when-no-consumers>
<max-hops>1</max-hops>
<discovery-group-ref discovery-group-name="dg-group"/>
</cluster-connection>
</cluster-connections>
<security-settings>
<security-setting match="#">
<permission type="send" roles="guest"/>
<permission type="consume" roles="guest"/>
<permission type="createNonDurableQueue" roles="guest"/>
<permission type="deleteNonDurableQueue" roles="guest"/>
</security-setting>
</security-settings>
<address-settings>
<address-setting match="#">
<dead-letter-address>jms.queue.DLQ</dead-letter-address>
<expiry-address>jms.queue.ExpiryQueue</expiry-address>
<max-size-bytes>10485760</max-size-bytes>
<page-size-bytes>2097152</page-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<redistribution-delay>1000</redistribution-delay>
</address-setting>
</address-settings>
<jms-connection-factories>
<connection-factory name="InVmConnectionFactory">
<connectors>
<connector-ref connector-name="in-vm"/>
</connectors>
<entries>
<entry name="java:/jms/MessageConnectionFactory"/>
<entry name="java:jboss/jms/MessageConnectionFactory"/>
<entry name="java:/ConnectionFactory"/>
</entries>
<reconnect-attempts>-1</reconnect-attempts>
</connection-factory>
<connection-factory name="RemoteConnectionFactory">
<connectors>
<connector-ref connector-name="http-connector"/>
</connectors>
<entries>
<entry name="java:jboss/exported/jms/MessageConnectionFactory"/>
<entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
</entries>
<ha>true</ha>
<block-on-acknowledge>true</block-on-acknowledge>
<reconnect-attempts>-1</reconnect-attempts>
</connection-factory>
<pooled-connection-factory name="hornetq-ra">
<transaction mode="xa"/>
<connectors>
<connector-ref connector-name="in-vm"/>
</connectors>
<entries>
<entry name="java:/JmsXA"/>
<entry name="java:jboss/DefaultJMSConnectionFactory"/>
</entries>
</pooled-connection-factory>
</jms-connection-factories>
<jms-destinations>
<jms-queue name="ExpiryQueue">
<entry name="jms/queue/ExpiryQueue"/>
<entry name="java:jboss/exported/jms/queue/ExpiryQueue"/>
<durable>true</durable>
</jms-queue>
<jms-queue name="DLQ">
<entry name="jms/queue/DLQ"/>
<entry name="java:jboss/exported/jms/queue/DLQ"/>
<durable>true</durable>
</jms-queue>
<jms-topic name="commonTopic">
<entry name="java:/jms/commonTopic"/>
<entry name="java:jboss/exported/jms/commonTopic"/>
</jms-topic>
</jms-topic>
</jms-destinations>
</hornetq-server>
</subsystem>
I found out the problem was caused by message persistence.
A 5 seconds time to live was not enough for node A to consume the message.
I found the expired messages on expiredQueue.
Each node sends one message every 10 seconds, so i don't understand the reason why the messages expire on node A.
I am trying to develop an MDB for listening to remote queue in JBoss7, I Didn't receive messages from the queue.
I got warnings in the server log:
11:32:41,882 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (hornetq-failure-check-thread) Connection failure has been detected: Did not receive data from /137.72.223.212:50835. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]
11:32:41,882 WARN [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Client connection failed, clearing up resources for session a79ee8f8-6a4e-11e4-b696-d65d20524153
11:32:41,882 WARN [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Cleared up resources for session a79ee8f8-6a4e-11e4-b696-d65d20524153
11:32:41,882 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (hornetq-failure-check-thread) Connection failure has been detected: Did not receive data from /137.72.223.212:50834. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]
11:32:41,897 WARN [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Client connection failed, clearing up resources for session a797bd07-6a4e-11e4-b696-d65d20524153
11:32:41,897 WARN [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Cleared up resources for session a797bd07-6a4e-11e4-b696-d65d20524153
11:32:41,897 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (hornetq-failure-check-thread) Connection failure has been detected: Did not receive data from /137.72.223.212:50833. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]
11:32:41,913 WARN [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Client connection failed, clearing up resources for session a7930216-6a4e-11e4-b696-d65d20524153
11:32:41,913 WARN [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Cleared up resources for session a7930216-6a4e-11e4-b696-d65d20524153
11:32:41,913 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (hornetq-failure-check-thread) Connection failure has been detected: Did not receive data from /137.72.223.212:50836. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]
My MDB configuration:
#MessageDriven(activationConfig = {
#ActivationConfigProperty( propertyName = "destinationType", propertyValue = "javax.jms.Queue"),
#ActivationConfigProperty(propertyName = "destination", propertyValue = "java:/queue/MyQueue") ,
#ActivationConfigProperty(propertyName = "connectionFactoryJndiName", propertyValue = "RemoteConnectionFactory"),
#ActivationConfigProperty(propertyName = "connectorClassName", propertyValue = "org.hornetq.core.remoting.impl.netty.NettyConnectorFactory"),
#ActivationConfigProperty(propertyName = "connectionParameters", propertyValue = "host=vm-thi-blg-qa2;port=5445"),
#ActivationConfigProperty(propertyName = "acknowledgeMode", propertyValue = "Auto-acknowledge"),
})
The JBoss standalone.xml:
<?xml version='1.0' encoding='UTF-8'?>
<server xmlns="urn:jboss:domain:1.2">
<extensions>
<extension module="org.jboss.as.clustering.infinispan"/>
<extension module="org.jboss.as.cmp"/>
<extension module="org.jboss.as.configadmin"/>
<extension module="org.jboss.as.connector"/>
<extension module="org.jboss.as.deployment-scanner"/>
<extension module="org.jboss.as.ee"/>
<extension module="org.jboss.as.ejb3"/>
<extension module="org.jboss.as.jacorb"/>
<extension module="org.jboss.as.jaxr"/>
<extension module="org.jboss.as.jaxrs"/>
<extension module="org.jboss.as.jdr"/>
<extension module="org.jboss.as.jmx"/>
<extension module="org.jboss.as.jpa"/>
<extension module="org.jboss.as.jsr77"/>
<extension module="org.jboss.as.logging"/>
<extension module="org.jboss.as.mail"/>
<extension module="org.jboss.as.messaging"/>
<extension module="org.jboss.as.naming"/>
<extension module="org.jboss.as.osgi"/>
<extension module="org.jboss.as.pojo"/>
<extension module="org.jboss.as.remoting"/>
<extension module="org.jboss.as.sar"/>
<extension module="org.jboss.as.security"/>
<extension module="org.jboss.as.threads"/>
<extension module="org.jboss.as.transactions"/>
<extension module="org.jboss.as.web"/>
<extension module="org.jboss.as.webservices"/>
<extension module="org.jboss.as.weld"/>
</extensions>
<management>
<security-realms>
<security-realm name="ManagementRealm">
<authentication>
<properties path="mgmt-users.properties" relative-to="jboss.server.config.dir"/>
</authentication>
</security-realm>
<security-realm name="ApplicationRealm">
<authentication>
<properties path="application-users.properties" relative-to="jboss.server.config.dir"/>
</authentication>
</security-realm>
</security-realms>
<management-interfaces>
<native-interface security-realm="ManagementRealm">
<socket-binding native="management-native"/>
</native-interface>
<http-interface security-realm="ManagementRealm">
<socket-binding http="management-http"/>
</http-interface>
</management-interfaces>
</management>
<profile>
<subsystem xmlns="urn:jboss:domain:logging:1.1">
<console-handler name="CONSOLE">
<level name="INFO"/>
<formatter>
<pattern-formatter pattern="%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n"/>
</formatter>
</console-handler>
<periodic-rotating-file-handler name="FILE">
<formatter>
<pattern-formatter pattern="%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n"/>
</formatter>
<file relative-to="jboss.server.log.dir" path="server.log"/>
<suffix value=".yyyy-MM-dd"/>
<append value="true"/>
</periodic-rotating-file-handler>
<logger category="com.arjuna">
<level name="WARN"/>
</logger>
<logger category="org.apache.tomcat.util.modeler">
<level name="WARN"/>
</logger>
<logger category="sun.rmi">
<level name="WARN"/>
</logger>
<logger category="jacorb">
<level name="WARN"/>
</logger>
<logger category="jacorb.config">
<level name="ERROR"/>
</logger>
<root-logger>
<level name="INFO"/>
<handlers>
<handler name="CONSOLE"/>
<handler name="FILE"/>
</handlers>
</root-logger>
</subsystem>
<subsystem xmlns="urn:jboss:domain:cmp:1.0"/>
<subsystem xmlns="urn:jboss:domain:configadmin:1.0"/>
<subsystem xmlns="urn:jboss:domain:datasources:1.0">
<datasources>
<datasource jndi-name="java:jboss/datasources/ExampleDS" pool-name="ExampleDS" enabled="true" use-java-context="true">
<connection-url>jdbc:h2:mem:test;DB_CLOSE_DELAY=-1</connection-url>
<driver>h2</driver>
<security>
<user-name>jala1</user-name>
<password>jala</password>
</security>
</datasource>
<drivers>
<driver name="h2" module="com.h2database.h2">
<xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>
</driver>
</drivers>
</datasources>
</subsystem>
<subsystem xmlns="urn:jboss:domain:deployment-scanner:1.1">
<deployment-scanner path="deployments" relative-to="jboss.server.base.dir" scan-interval="5000"/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:ee:1.0"/>
<subsystem xmlns="urn:jboss:domain:ejb3:1.2">
<session-bean>
<stateless>
<bean-instance-pool-ref pool-name="slsb-strict-max-pool"/>
</stateless>
<stateful default-access-timeout="5000" cache-ref="simple"/>
<singleton default-access-timeout="5000"/>
</session-bean>
<mdb>
<resource-adapter-ref resource-adapter-name="hornetq-ra"/>
<bean-instance-pool-ref pool-name="mdb-strict-max-pool"/>
</mdb>
<pools>
<bean-instance-pools>
<strict-max-pool name="slsb-strict-max-pool" max-pool-size="20" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
<strict-max-pool name="mdb-strict-max-pool" max-pool-size="20" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
</bean-instance-pools>
</pools>
<caches>
<cache name="simple" aliases="NoPassivationCache"/>
<cache name="passivating" passivation-store-ref="file" aliases="SimpleStatefulCache"/>
</caches>
<passivation-stores>
<file-passivation-store name="file"/>
</passivation-stores>
<async thread-pool-name="default"/>
<timer-service thread-pool-name="default">
<data-store path="timer-service-data" relative-to="jboss.server.data.dir"/>
</timer-service>
<remote connector-ref="remoting-connector" thread-pool-name="default"/>
<thread-pools>
<thread-pool name="default">
<max-threads count="10"/>
<keepalive-time time="100" unit="milliseconds"/>
</thread-pool>
</thread-pools>
<iiop enable-by-default="false" use-qualified-name="false"/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:infinispan:1.2" default-cache-container="hibernate">
<cache-container name="hibernate" default-cache="local-query">
<local-cache name="entity">
<transaction mode="NON_XA"/>
<eviction strategy="LRU" max-entries="10000"/>
<expiration max-idle="100000"/>
</local-cache>
<local-cache name="local-query">
<transaction mode="NONE"/>
<eviction strategy="LRU" max-entries="10000"/>
<expiration max-idle="100000"/>
</local-cache>
<local-cache name="timestamps">
<transaction mode="NONE"/>
<eviction strategy="NONE"/>
</local-cache>
</cache-container>
</subsystem>
<subsystem xmlns="urn:jboss:domain:jacorb:1.1">
<orb>
<initializers security="on" transactions="spec"/>
</orb>
</subsystem>
<subsystem xmlns="urn:jboss:domain:jaxr:1.1">
<connection-factory jndi-name="java:jboss/jaxr/ConnectionFactory"/>
<properties/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:jaxrs:1.0"/>
<subsystem xmlns="urn:jboss:domain:jca:1.1">
<archive-validation enabled="true" fail-on-error="true" fail-on-warn="false"/>
<bean-validation enabled="true"/>
<default-workmanager>
<short-running-threads>
<core-threads count="50"/>
<queue-length count="50"/>
<max-threads count="50"/>
<keepalive-time time="10" unit="seconds"/>
</short-running-threads>
<long-running-threads>
<core-threads count="50"/>
<queue-length count="50"/>
<max-threads count="50"/>
<keepalive-time time="10" unit="seconds"/>
</long-running-threads>
</default-workmanager>
<cached-connection-manager/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:jdr:1.0"/>
<subsystem xmlns="urn:jboss:domain:jmx:1.1">
<show-model value="true"/>
<remoting-connector/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:jpa:1.0">
<jpa default-datasource=""/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:jsr77:1.0"/>
<subsystem xmlns="urn:jboss:domain:mail:1.0">
<mail-session jndi-name="java:jboss/mail/Default">
<smtp-server outbound-socket-binding-ref="mail-smtp"/>
</mail-session>
</subsystem>
<subsystem xmlns="urn:jboss:domain:messaging:1.1">
<hornetq-server>
<persistence-enabled>true</persistence-enabled>
<journal-file-size>102400</journal-file-size>
<journal-min-files>2</journal-min-files>
<connectors>
<netty-connector name="netty" socket-binding="messaging"/>
<netty-connector name="netty-throughput" socket-binding="messaging-throughput">
<param key="batch-delay" value="50"/>
</netty-connector>
<in-vm-connector name="in-vm" server-id="0"/>
</connectors>
<acceptors>
<netty-acceptor name="netty" socket-binding="messaging"/>
<netty-acceptor name="netty-throughput" socket-binding="messaging-throughput">
<param key="batch-delay" value="50"/>
<param key="direct-deliver" value="false"/>
</netty-acceptor>
<in-vm-acceptor name="in-vm" server-id="0"/>
</acceptors>
<security-settings>
<security-setting match="#">
<permission type="send" roles="guest"/>
<permission type="consume" roles="guest"/>
<permission type="createNonDurableQueue" roles="guest"/>
<permission type="deleteNonDurableQueue" roles="guest"/>
</security-setting>
</security-settings>
<address-settings>
<address-setting match="#">
<dead-letter-address>jms.queue.DLQ</dead-letter-address>
<expiry-address>jms.queue.ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<max-size-bytes>10485760</max-size-bytes>
<address-full-policy>BLOCK</address-full-policy>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
</address-setting>
</address-settings>
<jms-connection-factories>
<connection-factory name="InVmConnectionFactory">
<connectors>
<connector-ref connector-name="in-vm"/>
</connectors>
<entries>
<entry name="java:/ConnectionFactory"/>
</entries>
</connection-factory>
<connection-factory name="RemoteConnectionFactory">
<connectors>
<connector-ref connector-name="netty"/>
</connectors>
<entries>
<entry name="RemoteConnectionFactory"/>
<entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
</entries>
</connection-factory>
<pooled-connection-factory name="hornetq-ra">
<transaction mode="xa"/>
<connectors>
<connector-ref connector-name="in-vm"/>
</connectors>
<entries>
<entry name="java:/JmsXA"/>
</entries>
</pooled-connection-factory>
</jms-connection-factories>
<jms-destinations>
<jms-queue name="testQueue">
<entry name="queue/test"/>
<entry name="java:jboss/exported/jms/queue/test"/>
</jms-queue>
<jms-queue name="MyQueue">
<entry name="java:/jms/MyQueue"/>
<durable>false</durable>
</jms-queue>
<jms-queue name="q">
<entry name="java:/queue/MyQueue"/>
<durable>true</durable>
</jms-queue>
<jms-topic name="testTopic">
<entry name="topic/test"/>
<entry name="java:jboss/exported/jms/topic/test"/>
</jms-topic>
</jms-destinations>
</hornetq-server>
</subsystem>
<subsystem xmlns="urn:jboss:domain:naming:1.1"/>
<subsystem xmlns="urn:jboss:domain:osgi:1.2" activation="lazy">
<properties>
<property name="org.osgi.framework.startlevel.beginning">
1
</property>
</properties>
<capabilities>
<capability name="javax.servlet.api:v25"/>
<capability name="javax.transaction.api"/>
<capability name="org.apache.felix.log" startlevel="1"/>
<capability name="org.jboss.osgi.logging" startlevel="1"/>
<capability name="org.apache.felix.configadmin" startlevel="1"/>
<capability name="org.jboss.as.osgi.configadmin" startlevel="1"/>
</capabilities>
</subsystem>
<subsystem xmlns="urn:jboss:domain:pojo:1.0"/>
<subsystem xmlns="urn:jboss:domain:remoting:1.1">
<connector name="remoting-connector" socket-binding="remoting" security-realm="ApplicationRealm"/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:resource-adapters:1.0"/>
<subsystem xmlns="urn:jboss:domain:sar:1.0"/>
<subsystem xmlns="urn:jboss:domain:security:1.1">
<security-domains>
<security-domain name="other" cache-type="default">
<authentication>
<login-module code="Remoting" flag="optional">
<module-option name="password-stacking" value="useFirstPass"/>
</login-module>
<login-module code="RealmUsersRoles" flag="required">
<module-option name="usersProperties" value="${jboss.server.config.dir}/application-users.properties"/>
<module-option name="rolesProperties" value="${jboss.server.config.dir}/application-roles.properties"/>
<module-option name="realm" value="ApplicationRealm"/>
<module-option name="password-stacking" value="useFirstPass"/>
<module-option name="unauthenticatedIdentity" value="guest"/>
</login-module>
</authentication>
</security-domain>
<security-domain name="jboss-web-policy" cache-type="default">
<authorization>
<policy-module code="Delegating" flag="required"/>
</authorization>
</security-domain>
<security-domain name="jboss-ejb-policy" cache-type="default">
<authorization>
<policy-module code="Delegating" flag="required"/>
</authorization>
</security-domain>
</security-domains>
</subsystem>
<subsystem xmlns="urn:jboss:domain:threads:1.1"/>
<subsystem xmlns="urn:jboss:domain:transactions:1.1">
<core-environment>
<process-id>
<uuid/>
</process-id>
</core-environment>
<recovery-environment socket-binding="txn-recovery-environment" status-socket-binding="txn-status-manager"/>
<coordinator-environment default-timeout="300"/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" native="false">
<connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>
<virtual-server name="default-host" enable-welcome-root="true">
<alias name="localhost"/>
<alias name="example.com"/>
</virtual-server>
</subsystem>
<subsystem xmlns="urn:jboss:domain:webservices:1.1">
<modify-wsdl-address>true</modify-wsdl-address>
<wsdl-host>${jboss.bind.address:127.0.0.1}</wsdl-host>
<endpoint-config name="Standard-Endpoint-Config"/>
<endpoint-config name="Recording-Endpoint-Config">
<pre-handler-chain name="recording-handlers" protocol-bindings="##SOAP11_HTTP ##SOAP11_HTTP_MTOM ##SOAP12_HTTP ##SOAP12_HTTP_MTOM">
<handler name="RecordingHandler" class="org.jboss.ws.common.invocation.RecordingServerHandler"/>
</pre-handler-chain>
</endpoint-config>
</subsystem>
<subsystem xmlns="urn:jboss:domain:weld:1.0"/>
</profile>
<interfaces>
<interface name="management">
<inet-address value="${jboss.bind.address.management:0.0.0.0}"/>
</interface>
<interface name="public">
<inet-address value="${jboss.bind.address:127.0.0.1}"/>
</interface>
<interface name="unsecure">
<inet-address value="${jboss.bind.address.unsecure:127.0.0.1}"/>
</interface>
</interfaces>
<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
<socket-binding name="management-native" interface="management" port="${jboss.management.native.port:9999}"/>
<socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>
<socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9443}"/>
<socket-binding name="ajp" port="8009"/>
<socket-binding name="http" port="8080"/>
<socket-binding name="https" port="8443"/>
<socket-binding name="jacorb" interface="unsecure" port="3528"/>
<socket-binding name="jacorb-ssl" interface="unsecure" port="3529"/>
<socket-binding name="messaging" port="5445"/>
<socket-binding name="messaging-throughput" port="5455"/>
<socket-binding name="osgi-http" interface="management" port="8090"/>
<socket-binding name="remoting" port="4447"/>
<socket-binding name="txn-recovery-environment" port="4712"/>
<socket-binding name="txn-status-manager" port="4713"/>
<outbound-socket-binding name="mail-smtp">
<remote-destination host="localhost" port="25"/>
</outbound-socket-binding>
</socket-binding-group>
<deployments>
<deployment name="loadSimulator.war" runtime-name="loadSimulator.war">
<content sha1="ad5c4080accb0e644bcb0bf2842f8f230fbe8e75"/>
</deployment>
<deployment name="Hops.war" runtime-name="Hops.war" enabled="false">
<content sha1="ba2746c8e115729b37343704b2540d258f0ec614"/>
</deployment>
<deployment name="Agent_Footprint.war" runtime-name="Agent_Footprint.war">
<content sha1="9d9893741e9f24d609c3d39eb9b1bd5ec08d7556"/>
</deployment>
</deployments>
</server>
If the server doesn't receive the ping in a certain time then it closes the connection, in this instance it is an invm connection from the MDB.
The usual causes for this are the server or client becoming unresponsive because of overload, memory issues etc. For that, you must to add the following elements to connection-factory in standalone.xml.
<connection-ttl>-1</connection-ttl>
<client-failure-check-period>-1</client-failure-check-period>
A value of -1 for ConnectionTTL means the server will never time out the connection on the server side.