Spring JMSTemplate - lost messages - jms

We are using Spring JMSTemplate 3.0.1RELEASE to send JMS messages to an ActiveMQ cluster.
I am using useAsynSend=true to be able to send Asynchronous requests as these are all Fire and Forget. However they are still persistent and I can confirm they do get persisted on the AMQ Kaha-DB first and then forwarded on to Message Listeners. No CorelationID or JMSReplyTo as I am not listening back on any response.
<amq:connectionFactory id="amqJmsFactory"
brokerURL="failover:(tcp://MY-SERVER-01:61616,tcp://MY-SERVER-02:61616)?randomize=true&jms.useAsyncSend=true" />
<!-- JMS Producer Configuration -->
<bean id="jmsProducerTemplate" class="org.springframework.jms.core.JmsTemplate"
p:connectionFactory-ref="amqJmsFactory" />
<bean id="activeMQBinding" class="com.my.company.activemq.ActiveMQProductBinding">
<property name="template" ref="jmsProducerTemplate" />
</bean>
In the ActiveMQProductBinding class we have the following method
public void sendActiveMQMessageAsync(final Object message, String qName) {
log.info("About to send message");
template.send(qName, new MessageCreator() {
public Message createMessage(Session session) throws JMSException {
ObjectMessage objectMessage = session.createObjectMessage((Serializable) message);
log.info("Sending message: " + objectMessage);
return objectMessage;
}
});
}
Now I can see in the logs that the logs are getting printed. No Exception is thrown. Still some messages are totally getting lost. It probably doesn't reach ActiveMQ. I have Mule JMS Listeners configured to listen to messages on the ActiveMQ queues defined by 'qName' above. Most of the messages get delivered to AMQ fine and Mule picks them up within fraction of a second. However it is only some messages which are getting lost. When I checked the Queues on ActiveMQ I can see all messages received have been dispatched fine to Mule. Hence I am suspecting the messages aren't reaching ActiveMQ at all or AMQ is rejecting. However no logs on JMSTemplate spring or ActiveMQ.
The Message created looks like this (It is printed in the method above).
2013-03-11 16:33:11,752 INFO [com.my.company.activemq.ActiveMQProductBinding$1] Sending message: ActiveMQObjectMessage {commandId = 0, responseRequired = false, messageId = null, originalDestination = null, originalTransactionId = null, producerId = null, destination = null, transactionId = null, expiration = 0, timestamp = 0, arrival = 0, brokerInTime = 0, brokerOutTime = 0, correlationId = null, replyTo = null, persistent = false, type = null, priority = 0, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.ByteSequence#61408493, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties = false, readOnlyBody = false, droppable = false}
Anything I am missing on JMSTemplate config or some behaviour on AMQ? Please help!
Any help guys?

2 Possible reasons I can think of.
1) We are using the wrong connection factory. We should use Pooled Connection Factory with JMSTemplate to send messages.
org.apache.activemq.pool.PooledConnectionFactory
2) We are using useAsyncSend=true - i.e. the sender sends the message and does not wait for acknowledgement and there are chances of message losses. This seems more probable but not sure.
Still not accepting this as Answer as someone may have a more concrete explanation. In the meantime if anyone stumbles upon this question may be benefitted by this cues.

Related

Tuning Kafka for latency, packet loss, and unreachability

I am trying to optimize the performances of Kafka in a scenario where there is high latency (>500ms) and intermittent packet loss. I am working with JAVA and using 'kafka_2.13', version: '2.5.0' API. I have 24
nodes connected to a single broker, each node tries to send a small message to all the other subscribers. I observe that all nodes are able to communicate when there is no packet loss nor latency but they don't seem to be able to communicate soon after I add latency and packet loss. I will do more tests on Monday but I was wondering if anyone had any suggestions on possible configuration improvements.
Following you can see the code that I see to publish and receive messages and then the different configurations that used for consumers and producers.
Publishers:
boolean sendAsyncMessage (byte[] value, String topic) {
ProducerRecord<Long, byte[]> record = new ProducerRecord<> (topic, System.currentTimeMillis (), value);
long msStart = System.currentTimeMillis ();
producer.send (record, (metadata, exception) -> {
long msDelta = System.currentTimeMillis () - msStart;
logger.info ("Message with topic {} sent at {}, was ack after {}", topic, msStart, msDelta);
if (metadata == null) {
logger.info ("An exception was triggered during send:" + exception.toString ());
}
});
producer.flush ();
return true;
}
Subscribers:
while (keepGoing.get ()) {
try {
// java example do it every time!
subscribe ();
ConsumerRecords<Long, byte[]> consumerRecords = consumer.poll (Duration.ofMillis (2000));
manageMessage (consumerRecords);
//Thread processRecords = new Thread (() -> manageMessage (consumerRecords));
//processRecords.start ();
} catch (Exception e) {
logger.error ("Problem in polling: " + e.toString ());
}
}
Producer:
properties.put (ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, KafkaBroker.KEY_SERIALIZER);
properties.put (ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaBroker.VALUE_SERIALIZER);
properties.put (ProducerConfig.ACKS_CONFIG, reliable ? "1" : 0);
// host1:port1,host2:port2
properties.put (ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, server);
// how many bytes to buffer records waiting to be sent to the server
//properties.put (ProducerConfig.BUFFER_MEMORY_CONFIG, 33554432);
properties.put (ProducerConfig.COMPRESSION_TYPE_CONFIG, "gzip");
properties.put (ProducerConfig.CLIENT_ID_CONFIG, clientID);
//15 MINUTES
properties.put (ProducerConfig.CONNECTIONS_MAX_IDLE_MS_CONFIG, 54000000);
// MAX UNCOMPRESSED MESSAGE SIZE
// properties.put (ProducerConfig.MAX_REQUEST_SIZE_CONFIG, 1048576);
// properties.put (ProducerConfig.RECONNECT_BACKOFF_MAX_MS_CONFIG, 1000);
properties.put (ProducerConfig.RECONNECT_BACKOFF_MS_CONFIG, 300);
properties.put (ProducerConfig.RETRY_BACKOFF_MS_CONFIG, 300);
Consumer
properties.put (ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, KafkaBroker.KEY_DESERIALIZER);
properties.put (ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, KafkaBroker.VALUE_DESERIALIZER);
// host1:port1,host2:port2
properties.put (ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, server);
// should be the topic
properties.put (ConsumerConfig.GROUP_ID_CONFIG, groupID);
properties.put (ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, "3000");
properties.put (ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "30000");
properties.put (ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "1000");
Before trying to change all settings, I'd make a few changes in your logic:
Producer
Currently you are calling flush() after sending each message, effectively doing a synchronous send. This is not recommended as it forces the Kafka client to make a request to the cluster for every single message. This is pretty inefficient. In most cases, it's best to let the client decide when to actually send messages and not use flush().
Consumer
In each iteration, you are calling subscribe(), this is not needed. You should only call subscribe() when you want to change the subscription. Also creating a new thread in each poll() loop is not recommended! In addition to being slow, you risk creating hundreds or thousands of threads if the consumer starts fetching large amounts of messages.
Kafka is using a TCP protocol, so packets lost should be automatically retried. By default, Kafka clients are configured to retry most operations and automatically reconnect to brokers if a connection is lost.
When doing your tests, before changing configurations, you should see how the Kafka client is behaving by monitoring its metrics and logs. Are timeouts reached because of the latency? Are messages retried?
In the end, the biggest factor that was preventing my distributed system to correctly communicate was the producer acks option. In the beginning, we had set this to all (the strictest option) and it seems that that, paired with the deteriorated network was preventing Kafka to have performances similar to other TCP based protocols. We now use 0 for unreliable messages and 1 for reliable.

ActiveMQ - After a long time running, broker accepts messages but doesn't enqueue anything

My configuration is:
ActiveMQ 5.15.1
atomikos transactions jms 4.0.6
Spring boot 2.1.7.RELEASE
My application is sending messages by Spring JmsTemplate.convertAndSend method :
import org.springframework.jms.core.JmsTemplate;
#Component
#Slf4j
public class AsynchronousMajProducer {
#Autowired
private ActiveMQConfig activeMQConfig;
#Autowired
private JmsTemplate jmsTemplate;
private void convertAndSend(String msg) {
jmsTemplate.convertAndSend(activeMQConfig.privateQueue(), msg, message -> {
String[] versionParts = buildProperties.getVersion().split("\\.");
String version = versionParts[0] + "." + versionParts[1];
message.setStringProperty("VERSION", version);
message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, 2000);
return message;
});
}
}
After a long time running (multiple weeks), some queues doesn't enqueue messages anymore (and some works well).
For those failing, the convertAndSend method doesn't throw any exceptions, all seems to be OK, but finally, nothing is enqueued ... (enqueue count on AMQ console never increases).
Below is an AMQ log on a successful sending :
2019-11-22 09:37:29,647 | INFO | Sending message: ActiveMQTextMessage {commandId = 8, responseRequired = false,
messageId = ID:NMI16510-60675-1574375565048-1:23:2:1:1, originalDestination = null, originalTransactionId = null,
producerId = ID:NMI16510-60675-1574375565048-1:23:2:1, destination = queue://testportail,
transactionId = XID:[1096044365,globalId=3137322e31372e36352e37372e746d313537343337353834383839343030343731,
branchId=3137322e31372e36352e37372e746d343732], expiration = 1574375885026, timestamp = 1574375849026,
arrival = 0, brokerInTime = 0, brokerOutTime = 0, correlationId = null, replyTo = null, persistent =
true, type = null, priority = 9, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false,
userID = null, content = org.apache.activemq.util.ByteSequence#15d17509, marshalledProperties = org.apache.activemq.util.ByteSequence#34beb533,
dataStructure = null, redeliveryCounter = 0, size = 1122, properties = {VERSION=2.0, AMQ_SCHEDULED_DELAY=2000},
readOnlyProperties = false, readOnlyBody = false, droppable = false, jmsXGroupFirstForConsumer = false,
text = {
"type" : "MAJ_CONTACT",
"id" : 124
}} | org.apache.activemq.broker.util.LoggingBrokerPlugin | ActiveMQ Transport: tcp:///172.17.65.77:61220#61611
2019-11-22 09:37:32,001 | DEBUG | Firing: Job [id=ID:NMI16510-60675-1574375565048-1:23:2:1:1, startTime=Fri Nov 22 09:37:30 NCT 2019,
delay=2000, period=0, repeat=0, nextTime=Fri Nov 22 09:37:32 NCT 2019, executionCount = 1] | org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl
| JobScheduler:JMS
2019-11-22 09:37:32,002 | DEBUG | Set message ID:NMI16510-60675-1574375565048-1:23:2:1:1 timestamp from 1574375849026 to 1574375852002
| org.apache.activemq.broker.scheduler.SchedulerBroker | JobScheduler:JMS
2019-11-22 09:37:32,003 | DEBUG | ACTIVEMQ Message ID:NMI16510-60675-1574375565048-1:23:2:1:1 sent to queue://testportail
| org.apache.activemq.broker.region.Queue | JobScheduler:JMS
When it fails, I only get the first log Sending message.
No ERROR or WARN messages on AMQ log.
I noticed some queues are affected and some not.
To 'fix' the problem, I have to :
Stop the broker
Delete this folder : /apache-activemq-5.15.1/data/ACTIVEMQ/scheduler
Restart the broker
In production, we do it every day to anticipate this problem.
It's not a good solution, because messages previously send by the application but not enqueued are definitly lost.
Is it a known issue of AMQ scheduler ?
Is it a bad/good practice do restart AMQ frequently ?
I've just found a relative issue but for Topic here :
[http://activemq.2283324.n4.nabble.com/ActiveMQ-Topic-Messages-Not-Delivered-td4703502.html]
It has been resolved by touching to storeUsage and tempUsage in activemq.xml :
Original
<storeUsage>
<storeUsage limit="100 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb"/>
</tempUsage>
Modification
<storeUsage>
<storeUsage limit="20 mb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="20 mb"/>
</tempUsage>
The problem was a storage issue for kahadb wich was in a smaller storage than 100Go.
Thanks to swone!

google-cloud pubsub leaves messages in queue after sending ack

I have this subscriber code:
try {
//subscriber
syncSubscriber.createSubscriber(SdkServiceConfig.s.SUBSCRIPTION_NAME_PARTNER_REQUEST);
final List<ReceivedMessage> messages = syncSubscriber.fetch(10, true);//get all current messages.
List<String> ackIds = new ArrayList<>();
for (ReceivedMessage message : messages) {
requestToCofmanSender.receiveMessage(message.getMessage());
ackIds.add(message.getAckId());
}
//preferred bulk ack, due to network performance
syncSubscriber.sendAck(ackIds);
requestToCofmanSender.getWazePublisher().shutdown();
}
and
public void sendAck(Collection<String> ackIdList) {
if (ackIdList != null && ackIdList.size() != 0) {
String subscriptionName = SubscriptionName.format(this.getProjectId(), this.subscriptionId);
AcknowledgeRequest acknowledgeRequest = AcknowledgeRequest.newBuilder().setSubscription(subscriptionName).addAllAckIds(ackIdList).build();
this.subscriber.acknowledgeCallable().call(acknowledgeRequest);
}
}
I poll the pubsub queue in loop
and even though the code sends ack i still get the same messages.
how should i ack otherwise?
My problem was that i had a break point between receiving the message and sending ack. My pubsub was configured to 10 seconds timeout.

Deleting all messages in JMS queue in Widlfy 9

I am trying to remove messages from JMS queue on Wildfly 9.0.2 (JBoss) using JMX, see following code:
MBeanServer server = ManagementFactory.getPlatformMBeanServer();
try {
String name = "jboss.as.expr:subsystem=messaging,hornetq-server=default,jms-queue=MyQueue";
ObjectName objectName = new ObjectName(objectNameString);
String result = (String) server.invoke(objectName, "removeMessages", new Object[]{null},
new String[]{"java.lang.String"});
return result;
} catch (MalformedObjectNameException | InstanceNotFoundException | MBeanException | ReflectionException ignored) {
log.errorv(ignored, "Error removing messages from JMS queue [{0}]", name);
return null;
}
There is an active consumer on that queue. The code runs without exception and returns string "0", but no messages are actually removed. I tried also to use some value as message filter (*), but got failure:
javax.management.ReflectionException: "HQ119020: Invalid filter: *"
Any idea how to remove the messages?

RabbitMQ Shovel basic example

I am working on a basic example and am not able to work it out.
I need to forward messages from one machine (Machine1) to another (Machine2) via a queue (TestQ).
Producer is running on the Machine1 and a consumer on the Machine2.
My settings in the Machine1's rabbit broker config:
{rabbitmq_shovel, [ {shovels, [
{shovel_test, [
{sources, [{broker, "amqp://" }]},
{destinations, [{broker, "amqp://Machine2" }]},
{queue, <<"TestQ">>},
{ack_mode, on_confirm},
{reconnect_delay, 5}
]}
]} ]}
Machine2 has a default config and no shovel plugin enabled.
Producer's code running on the Machine1:
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
channel.queueDeclare("TestQ", true, false, false, null);
channel.basicPublish("", "TestQ", null, "Hello World!".getBytes());
Consumer's code running on the Machine2:
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
channel.queueDeclare("TestQ", true, false, false, null);
QueueingConsumer consumer = new QueueingConsumer(channel);
channel.basicConsume("TestQ", true, consumer);
while (true) {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
String message = new String(delivery.getBody());
System.out.println(" [x] Received '" + message + "'");
}
Executing rabbitmqctl eval 'rabbit_shovel_status:status().' on the Machine1:
[{shovel_test,starting,{{2014,1,7},{9,47,38}}}]
...done.
Producer sends ok, but I never get a receive from the consumer on the Machine2.
Where is a problem? Something is missing in the conf of Machine1's broker, or Machine2's broker?
Thank you!
The status of your shovel should be running, not starting. If it stays in the starting phase, that means it can't start up correctly (e.g. can't connect to the destination broker).
One problem I spotted is that you used broker instead of brokers for specifying the list of sources. Try this:
{rabbitmq_shovel,
[{shovels, [{shovel_test,
[{sources, [{brokers, ["amqp://"]}]},
{destinations, [{broker, "amqp://Machine2"}]},
{queue, <<"TestQ">>},
{ack_mode, on_confirm},
{reconnect_delay, 5}
]}
]}
]}.

Resources