I am experimenting with assured delivery in MQ Light.
I am using Node-RED with a modified mqlight input node. I have added the following options the the subscribe() call:
qos: mqlight.QOS_AT_LEAST_ONCE, autoConfirm: false, ttl: (60 * 60 * 24 * 1000)
This requires that I call delivery.message.confirmDelivery() to acknowledge to MQ Light the receipt of the message.
The screenshot below is when the subscription from mqlight_NodeREDClient is set up with autoConfirm false, a message was received, BUT no delivery.message.confirmDelivery() was called. This was to simulate some sort of error occuring in the Node-RED flow.
I have since modified the Node-RED flow to do a confirmDelivery(), and any messages consumed by the flow now are acknowledged OK, even if Node-RED isn't running at the time of publish. The message is held by MQ Light since there is a TTL on the destination and arrives as soon as i start up Node-RED again.
However, the message in this screenshot, which has been sent once already but never confirmed, is never resent. A restart of Node-RED does not change this, the message is still pending. What are the criteria that needs to be fulfilled in order for MQ Light to retransmit a message already sent before but never confirmed by a client?
If you hadn't restarted NodeRED I'd say that it was because MQ Light won't redeliver to a connected client because it thinks the client is still processing the message. However since you have it must be something else.
I've just tried the same basic setup (without NodeRED) and the behaviour is as you'd expect - when you reconnect the receiving client MQ Light redelivers the message and the MQ Light UI ticks it off.
The remaining things I can think of are:
Is it possible when that particular message was sent you had a QoS 0
subscription?
What TTL are you setting on the message at the sender?
What destination TTL have you set on your subscribe call?
If 2. is too low, the message will have been expired from the destination regardless of the QoS of the subscriber or the value of 3.
If 3. is too low and NodeRED was stopped for long enough, the entire destination will have been expired.
Related
I'm new to NSQ and was wondering whether it is possible to acknowledge that a message was consumed (just like in Google PubSub). I'm using Golang, and while going through the NSQ documentation (https://github.com/nsqio/nsq/blob/v1.2.1/nsqd/channel.go#L350) I've found the method FinishMessage which according to its comment, it's supposed to successfully discards an in-flight message. Would that be considered the same as acknowledged? If a message is not acknowledged in NSQ, how is it ensured it's gone once consumed so it won't be consumed over again?
Thanks
NSQ automatically removes a message from the queue after sending it to a consumer. And automatically re-queues the message if there is no response from the consumer after a set timeout.
And yes, depending on which programming language client you are using you have to send 'finish'.
Here are the docs for Golang
https://pkg.go.dev/github.com/nsqio/go-nsq#Finish
But because of the way NSQ is designed you have to re-queue a message from your app if the intended action was not successful.
Here is an excerpt from their website
NSQ guarantees that a message will be delivered at least once, though duplicate messages are possible. Consumers should expect this and de-dupe or perform idempotent operations.
This guarantee is enforced as part of the protocol and works as follows (assume the client has successfully connected and subscribed to a topic):
client indicates they are ready to receive messages
NSQ sends a message and temporarily stores the data locally (in the event of re-queue or timeout)
client replies FIN (finish) or REQ (re-queue) indicating success or failure respectively. If client does not reply NSQ will timeout after
a configurable duration and automatically re-queue the message)
You can read more here: https://nsq.io/overview/design.html
Hope my answer helps you.
I have an application using jms that sends data to an ActiveMQ Artemis queue. I got an exception with this message:
The transaction was rolled back on failover however commit may have been successful
This exception is basically telling me that the message may or may not have reached the queue so I don't know if I need to send the message again. Whats the best way to handle an exception like this when:
I cannot send duplicate messages to applications on the other end of the queue.
and
I cannot skip a message.
I can't state it better than the ActiveMQ Artemis documentation:
When sending messages from a client to a server, or indeed from a server to another server, if the target server or connection fails sometime after sending the message, but before the sender receives a response that the send (or commit) was processed successfully then the sender cannot know for sure if the message was sent successfully to the address.
If the target server or connection failed after the send was received and processed but before the response was sent back then the message will have been sent to the address successfully, but if the target server or connection failed before the send was received and finished processing then it will not have been sent to the address successfully. From the senders point of view it's not possible to distinguish these two cases.
When the server recovers this leaves the client in a difficult situation. It knows the target server failed, but it does not know if the last message reached its destination ok. If it decides to resend the last message, then that could result in a duplicate message being sent to the address. If each message was an order or a trade then this could result in the order being fulfilled twice or the trade being double booked. This is clearly not a desirable situation.
Sending the message(s) in a transaction does not help out either. If the server or connection fails while the transaction commit is being processed it is also indeterminate whether the transaction was successfully committed or not!
To solve these issues Apache ActiveMQ Artemis provides automatic duplicate messages detection for messages sent to addresses.
See more details about how to configure and use duplicate detection in the ActiveMQ Artemis documentation.
I'm evaluating NATS for migrating an existing msg based software
I did not find documentation about msg timeout exception and overload.
For Example:
After Subscriber has been chosen , Is it aware of timeout settings posted by Publisher ? Is it possible to notify an additional time extension ?
If the elected subscriber is aware that some DBMS connection is missing and cannot complete It could be possible to bounce the message
NATS server will pickup another subscriber and will re-post the same message ?
Ciao
Diego
For your first question: It seems to me that you are trying to publish a request message with a timeout (using the nc.Request). If so, the timeout is managed by the client. Effectively the client publishes the request message and creates a subscription on the reply subject. If the subscription doesn't get any messages within the timeout it will notify you of the timeout condition and unsubscribe from the reply subject.
On your second question - are you using a queue group? A queue group in NATS is a subscription that specifies a queue group name. All subscriptions having the same queue group name are treated specially by the server. The server will select one of the queue group subscriptions to send the message to rotating between them as messages arrive. However the responsibility of the server is simply to deliver the message.
To do what you describe, implement your functionality using request/reply using a timeout and a max number of messages equal to 1. If no responses are received after the timeout your client can then resend the request message after some delay or perform some other type of recovery logic. The reply message should be your 'protocol' to know that the message was handled properly. Note that this gets into the design of your messaging architecture. For example, it is possible for the timeout to trigger after the request recipient received the message and handled it but before the client or server was able to publish the response. In that case the request sender wouldn't be able to tell the difference and would eventually republish. This hints that such type of interactions need to make the requests idempotent to prevent duplicate side effects.
Does anyone know what the Watson IoT broker does if publishes a QoS1 o 2 message and doesn't receive the appropriate acknowledgement from the client? Does it implement a time out (say 20 seconds or so) and then resend the message again? It seems that some brokers do this while others only resend the message on a new connection (if retain is set to 1 of course). The MQTT spec is a little vague on this point.
the message would be considered in-flight for the client that hasn’t acknowledged it, and that message redelivery will only occur when that client disconnects and reconnects (but only if the client was clean session = 0)
For QoS1 and 2:
At least once (QoS1)
With quality of service level 1 (QoS1), the message is always delivered at least once. If a failure occurs before an acknowledgment is received by the sender, a message can be delivered multiple times. The message must be stored locally at the sender until the sender receives confirmation that the message was published by the receiver. The message is stored in case the message must be sent again.
Exactly once (QoS2)
The "exactly once" quality of service level 2 (QoS2) is the safest, but slowest mode of transfer. The message is always delivered exactly once and must also be stored locally at the sender, until the sender receives confirmation that the message was published by the receiver. The message is stored in case the message must be sent again. With quality of service level 2, a more sophisticated handshaking and acknowledgment sequence is used than for level 1 to ensure that messages are not duplicated
MQTT keep alive interval
The MQTT keep alive interval, which is measured in seconds, defines the maximum time that can pass without communication between the client and broker. The MQTT client must ensure that, in the absence of any other communication with the broker, a PINGREQ packet is sent. The keep alive interval allows both the client and the broker to detect that the network failed, resulting in a broken connection, without needing to wait for the TCP/IP timeout period to be reached.
If your Watson IoT Platform MQTT clients use shared subscriptions, the keep alive interval value can be set only to between 1 and 3600 seconds. If a value of 0 or a value that is greater than 3600 is requested, the Watson IoT Platform broker sets the keep alive interval to 3600 seconds.
Retained messages
Watson IoT Platform provides limited support for the retained messages feature of MQTT messaging. If the retained message flag is set to true in an MQTT message that is sent from a device, gateway, or application to Watson IoT Platform, the message is handled as an unretained message. Watson IoT Platform organizations are not authorized to publish retained messages. The Watson IoT Platform service overrides the retained message flag when it is set to true and processes the message as if the retained message flag is set to false.
I am using MQ 7.5.0.2 and Datapower client IDG7
When MQ send messages to Datapower, Datapower receive those messages using MQ front side handlers and also same way it do send messages using Backend URL
But the problem I am facing it when ever Datapower connects to MQ, Queue Input/Output count increases to (10 ~20) and remains same and the Handle state is INACTIVE.
When I see queue details using below commands it is displaying as below
display qstatus(******) type(handle)
QUEUE(********) TYPE(HANDLE)
APPLDESC(WebSphere MQ Channel)
APPLTAG(WebSphere Datapower MQClient)
APPLTYPE(SYSTEM) BROWSE(NO)
CHANNEL(*****) CONNAME(******)
ASTATE(NONE) HSTATE(INACTIVE)
INPUT(SHARED) INQUIRE(NO)
OUTPUT(NO) PID(25391)
QMURID(0.1149) SET(NO)
TID(54)
URID(XA_FORMATID[] XA_GTRID[] XA_BQUAL[])
URTYPE(QMGR)
Can any one help me in this.It only clearing when ever i restart the queue manager but I dont want to restart the qmgr every time.
HSTATE in INACTIVE state indicates "No API call from a connection is currently in progress for this object. For a queue, this condition can arise when no MQGET WAIT call is in progress.". This is likely to happen if the application(DP in this case) opened the queue and then not issuing any API calls on the opened object. Pid 25391 - is this for an amqrmppa process? Is DP expected to consume messages on this queue continuously?