I am trying to put message to Azure service bus queue, and using QueueClient to send the message. But it is not sending message to queue. But it is not even giving any error or exception. It is silently failing not sure what is going wrong here.
queueClient.Send(message);
This line is not throwing exception but not even sending message for Q1, but able to send message to Q2 in same service bus namespace.
QueueClient is from Microsoft.ServiceBus.Messaging.QueueClient.
Thanks
Please check out whether there is any autocomplete option enabled because if there is no exception or any error then it might be the problem. Also, check whether you have used any autocomplete function. If it so, that function would delete all of your message and remove it.
Related
I have a spring boot app with single kafka consumer to get messages from some topic.
But sometime errors are occurred while message handling.
I want to continue to receive the following messages as usual and at the same time be able not to lose that message and receive it, for example, the next time the service is restarted with the consumer after fixing it.
Is it possible to do this?
I understand that I need to disable auto-commit and commit successful messages manually, but, in this case, if I don't throw any exception for this exception case and commit each next successful message manually, then I will lose the previous unsuccessful one, right?
If I understand your question correctly, your assumption is that the exception occurs due to a problem in your code and not while reading the message from the topic. In that case no retry or other measures will solve your problem.
What we usually do is to catch the exception and send it to another Kafka topic. Ideally, you will also add some details on why or in which code part the exception occurred. After you have fixed the bug in your application you can consume the messages from that other topic.
I understand that I need to disable auto-commit and commit successful messages manually, but, in this case, if I don't throw any exception for this exception case and commit each next successful message manually, then I will lose the previous unsuccessful one, right?
Yes, your understanding is correct. To be more precise, you will not "loose" the message but as soon as your ConsumerGroup commits a higher offset it will never try to read the lower offset again without any manual modification.
Alternative
If you only expect very rare cases where an exception could be thrown, but you just ignore it, you can always use the consumer.seek() method in pure Kafka
public void seek(TopicPartition partition, long offset)
to start reading from a particular offset out of a topic partition.
Yes you have to manually commit them. You retry a particular message 2-3 times. If it fails after retries then you can move those messages to another topic and consume those messages when you fix whatever is causing it to fail. This will not block your queue and you won't lose and messages too.
I want to continue to receive the following messages as usual and at
the same time be able not to lose that message and receive it, for
example, the next time the service is restarted with the consumer
after fixing it.
Is it possible to do this?
You don't need to do a manual commit, instead, you can choose to implement a mechanism to do a retrial, by publishing the event in another queue and delayed consuming the event. =====> Amazon SQS has delay Queue but unfortunately there is no such thing in kafka and you have to write the implementation by yourself.
Reference articles:
Article 1
Article 2
If you are retrying the message processing, then the order of the messages can change based on your implementation. Please do keep it in mind.
Do remember that kafka does consider a consumer dead in case the message processing time exceeds max.poll.interval. Read this
I saw a question & its answer below;
https://stackoverflow.com/a/46128844/7419921
Although I understood that I cannot do anything for the error queue via MassTransit, what should I handle the error queue?
Error messages would be accumlating. It's pressing storage capacity.
It seems that I have nothing to do for the error queue. Is there no choice but to remove them?
If so, I cannot imagine a meaning of the error queue.
The meaning of error queues is very simple. Messages come to error queues because, well, of errors! When you fix issues in your application, you can move messages from the error queue back to the regular queue using Shovel plugin, and voila - you recovered lost data. We do this very often.
If you cannot move them back because these messages aren't actual anymore or they contain wrong data - this is also very valuable since using these messages you can reproduce the issue and see if you can fix the sender.
I have an application listening to messages on an IBM Websphere MQ queue.
Once a message is consumed, the application performs some processing logic.
If the processing completed OK, I would like the application to acknowledge the message and have it removed from the queue.
If an error occurred while processing, I would like the message to remain in the queue.
How is this implemented? (I'm using the .NET API)
Thanks.
MQ supports a single-phase commit protocol. You specify syncpoint when you get the message, then issue COMMIT or ROLLBACK as required. The default action if the connection is lost is ROLLBACK and if the program deliberately ends without resolving the transaction a COMMIT is assumed. (This is platform dependent so the customary advice is to explicitly call COMMIT and not rely on the class destructors to do it for you.)
This works whether the message is persistent or not. However if the message has an expiry specified and expires after being rolled back there's a chance it won't be seen again.
Of course, if the program issues a ROLLBACK the message will normally be seen again since it goes back to the same spot int he queue and for a FIFO queue that's the top. If the problem with the message is not transient then this causes a poison message loop of read/rollback/repeat. To avoid that the app can check the backout count and if it exceeds some threshold requeue the message to an exception queue.
When using JMS or XMS this is done for you by the class libraries. If the input queue's BOQNAME and BOQTHRESH attributes are set the requeue is to the queue names in BOQNAME. Otherwise a requeue to the Dead Queue is attempted. IF that fails (as it should if the system is properly secured) the listener will stop receiving messages.
The usual advice is to always specify a backout queue and either let the classes use it or code the app to use it.
Please see Usage Notes for MQGET in the MQAPI Reference and the MQGetMessageOptions.NET page in the .Net class reference.
You may want to look at the MQ Reporting Options.
Expiry, Confirmation of Arrival and Confirmation of Delivery can be requested and sent via a response queue back to the sending application by the receiving Queue Manager.
Positive and Negative Acknowledgements can also be generated by the receiving application provided they use the related reporting attributes found in the Message Descriptor.
Exception can be requested and sent via a response queue back to the sending application by any Queue Manager in the transmission chain or generated by the receiving application.
1 Read the message using MQC.MQGMO_SYNCPOINT,
2 process it
3 call MQQueueManager.Commit()
If Commit() is not called explicitly, or implicitly (eg exception is thrown), all messages that have been de-queued will be re-enqueued.
I try to make example project using spring and jms.
so, I refer a site(https://spring.io/guides/gs/messaging-jms/).
However, when i run the server, i don't know how to subscribe message in the other program.
please give me some hint.
If you look at the code, author has used destination name as - "mailbox-destination". When you need to receive a message, you need to listen to the same destination. Author has also given the code of Receiver. Please take a look at src/main/java/hello/Receiver.java.
You will need the same code in other application to receive the message. If you would like to try some more examples, take a look at :
receive message using direct message listener
receive message using message listener adapter
Currently I'm using bluelock's camel-spring-amqp component for my application.
What I want to achieve is:
Pull a message from RabbitMQ server.
Persist it to a database on successful processing / Send it to another "Error" queue on Exception
Tell the original queue that it is now safe to remove the message from the queue.
As of this writing, I'm able to pull from rabbit and persist to database using camel routes. What I don't really know how to do is acknowledge that my processing is done to the original queue. Here is my current route:
from("spring-amqp:EXCHANGE:queuename?autodelete=false&durable=true&type=direct&routingKey=")
.bean(Transform.class, "transform(byte[])")
.to("jpa:com.my.company.models.MyModel?entityType=java.util.ArrayList")
I realize I can set the acknowledgmentMode to NONE. But I don't know how to "manually" acknowledge once I have persisted my message.
Thanks for any help!
I'm new in Camel but I know a thing or two about RabbitMQ.
With RabbitMQConsumer the message is acknowledged if the processor doesn't throw any exception (line 133 at RabbitMQConsumer source).
So I suppose if you let your processor propagate the exception, the message won't be acknowledged. I haven't used spring-amqp but I guess it should have a similar behaviour.