When trying to use c# and ibm mq client (9.1.5), I want to use the syncpoint functionality.
var getMessageOptions = new MQGetMessageOptions();
getMessageOptions = new MQGetMessageOptions();
getMessageOptions.Options += MQC.MQGMO_WAIT + MQC.MQGMO_SYNCPOINT;
getMessageOptions.WaitInterval = 20000; // 20 seconds wait
Hashtable props = new Hashtable();
props.Add(MQC.HOST_NAME_PROPERTY, "localhost");
props.Add(MQC.CHANNEL_PROPERTY, "DOTNET.SVRCONN");
props.Add(MQC.PORT_PROPERTY, 3636);
MQQueueManager qm = new MQQueueManager("QM", props);
MQQueue queue = qm.AccessQueue("Q1", MQC.MQOO_INPUT_AS_Q_DEF);
try
{
var message = new MQMessage();
queue.Get(message, getMessageOptions);
string messageStr = message.ReadString(message.DataLength);
SaveTheMessageToAFile(messageStr);
//qm.Commit();
}
catch (MQException e) when (e.Reason == 2033)
{
// Report exceptions other than "no messages in the queue"
Log.Information("No messages in the queue");
}
catch (Exception ex)
{
Log.Error($"Exception when trying to capture a message from the queue:
}
I would have expected to see the same message each time if i didn't call commit. Is there something that needs to be enabled on the queuemanager?
In your example you are not issuing a Commit() or a Backout(), so at that point the message will just be in a uncommitted state. If you were to then kill off your process the message would get rolled back to the queue. As you mentioned in the comments if you call Disconnect(), in most cases this will implicitly commit uncommitted messages.
This is documented in the IBM MQ KC in a few pages:
Reference>Developing applications reference>The IBM MQ .NET classes and interfaces>MQQueueManager.NET class
Disconnect();
...
Generally, any work performed as part of a unit of work is committed. However, if the unit of work is managed by .NET, the unit of work might be rolled back.
NOTE: managed by .NET means a Distributed transactions, not what you are doing.
Developing applications>Developing .NET applications
When you use the procedural interface, you disconnect from a queue manager by using the call MQDISC( Hconn, CompCode, Reason), where Hconn is a handle to the queue manager.
In the .NET interface, the queue manager is represented by an object of class MQQueueManager. You disconnect from the queue manager by calling the Disconnect() method on that class.
Developing applications>Developing MQI applications with IBM MQ>Writing a procedural application for queuing>Committing and backing out units of work>Syncpoint considerations in IBM MQ applications
Except on z/OS batch with RRS, if a program issues the MQDISC call while there are uncommitted requests, an implicit syncpoint occurs. If the program ends abnormally, an implicit backout occurs.
Related
Dears,
I am trying to do some kind of event-driven Microservices. Currently, I was able to consume a message from Kafka and update database record when message is received using Quarkus & Smallrye-Reactive messaging extension. What I want to achieve further is to be able to send a message to other topic in case of success and send a message to error topic otherwise. I know that we can use return and #outgoing annotation for emitting new message but I don't think it will fit in my use case. I need a guidance here, if error happens while consuming a message. Should I return message to the original topic (by not acknowledging the message) or should I consume it and produce error message to different topic to rollback the original transaction.
Here is my code :
#Incoming("new-payment")
public void newMessage(String msg) {
LOG.info("New payment has been received.");
LOG.info("Payload is {}", msg);
PaymentEvent pe = jsob.fromJson(msg, PaymentEvent.class);
mysqlPool.preparedQuery("select totalBuyers from Book where isbn = ? ",
Tuple.of(pe.getIsbn()))
.thenApply(rs -> {
RowIterator<Row> iterator = rs.iterator();
if (iterator.hasNext()) {
return iterator.next().getInteger(0) + 1;
} else {
return Integer.valueOf(0);
}
})
.thenApply(totalCount -> {
return mysqlPool.preparedQuery("update Book set totalBuyers = ?",
Tuple.of(totalCount));
})
.whenComplete((rs, err) -> {
if (err != null) {
//Emit an error to error topic.
} else {
//Emit a msg to other service.
}
});
}
Also if you've better code please submit, I am still newbie in reactive programming :).
I've been doing enterprise integration for years and I think that you would want to do both.
Should I return message to the original topic (by not acknowledging
the message) or should I consume it and produce error message to
different topic to rollback the original transaction.
The event should remain on the topic for another instance to potentially pick up and process. And an error message should be logged as an event. Perhaps the same consumer could pick up and reprocess the event successfully.
An EDA (Event Driven Architecture) may offer different ways to handle this but on an ESB the message would be marked as tried. Generally three tried attempts would send it to a dead-letter queue so that it can be corrected and reprocessed later.
Our enterprise is also starting to design and build applications using EDA so I am interested to read what others have to say on this question. And KUDOS to you for focusing on Quarkus. I believe that this is one of the best technologies to come from Redhat that I have seen yet!
Another problem with this approach is that you are doing “2 writes in 1 service” e.g. one call to the db and another one to a topic. And this can become problematic when one of the 2 writes fails.
If you want to avoid this and use a pure event driven approach, then you need to reorder your events in such a way that writing to a db is the last event in the whole flow so that you can prevent 2 writes from 1 service.
Thus in your case: change the 2nd thenApply(..) method from updating the db into firing a new event to another topic. And the consumer of this new topic should do the db update. Thus the flow becomes like this:
Producer -> topic1 -> consumer (select from ...) & fire event to another topic -> topic2 -> consumer (update table).
I need to pick up messages off an IBM MQ queue without polling so I am looking at some async sample code from IBM. I see sample code to async get with JMS/XMS but not MQI (amqmdnet).
The closest I have come to not having to poll continuously with MQI is to wait, but not advised of course for long periods of time so you are still polling:
requestMessage = new MQMessage();
MQGetMessageOptions gmo = new MQGetMessageOptions();
gmo.Options = MQC.MQGMO_WAIT;
Is there a way to asynchronously get messages off a queue using MQI? Or is XMS the only way to go?
I have a client app that connects to an Oracle AQ multi-consumer queue. I want to use OracleAQQueue.Listen to listen for new messages on the queue. API docs show that the Listen method can be used for multi-consumer queues. My code for listening to the queue is shown below.
string consumerName = "APPINST1";
using (OracleConnection con = new OracleConnection(connectionString))
{
con.Open();
OracleAQQueue queue = new OracleAQQueue("MY_Q");
queue.MessageType = OracleAQMessageType.Udt;
queue.UdtTypeName = "MY_Q_MSG";
queue.DequeueOptions.DeliveryMode = OracleAQMessageDeliveryMode.Persistent;
queue.Connection = con;
Console.WriteLine("Listening for messages...");
queue.Listen(new string[] { consumerName });
}
The problem that I'm having is that on the line of code where I call queue.Listen(), I get the Oracle exception:
ORA-25295: Subscriber is not allowed to dequeue buffered messages
Googling for advice on this particular error hasn't been too helpful. I've removed and re-added my subscriber to the queue several times to no avail. My guess is that I'm not setting some property correctly before I make the call to Listen, but I can't figure out the issue.
Any ideas?
I ran across the following note in the Streams Advanced Queuing User's Guide, in Chapter 10 - Oracle Streams AQ Operations Using PL/SQL:
Note: Listening to multiconsumer queues is not supported in the Java API.
Although I can't find it explicitly stated anywhere, I'm guessing the same rule applies to the ODP.NET API.
You must set the visibility attribute to IMMEDIATE to use buffered messaging.
In my application, I have a queue (HornetQ) set up on JBoss 7 AS.
I have used Spring batch to do some work once the messages is received (save values in database etc.) and then the consumer commits the JMS session.
Sometimes when there is an exception while processing the message, the excecution of consumer is aborted abruptly.
And the message remains in "in delivery" state. There are about 30 messages in this state on my production queue.
I have tried restarting the consumer but the state of these messages is not changed. The only way to remove these
messages from the queue is to restart the queue. But before doing that I want a way to read these messages so
that they can be corrected and sent to the queue again to be processed.
I have tried using QueueBrowser to read them but it does not work. I have searched a lot on Google but could not
find any way to read these messages.
I am using a Transacted session, where once the message is processed, I am calling:
session.commit();
This sends the acknowledgement.
I am implementing spring's
org.springframework.jms.listener.SessionAwareMessageListener
to recieve messages and then to process them.
While processing the messages, I am using spring batch to insert some data in database.
For a perticular case, it tries to insert data too big to be inserted in a column.
It throws an exception and transaction is aborted.
Now, I have fixed my producer and consumer not to have such data, so that this case should not happen again.
But my question is what about the 30 "in delivery" state messages that are in my production queue? I want to read them so that they can be corrected and sent to the queue again to be processed. Is there any way to read these messages? Once I know their content, I can restart the queue and submit them again (after correcting them).
Thanking you in anticipation,
Suvarna
It all depends on the Transaction mode you are using.
for instance if you use transactions:
// session here is a TX Session
MessageConsumer cons = session.createConsumer(someQueue);
session.start();
Message msg = consumer.receive...
session.rollback(); // this will make the messages to be redelivered
if you are using non TX:
// session here is auto-ack
MessageConsumer cons = session.createConsumer(someQueue);
session.start();
// this means the message is ACKed as we receive, doing autoACK
Message msg = consumer.receive...
//however the consumer here could have a buffer from the server...
// if you are not using the consumer any longer.. close it
consumer.close(); // this will release messages on the client buffer
Alternatively you could also set consumerWindowSize=0 on the connectionFactory.
This is on 2.2.5 but it never changed on following releases:
http://docs.jboss.org/hornetq/2.2.5.Final/user-manual/en/html/flow-control.html
I"m covering all the possibilities I could think of since you're not being specific on how you are consuming. If you provide me more detail then I will be able to tell you more:
You can indeed read your messages in the queue using jmx (with for example jconsole)
In Jboss As7 you can do it the following way :
MBeans>jboss.as>messaging>default>myJmsQueue>Operations
listMessagesAsJson
[edit]
Since 2.3.0 You have a dedicated method for this specific case :
listDeliveringMessages
See https://issues.jboss.org/browse/HORNETQ-763
My experimental application is quite simple, trying what can be done with Actors and Akka.
After JVM start, it creates actor system with couple of plain actors, JMS consumer (akka.camel.Consumer) and JMS producer (akka.camel.Producer). It sends couple of messages between actors and also JMS producer -> JMS server -> JMS consumer. It basically talks to itself via JMS service.
From time to time I was experiencing weird behaviour: it seemed that from time to time, first of messages which where supposed to be sent to JMS server was somehow lost. By looking at my application logs, I could see that applications is trying to send the message but it was never received by JMS server. (For each run I have to start JVM&Application again).
Akka Camel Documentation mentions that it's possible that some components may not be fully initialized at the begining: "Some Camel components can take a while to startup, and in some cases you might want to know when the endpoints are activated and ready to be used."
I tried to implement following to wait for Camel initialization
val system = ActorSystem("actor-system")
val camel = CamelExtension(system)
val jmsConsumer = system.actorOf(Props[JMSConsumer])
val activationFuture = camel.activationFutureFor(jmsConsumer)(timeout = 10 seconds, executor = system.dispatcher)
val result = Await.result(activationFuture,10 seconds)
which seems to help with this issue. (Although, when removing this step now, I'm not able to recreate this issue any more... :/).
My question is whether this is correct way to ensure all components are fully initialized?
Should I use
val future = camel.activationFutureFor(actor)(timeout = 10 seconds, executor = system.dispatcher)
Await.result(future, 10 seconds)
for each akka.camel.Producer and akka.camel.Consumer actor to be sure that everything is initialized properly?
Is that all I should to do, or something else should be done as well? Documentation is not clean on that and it's not easy to test as issue was happening only occasionaly...
You need to initialize the camel JMS component and also Producer before sending any messages.
import static java.util.concurrent.TimeUnit.SECONDS;
import scala.concurrent.Future;
import scala.concurrent.duration.Duration;
import akka.dispatch.OnComplete;
ActorRef producer = system.actorOf(new Props(SimpleProducer.class), "simpleproducer");
Timeout timeout = new Timeout(Duration.create(15, SECONDS));
Future<ActorRef> activationFuture = camel.activationFutureFor(producer,timeout, system.dispatcher());
activationFuture.onComplete(new OnComplete<ActorRef>() {
#Override
public void onComplete(Throwable arg0, ActorRef arg1)
throws Throwable {
producer.tell("First!!");
}
},system.dispatcher());