I observe errors as below when Websphere server instance is stopped from admin console
Caused by: javax.ejb.TransactionRolledbackLocalException: ; nested exception
is: javax.transaction.TransactionRolledbackException: Transaction is ended due to timeout
at com.ibm.ws.Transaction.JTA.TranManagerImpl.completeTxTimeout(TranManagerImpl.java:576)
at com.ibm.ws.Transaction.JTA.TranManagerSet.completeTxTimeout(TranManagerSet.java:625)
These are inflight txns during server stop.
Increasing the timeouts from "Application servers->server->Transaction Service" does not seem to help.
Is this to do with the server quiesce timeouts? if yes is there a way to configure those.
Also the rollbacks are not observed when I "terminate" the server from the Admin Console, only observed when I "stop" the server.
Any ideas to debug this issue would be great.
What you want is probably Deployment for transactional high availability. The describe method is the only product feature that is available for finishing those transactions without getting actual errors.
What happens for you is that the WebSphere Application server gives each container some time to shut down. After the shutdown timeouts it will use force. The transactions get rolled back. Well, you could also change the heurestic policy to for instance COMMIT. That depends on whether it is better for your application that everything in the transaction gets lost or whether only the rest of the transaction gets lost.
Related
IBM MQ documentation says:
"If an application disconnects (MQDISC) from a queue manager while a global unit of work coordinated by IBM® MQ is still active, an attempt is made to commit the unit of work. If, however, the application terminates without disconnecting, the unit of work is rolled back as the application is deemed to have terminated abnormally."
The case of application issuing a MQDISC is clear. My doubt is about the second scenario: let's say I'm putting messages and the applicative servers suddenly loss power before the commit. How does the rollback work if the application is not available at all? Would those message end in the DLQ, or...?
We are experiencing relatively random situation when the messages which are sent to the particular queues ending up as Messages Pending. The only way to solve the problem is to restart an associated Managed Server where an MDB is deployed. There are no Errors/Exceptions in the Managed Server log files. In the Admin Console of the WebLogic we can see that there are "stuck threads" in the Managed Server where we have MDB. What's even more strange is the fact that despite having a lot of messages pending an MDB continues to listen to the Queue and process the messages which arrive later.
Is there any way to resolve the issue without restarting Managed Server?
My suggestion to you is to go on server -> Monitoring -> Threads
You will see a list of threads currently running on your managed server. Look for the threads marked as stuck or hogging and jot down their numbers..
Then, click on Thread Dump on top of the page and look for the stack with the number of your thread... You will see exactly which method is holding your thread and hogging your MDB.
Hope it helps !!!
What are the best practices regarding sessions in an application that is designed to fetch messages from a MQ server every 5 seconds?
Should I keep one session open for the whole time (could be weeks or longer), or better open a session, fetch the messages, and then close the session again?
I am using the .net IBM XMS v8 client library.
Adding to what #Attila Repasi's response, I would go for a consumer with message listener attached. The message listener would get called whenever a message needs to be delivered to application. This avoids application explicitly calling receive() to retrieve messages from queue and waste CPU cycles if there are no messages on the queue.
Check the XMS.NET best practices
Keep the connection and session open for a longer period if your application sends or receive message continuously. Creation of connection or session is a time consuming operation and consumes lot of resources and involves network flow (for client connections).
I'm not sure what you are calling a session, but typically applications connect to the queue manager serving them once at start, and keep that connection up while running.
I don't see a reason to disconnect just to reconnect 5 seconds later.
As for keeping the queues open, it depends on your environment.
If there are no special circumstances, I would keep the queue open.
I think the most worth thinking about is how you issue the GETs to read the messages.
We are using MQ base classes and MQ as a XA transaction coordinator.
Environment
MQ 7.5
Red Hat Linux 6.4
Java 1.7
Scenario :
1. MqManager.begin
queue. get (sync point set in get option)
db save
MqManager.commit/rollback
go back to step 1
Most of the time step 5 i. e. starting of the new transaction works fine however intermittently exception is thrown UoW already in progress. Since step 4 call was successful we believed transaction should be either committed or rolled back successfully. It shouldn't cause issues when a new transaction is being started. Can someone please suggest what can be causing UoW not being committed or rolled back even after successful commit/rollback call on the q manager?
Thanks
Vaibhav
You are probably caching JMS consumers or some other JMS objects.
Essentially at lower level MQ conversation is driven by various lower level client calls like PUT, GET or MQOPEN commands. Your problem is that issued MQBEGIN command was not matched by proper MQCLOSE (?) command. This all happens in JMS driver and it is hidden from Java developer.
If I remember correctly MQBEGIN call is mapped to the creation o jms consumer. Closing all jms object's but connection itself will work.
There is downfall to this. Problem with closing all JMS objects is that it increases cpu usage, as MQOPEN and similar calls tend to be CPU expensive.
We are using rabbitmq 3.0.1 on CentOS 6, and as a client Spring spring-rabbit version 1.1.2.RELEASE. (I know these aren't the latest versions, see later).
We send messages to rabbitmq via this client. These messages are initiated via an external rest call. Someone else calls our web service updates the database and sends the amqp message. I would like to be informed if rabbitmq blocks the client - for instance if the disk_free_limit threshold is reached.
Importantly, I would like to be informed in the same thread as that processing the web request, so that I can rollback the transaction.
Our web service can also update a database (within a transaction obviously). Normally, this works fine. However, under certain circumstances, rabbitmq can block our web server - the most obvious being when the disk_free_limit is reached. This blocks the web server Thread, indefinitely. The external caller of the web service will obviously time out after a sensible period, but the thread in our web service doesn't - it stays around, and keeps the resources, and importantly the transaction open.
The web server is blocking the thread because it is transactional. It isn't the initial message which is blocking, it is the commit. I assume that rabbitmq is blocking because it can't persist it or something like that. The thread is blocking until rabbitmq sends the commit ok message back. The bit of code is deep within the rabbitmq implementation - com.rabbitmq.client.impl.ChannelN
public Tx.CommitOk More ...txCommit() throws IOException
{
return (Tx.CommitOk) exnWrappingRpc(new Tx.Commit()).getMethod();
}
and this eventually calls the following method from com.rabbitmq.client.impl.AMQChannel
public T More ...getReply() throws ShutdownSignalException
{
return _blocker.uninterruptibleGetValue();
}
The preferable solution for this would be some sort of timeout on the txCommit - then I could throw an exception and fail the web service with a 500 or whatever. I can't find any way of doing this.
What I have found is:
addBlockedListener - this adds a listener on a message sent by rabbitmq when this it is blocked. This is good, but the message will is treated by another thread - so I can't fail the web service. Using this I can at least log the fact that rabbitmq is blocked, through syslog or whatever. However, this isn't available on the version that we run - we would have to upgrade to the latest. We would prefer not to do this because of the testing it would imply.
setConnectionTimeout(int) - this sets the connection timeout for the initial connection to rabbitmq. This doesn't apply in my case, because rabbitmq is up and running and accepts the connection.
AmqpTemplate.setReplyTimeout() - as shown above, this reply timeout does not apply to the commit.
I fully understand that this situation (disk_free_limit threshold is breached) is a situation which should not occur in a production system. However, I would like to be able to cope nicely with this situation so that my application behaves nicely when one of its components (rabbitmq) has a problem.
So, what other options do I have? Is there any way, short of rewriting portions of the spring amqp client or removing the transactionality of doing what I want?