I have an enterprise event-driven application.
I have 16 MDBs that read from a JMS queue. When I got some uncontrolled exceptions I receive this stack :
####<Apr 18, 2011 11:47:15 AM CEST> <Error> <WebLogicServer> <sapmop11> <cont0Server11> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1303120035886> <BEA-000337> <[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)' h
as been busy for "647" seconds working on the request "weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl#16b13f90", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.lang.Thread.sleep(Native Method)
weblogic.ejb.container.internal.JMSPollerManager.waitForPollersToStop(JMSPollerManager.java:82)
weblogic.ejb.container.internal.JMSConnectionPoller.waitForMessagePollerExit(JMSConnectionPoller.java:795)
weblogic.ejb.container.internal.JMSConnectionPoller.disconnect(JMSConnectionPoller.java:920)
weblogic.ejb.container.internal.MDConnectionManager.timerExpired(MDConnectionManager.java:166)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
>
####<Apr 18, 2011 11:48:15 AM CEST> <Error> <WebLogicServer> <sapmop11> <cont0Server11> <[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1303120095935> <BEA-000337> <[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)' h
as been busy for "707" seconds working on the request "weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl#16b13f90", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.lang.Thread.sleep(Native Method)
weblogic.ejb.container.internal.JMSPollerManager.waitForPollersToStop(JMSPollerManager.java:82)
weblogic.ejb.container.internal.JMSConnectionPoller.waitForMessagePollerExit(JMSConnectionPoller.java:795)
weblogic.ejb.container.internal.JMSConnectionPoller.disconnect(JMSConnectionPoller.java:920)
weblogic.ejb.container.internal.MDConnectionManager.timerExpired(MDConnectionManager.java:166)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
>
After this all the MDBs stop reading from the queue and the application need to be restarted.
This is the MDB onMessage code:
#Override
#TransactionAttribute(TransactionAttributeType.REQUIRED)
public void onMessage(Message message) {
OnlineEventMessage<? extends Serializable> eventMessage = null;
try {
TextMessage textMessage = (TextMessage) message;
String stringMessage = textMessage.getText();
logger.debug("e' stato ricevuto il messaggio {}", stringMessage);
logger.debug("eventMessageParser {}", onlineEventMessageParser);
logger.debug("legacyEventMessageService {}", legacyEventMessageService);
EventMessageAnagrafica anagrafica = anagraficaParser.parse(stringMessage, true);
String multichannelId = anagrafica.getMultichannelId();
eventMessage = onlineEventMessageParser.parse(stringMessage, true);
MDC.put("multichannelId", multichannelId);
MDC.put("eventType", eventMessage.getEventType().getEventCode());
legacyEventMessageService.handle(eventMessage);
} catch (JMSException e) {
retryJmsTemplate.send(destination, new RedeliveryMessageCreatorUtil(message));
logger.error("si e' verificato un errore JMS nel processamento dell'evento {}, riaccodamento in corso...", e);
} catch (RuntimeException e) {
logger.error("si e' verificata una runtime exception per il messaggio {}", e);
legacyErrorHandlerService.handle(eventMessage);
} catch (Exception e) {
logger.error("si e' verificato un errore inaspettato per il messaggio {}", e);
if (eventMessage != null && eventMessage.getIdEventMessage() != null) {
logger.error("Il messaggio gia' presente nelle catene di elaborazione");
OnlineEventMessage<?> onlineMessage = eventMessageDao.getItemByID(eventMessage.getIdEventMessage());
if (onlineMessage != null) {
onlineMessage.setEventState(EventState.PROCESSING_ERROR);
eventMessageDao.updateItem(onlineMessage);
eventMessageDao.flush();
}
}
context.setRollbackOnly();
}
}
Any idea?
Kind regards
Massimo
This seems like a transaction issue. Try taking a Thread Dump next time this happens and take a look at what the threads are doing. You can do this through the Admin Console.
Related
I am testing ZMQ PUB/SUB socket performance with NetMQ.
In the test program below, SUB socket is used as a server to receive messages and PUB socket is used as a client to receive messages. The topic of a message is simply a sequence number and the content of a message is the time when it is sent by the PUB client. The SUB server measures the latency by comparing between the time when the message is received time and the time (which is the message sent time) inside the message.
For 1,000 messages sent over ~1 second, the performance results I find are the average latency ~4ms and maximum latency ~40ms. However, based on http://wiki.zeromq.org/results:more-precise-0mq-tests, ZMQ's latency should be in the magnitude of microseconds which is much less than the latency I measured.
Do I use ZMQ PUB/SUB sockets wrongly? Is there any way to reduce the latency? Any ideas will be welcome and appreciated.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using NetMQ;
using NetMQ.Sockets;
namespace NetMQ_Hello
{
class Program
{
static void Main(string[] args)
{
if (args.Length == 0)
{
Console.WriteLine("Please use \"server\" or \"client\" as the first argument.");
return;
}
string mode = args[0];
if (mode == "server")
{
RunSubAsServer();
}
else if (mode == "client")
{
RunPubAsClient();
}
}
static void RunPubAsClient()
{
string address = "tcp://localhost:1234";
PublisherSocket pubSocket = new PublisherSocket();
pubSocket.Connect(address);
Thread.Sleep(10);
Console.WriteLine("Starting to send messages...");
DateTime startTime = DateTime.UtcNow;
for (int i = 0; i < 1000; i++)
{
long data = DateTime.UtcNow.ToBinary();
byte[][] frames = new byte[2][] { BitConverter.GetBytes(i), BitConverter.GetBytes(data) };
pubSocket.SendMultipartBytes(frames);
Console.WriteLine(string.Format("Sent: {0}, {1}", i, data));
}
DateTime endTime = DateTime.UtcNow;
Console.WriteLine(string.Format("1,000 messages sent in {0}", endTime - startTime));
Console.WriteLine("Press enter to exit...");
Console.ReadLine();
pubSocket.Disconnect(address);
}
static void RunSubAsServer()
{
string address = "tcp://*:1234";
SubscriberSocket subSocket = new SubscriberSocket();
subSocket.Bind(address);
subSocket.SubscribeToAnyTopic();
Thread.Sleep(10);
// Stats
List<TimeSpan> latencies = new List<TimeSpan>();
Console.WriteLine("Starting to receive messages...");
List<byte[]> frames = new List<byte[]>();
while (true)
{
if (subSocket.TryReceiveMultipartBytes(
timeout: TimeSpan.FromSeconds(1),
frames: ref frames,
expectedFrameCount: 2))
{
DateTime now = DateTime.UtcNow;
int topic = BitConverter.ToInt32(frames[0], 0);
DateTime sentTime = DateTime.FromBinary(BitConverter.ToInt64(frames[1], 0));
TimeSpan latency = now - sentTime;
Console.WriteLine(string.Format("Received: {0}, {1}, delay {2}",
topic, sentTime, latency));
latencies.Add(latency);
if (topic == 1000 - 1)
{
break;
}
}
}
int n = latencies.Count;
double max = latencies.Max().TotalMilliseconds;
double mean = latencies.Sum(s => s.TotalMilliseconds) / n;
Console.WriteLine(String.Format("Latency\nMax: {0}ms\nMean: {1}ms", max, mean));
Console.WriteLine("Press enter to exit...");
Console.ReadLine();
subSocket.Unbind(address);
}
}
}
Sample output from Sub server:
Received: 990, 9/30/2020 12:53:11 PM, delay 00:00:00.0010009
Received: 991, 9/30/2020 12:53:11 PM, delay 00:00:00
Received: 992, 9/30/2020 12:53:11 PM, delay 00:00:00.0019993
Received: 993, 9/30/2020 12:53:11 PM, delay 00:00:00.0010016
Received: 994, 9/30/2020 12:53:11 PM, delay 00:00:00
Received: 995, 9/30/2020 12:53:11 PM, delay 00:00:00.0019993
Received: 996, 9/30/2020 12:53:11 PM, delay 00:00:00.0010007
Received: 997, 9/30/2020 12:53:11 PM, delay 00:00:00
Received: 998, 9/30/2020 12:53:11 PM, delay 00:00:00
Received: 999, 9/30/2020 12:53:11 PM, delay 00:00:00.0030002
Latency
Max: 39ms
Mean: 4.49101730769231ms
public string WriteMsg(string strInputMsg)
{
string strReturn = "";
try
{
MQQueue queue = null;
MQQueueManager QueueManagerName = null ;
QueueManagerName = new MQQueueManager("GRBAAQM");
queue = QueueManagerName.AccessQueue(QueueName, MQC.MQOO_OUTPUT
+ MQC.MQOO_FAIL_IF_QUIESCING);
message = strInputMsg;
queueMessage = new MQMessage();
queueMessage.WriteString(message);
queueMessage.Format = MQC.MQFMT_STRING;
queuePutMessageOptions = new MQPutMessageOptions();
queue.Put(queueMessage, queuePutMessageOptions);
strReturn = "Message sent to the queue successfully";
}
catch (MQException MQexp)
{
strReturn = "Exception: " + MQexp.Message;
}
catch (Exception exp)
{
strReturn = "Exception: " + exp.Message;
}
return strReturn;
}
public string ReadMsg()
{
String strReturn = "";
try
{
MQQueue queue = null;
MQQueueManager QueueManagerName = null;
QueueManagerName = new MQQueueManager("GRBAAQM");
queue = QueueManagerName.AccessQueue(QueueName, MQC.MQOO_INPUT_AS_Q_DEF +
MQC.MQOO_FAIL_IF_QUIESCING);
queueMessage = new MQMessage();
queueMessage.Format = MQC.MQFMT_STRING;
queueGetMessageOptions = new MQGetMessageOptions();
queue.Get(queueMessage, queueGetMessageOptions);
strReturn =
queueMessage.ReadString(queueMessage.MessageLength);
}
catch (MQException MQexp)
{
strReturn = "Exception : " + MQexp.Message;
}
catch (Exception exp)
{
strReturn = "Exception: " + exp.Message;
}
return strReturn;
}
These two methods in this program helps us to read the messages from queue and displays but how to insert this feature while reading message FROM queue, read only if the message count has reached 10.
Why do you care how many messages are in the queue? MQ is NOT a database. If a message is in the queue then it should be processed. If you need to group messages together then have the sender use MQ's message grouping feature.
Did you read about MQ triggering? A program can be triggered (started) based on a triggering event. i.e. Trigger-first, trigger-every & trigger-depth.
I have created Simple JMS Client and Receiver Program. I have used JDk1.7 and activeMQ 5.10.0.My Sender code is executing with particlur message
MessageProducer mp= session.createProducer(destinatin);
Message message = session.createTextMessage("Hi Welcome to ActiveMQ Example");
mp.send(message);
System.out.println("Message Has Sent");
and
Receiver code My Reciver Code is this one where it is not printing anything.
and after some time it gives me error of connection timeout.Could you find out where I am creating mistake...
ConnectionFactory connectionFactory = new ActiveMQConnectionFactory("tcp://Name-PC:61616");
//connectin creation
Connection con = connectionFactory.createConnection();
Session session = con.createSession(false,Session.AUTO_ACKNOWLEDGE);
Destination dest= new ActiveMQQueue("Test1.Queue");
MessageConsumer Consumer = session.createConsumer(dest);
Message message = Consumer .receive();
System.out.println("End of Message1");
TextMessage text = (TextMessage) message;
System.out.println("Message" +text.getText());
System.out.println("End of Message");
In http://localhost:8161/admin/queues.jsp it is showing content on
Name Test1.Queue, Number Of Pending Messages =1, Number Of Consumers=1 Messages Enqueued =1,but Messages Dequeued not showing anything
You need to start the connection.
This groovy code works for me:
factory = new ActiveMQConnectionFactory("tcp://tpmint:61616");
dest = new ActiveMQQueue("foo.bar");
conn = null;
session = null;
consumer = null;
try {
conn = factory.createConnection();
println "Connected: $conn";
session = conn.createSession(false,Session.AUTO_ACKNOWLEDGE);
println "Session: $session";
consumer = session.createConsumer(dest);
println "Consumer: $consumer";
conn.start();
msg = consumer.receive(1000);
if(msg==null) println "Timeout";
else println "Msg:$msg";
} finally {
if(consumer!=null) try { consumer.close(); println "Consumer Closed";} catch (e) {}
if(session!=null) try { session.close(); println "Session Closed";} catch (e) {}
if(conn!=null) try { conn.close(); println "Connection Closed"; } catch (e) {e.printStackTrace(System.err);}
}
Output:
Connected: ActiveMQConnection {id=ID:tpmint-51137-1445798087365-0:8,clientId=null,started=false}
Session: ActiveMQSession {id=ID:tpmint-51137-1445798087365-0:8:1,started=false}
Consumer: ActiveMQMessageConsumer { value=ID:tpmint-51137-1445798087365-0:8:1:1, started=false }
Msg:ActiveMQTextMessage {commandId = 7, responseRequired = false, messageId = ID:tpmint-58446-1445793097761-4:2:1:1:3, originalDestination = null, originalTransactionId = null, producerId = ID:tpmint-58446-1445793097761-4:2:1:1, destination = queue://foo.bar, transactionId = null, expiration = 0, timestamp = 1445798905534, arrival = 0, brokerInTime = 1445798905534, brokerOutTime = 1445802982704, correlationId = , replyTo = null, persistent = false, type = , priority = 0, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = null, marshalledProperties = org.apache.activemq.util.ByteSequence#5346e84e, dataStructure = null, redeliveryCounter = 2, size = 0, properties = {JMSXMessageCounter=1}, readOnlyProperties = true, readOnlyBody = true, droppable = false, text = hey hey}
Consumer Closed
Session Closed
Connection Closed
I have installed gvnix geo-petclinic example over JBoss AS 7.3 and Oracle 11g without problems.
But when I access the screen Maps -> Mapview I get this error in logs:
16:55:17,624 WARN [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (http-localhost/127.0.0.1:8080-4) SQL Error: 6553, SQLState: 65000
16:55:17,624 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (http-localhost/127.0.0.1:8080-4) ORA-06553: PLS-306: número o tipos de argumentos erróneos al llamar a 'FROM_SDO_GEOM'
Reviewing code in "src/main/java/org/gvnix/domain/Owner_Roo_GvNIXEntityMapLayer.aj", I update the following method
and running fine.
public static List<Owner> Owner.findAllOwnersByBoundingBox(String bbox) {
TypedQuery<Owner> q = entityManager().createQuery("SELECT o FROM Owner o WHERE intersects(o.location, :bbox) = true OR intersects(o.distance, :bbox) = true OR intersects(o.area, :bbox) = true ", Owner.class);
//q.setParameter("bbox", String.format("POLYGON((%s))", bbox));
WKTReader fromText = new WKTReader();
Geometry geom = null;
try {
geom = fromText.read(String.format("POLYGON((%s))", bbox));
} catch (ParseException e) {
throw new RuntimeException("Not a WKT string:" + bbox);
}
q.setParameter("bbox", geom);
return q.getResultList();
}
It is a bug in gvNIX?
Maybe it could fix for the next version
This is a weird situation and I normally would never do it but our system has unfortunately now required this kind of scenario.
The System
We are running a Spring/Hibernate applications that is using OpenSessionInView and TransactionInterceptor to manage our transactions. For the most part it works great. However, we have recently required the need to spawn a number of threads to make some concurrent HTTP requests to providers.
The Problem
We need the entity that is passed into the thread to have all of the data that we have updated in our current transaction. The problem is we spawn the thread deep down in the guts of our service layer and it's very difficult to make a smaller transaction to allow this work. We tried originally just passing the entity to the thread and just calling:
leadDao.update(lead);
The problem is that we than get the error about the entity living in two sessions. Next we try to commit the original transaction and reopen as soon as the threads are complete.
This is what I have listed here.
try {
logger.info("------- BEGIN MULTITHREAD PING for leadId:" + lead.getId());
start = new Date();
leadDao.commitTransaction();
List<Future<T>> futures = pool.invokeAll(buyerClientThreads, lead.getAffiliate().getPingTimeout(), TimeUnit.SECONDS);
for (int i = 0; i < futures.size(); i++) {
Future<T> future = futures.get(i);
T leadStatus = null;
try {
leadStatus = future.get();
if (logger.isDebugEnabled())
logger.debug("Retrieved results from thread buyer" + leadStatus.getLeadBuyer().getName() + " leadId:" + leadStatus.getLead().getId() + " time:" + DateUtils.formatDate(start, "HH:mm:ss"));
} catch (CancellationException e) {
leadStatus = extractErrorPingLeadStatus(lead, "Timeout - CancellationException", buyerClientThreads.get(i).getBuyerClient().getLeadBuyer(), buyerClientThreads.get(i).getBuyerClient().constructPingLeadStatusInstance());
leadStatus.setTimeout(true);
leadStatus.setResponseTime(new Date().getTime() - start.getTime());
logger.debug("We had a ping that didn't make it in time");
}
if (leadStatus != null) {
completed.add(leadStatus);
}
}
} catch (InterruptedException e) {
logger.debug("There was a problem calling the pool of pings", e);
} catch (ExecutionException e) {
logger.error("There was a problem calling the pool of pings", e);
}
leadDao.beginNewTransaction();
The begin transaction looks like this:
public void beginNewTransaction() {
if (getCurrentSession().isConnected()) {
logger.info("Session is not connected");
getCurrentSession().reconnect();
if (getCurrentSession().isConnected()) {
logger.info("Now connected!");
} else {
logger.info("STill not connected---------------");
}
} else if (getCurrentSession().isOpen()) {
logger.info("Session is not open");
}
getCurrentSession().beginTransaction();
logger.info("BEGINNING TRANSAACTION - " + getCurrentSession().getTransaction().isActive());
}
The threads are using TransactionTemplates since my buyerClient object is not managed by spring (long involved requirements).
Here is that code:
#SuppressWarnings("unchecked")
private T processPing(Lead lead) {
Date now = new Date();
if (logger.isDebugEnabled()) {
logger.debug("BEGIN PINGING BUYER " + getLeadBuyer().getName() + " for leadId:" + lead.getId() + " time:" + DateUtils.formatDate(now, "HH:mm:ss:Z"));
}
Object leadStatus = transaction(lead);
if (logger.isDebugEnabled()) {
logger.debug("PING COMPLETE FOR BUYER " + getLeadBuyer().getName() + " for leadId:" + lead.getId() + " time:" + DateUtils.formatDate(now, "HH:mm:ss:Z"));
}
return (T) leadStatus;
}
public T transaction(final Lead incomingLead) {
final T pingLeadStatus = this.constructPingLeadStatusInstance();
Lead lead = leadDao.fetchLeadById(incomingLead.getId());
T object = transactionTemplate.execute(new TransactionCallback<T>() {
#Override
public T doInTransaction(TransactionStatus status) {
Date startTime = null, endTime = null;
logger.info("incomingLead obfid:" + incomingLead.getObfuscatedAffiliateId() + " affiliateId:" + incomingLead.getAffiliate().getId());
T leadStatus = null;
if (leadStatus == null) {
leadStatus = filterLead(incomingLead);
}
if (leadStatus == null) {
leadStatus = pingLeadStatus;
leadStatus.setLead(incomingLead);
...LOTS OF CODE
}
if (logger.isDebugEnabled())
logger.debug("RETURNING LEADSTATUS FOR BUYER " + getLeadBuyer().getName() + " for leadId:" + incomingLead.getId() + " time:" + DateUtils.formatDate(new Date(), "HH:mm:ss:Z"));
return leadStatus;
}
});
if (logger.isDebugEnabled()) {
logger.debug("Transaction complete for buyer:" + getLeadBuyer().getName() + " leadId:" + incomingLead.getId() + " time:" + DateUtils.formatDate(new Date(), "HH:mm:ss:Z"));
}
return object;
}
However, when we begin our new transaction we get this error:
org.springframework.transaction.TransactionSystemException: Could not commit Hibernate transaction; nested exception is org.hibernate.TransactionException: Transaction not successfully started
at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:660)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:754)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:120)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
My Goal
My goal is to be able to have that entity fully initalized on the other side or Does anyone have any ideas on how I can commit the data to the database so the thread can have a fully populated object. Or, have a way to query for a full object?
Thanks I know this is really involved. I apologize if I haven't been clear enough.
I have tried
Hibernate.initialize()
saveWithFlush()
update(lead)
I didn't follow everything - you can try one of this to workaround the issue that you get about the same object being associated with two sessions.
// do this in the main thread to detach the object
// from the current session
// if it has associations that also need to be handled the cascade=evict should
// be specified. Other option is to do flush & clear on the session.
session.evict(object);
// pass the object to the other thread
// in the other thread - use merge
session.merge(object)
Second approach - create a deep copy of the object and pass the copy. This can be easily achieved if your entity classes are serializable - just serialize the object and deserialize.
Thanks #gkamal for your help.
For everyone living in posterity. The answer to my dilemma was a left over call to hibernateTemplate instead of getCurrentSession(). I made the move about a year and a half ago and for some reason missed a few key places. This was generating a second transaction. After that I was able to use #gkamal suggestion and evict the object and grab it again.
This post helped me figure it out:
http://forum.springsource.org/showthread.php?26782-Illegal-attempt-to-associate-a-collection-with-two-open-sessions