In start, it works fine, but after a certain time (1-2 hours) it crashes with the following exception in server logs.
ERROR 1 --- [-ignite-server%] : JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.i.IgniteDeploymentCheckedException: Failed to obtain deployment for class: com.event.audit.AuditEventListener$$Lambda$1484/0x0000000800a7ec40]]
public static void remoteListener(Ignite ignite) {
// This optional local callback is called for each event notification
// that passed remote predicate listener.
IgniteBiPredicate<UUID, CacheEvent> locLsnr = new IgniteBiPredicate<UUID, CacheEvent>() {
#Override public boolean apply(UUID nodeId, CacheEvent evt) {
System.out.println("Listener caught an event");
//--- My custom code to persists the event in another cache
};
IgnitePredicate<CacheEvent> remoteListener = cacheEvent -> {
return true;
};
// Register event listeners on all nodes to listen for task events.
UUID lsnrId = ignite.events(ignite.cluster()).remoteListen(locLsnr, remoteListener, EVT_CACHE_OBJECT_PUT, EVT_CACHE_OBJECT_REMOVED);
}
}
As I understand you, you try to perform cache operations in event listener:
//--- My custom code to persists the event in another cache
Event listeners are called under the locks and this is bad idea to make any other cache operations in listeners. I suppose it could be the root cause of your issue.
Try to change you design, for example you can add your caught event in a queue and then read this queue in another thread and save the data in another cache.
Related
I'm new to mass transit and have a question regarding how I should solve a failure to consume a message. Given the below code I am consuming INotificationRequestContract's. As you can see the code will break and not complete.
public class NotificationConsumerWorker : IConsumer<INotificationRequestContract>
{
private readonly ILogger<NotificationConsumerWorker> _logger;
private readonly INotificationCreator _notificationCreator;
public NotificationConsumerWorker(ILogger<NotificationConsumerWorker> logger, INotificationCreator notificationCreator)
{
_logger = logger;
_notificationCreator = notificationCreator;
}
public Task Consume(ConsumeContext<INotificationRequestContract> context)
{
try
{
throw new Exception("Horrible error");
}
catch (Exception e)
{
// >>>>> insert code here to put message back for later consumption. <<<<<
_logger.LogError(e, "Failed to consume message");
throw;
}
}
}
How do I best handle a scenario such as this where the consumption fails? In my specific case this is likely to occur if a required external service is unavailable.
I can see two solutions.
If there is a way to put the message back, or cancel the consumption so that it will be tried again.
I could store it locally in a database and create my own re-try method to wrap this (but would prefer not to for sake of simplicity).
The exceptions section of the documentation provides sufficient guidance for dealing with consumer exceptions.
There are two retry approaches, which can be used in combination:
Message Retry, which waits while the message is locked, in-process, for the next retry. Therefore, these should be short, to deal with transient issues.
Message Redelivery, which delays the message using either the broker delayed delivery, or a message scheduler, so that it is redelivered to the receive endpoint at some point in the future.
Once all retry/redelivery attempts are exhausted, the message is moved to the _error queue.
I am using Masstransit with RabbitMQ. As part of some deployment procedure, At some point in time I need my service to disconnect and stop receiving any messages.
Assuming that I won't need the bus until the next restart of the service, will it be Ok to use bus.StopAsync()?
Is there a way to get list of end points and then remove them from listining ?
You should StopAsync the bus, and then when ready, call StartAsync to bring it back up (or start it at the next service restart).
To stop receiving messages without stopping the buss I needed a solution that will avoid the consume message pipeline from consuming any type of message. I tried with observers but unsuccessfully. My solution came up with custom consuming message filter.
The filter part looks like this
public class ComsumersBlockingFilter<T> :
IFilter<ConsumeContext<T>>
where T : class
{
public void Probe(ProbeContext context)
{
var scope = context.CreateFilterScope("messageFilter");
}
public async Task Send(ConsumeContext<T> context, IPipe<ConsumeContext<T>> next)
{
// Check if the service is degraded (true for this demo)
var isServiceDegraded = true;
if (isServiceDegraded)
{
//Suspend the message for 5 seconds
await Task.Delay(TimeSpan.FromMilliseconds(5000), context.CancellationToken);
if (!context.CancellationToken.IsCancellationRequested)
{
//republish the message
await context.Publish(context.Message);
Console.WriteLine($"Message {context.MessageId} has been republished");
}
// NotifyConsumed to avoid skipped message
await context.NotifyConsumed(TimeSpan.Zero, "messageFilter");
}
else
{
//Next filter in the pipe is called
await next.Send(context);
}
}
}
The main idea is to delay with cancellation token and the republish the message. After that call contect.NotifyConsumed to avoid the next pipeline filters and return normally.
I have a code where event is published in a method annotated with Spring #Transactional annotation.
#Override
#Transactional
public Task updateStatus(Integer taskId, ExecutionStatus newStatus) {
Task task = Task.builder().executionStatus(newStatus).build();
return updateStatusInternal(taskId, rteWithMetadata);
}
private TaskExecution updateStatusInternal(Integer taskId,
Task newStatus) {
Task task = taskService.findById(taskId);
TaskExecution te = task.getFirstExecution();
TaskExecution.ExecutionStatus oldStatus = te.getExecutionStatus();
TaskExecution.ExecutionStatus newStatus = newStatus.getExecutionStatus();
log.info(
"Task Execution status changed. Task id={}, from={}, to={}. Manual override : {}",
task.getId(), oldStatus, newStatus,
newStatus.isManualOverrideInitiated());
te.setExecutionStatus(newStatus);
if (te.getExecutionStatus() == ExecutionStatus.COMPLETED
|| te.getExecutionStatus() == ExecutionStatus.FAILED) {
te.setEndDate(DateTimeHelper.getUtcNow());
if (rte.isManualOverrideInitiated()) {
rte.setManualOverrideEndDate(DateTimeHelper.getUtcNow());
}
}
publisher.publishEvent(TaskStatusChanged.of(task, oldStatus, newStatus));
log.info("Published TaskStatusChanged event. task Id={}", task.getId());
// Send STOMP message
final Object payload = StompMessageHelper.getTaskExecutionUpdateMessage(task);
messageTemplate.convertAndSend(taskDestination(task), payload);
log.info("STOMP message for task status update sent. task Id={}",
task.getId());
return te;
}
There is a corresponding listener method for the application event which is annotated with #TransactionalEventListener.
#Async("changeEventExecutor")
#TransactionalEventListener(phase=TransactionPhase.AFTER_COMMIT)
public void taskStatusChanged(final TaskStatusChanged e) {
log.info("taskStatusChanged called");
}
Problem is listener is not fired on one of our production boxes. It works fine consistently on local dev environment but fails consistently in production.
Did somebody face this issue earlier? Only solution I can think of is to manually fire the application event.
Note: I have checked the existing similar posting. My scenario does not match with any of the existing posting.
The only thing I can think of comes from Spring's javadoc:
If the event is not published within the boundaries of a managed
transaction, the event is discarded unless the fallbackExecution()
flag is explicitly set. If a transaction is running, the event is
processed according to its TransactionPhase.
Could there be no transaction running? I assume your code sample isn't complete, so perhaps the transaction is being rolled-back when the event is fired or something along those lines.
In any case, you could try with the following (I know you are referring to a production box, so I'm not sure what are your options in trying things out):
#TransactionalEventListener(fallbackExecution=true, phase=TransactionPhase.AFTER_COMMIT)
Here i'm using following code for sending message. I added the callback listener to know message status but its not printing any log message.
gamesClient.sendReliableRealTimeMessage(new RealTimeReliableMessageSentListener() {
#Override
public void onRealTimeMessageSent(int statusCode, int tokenId, String recipientParticipantId) {
// TODO Auto-generated method stub
switch (statusCode) {
case GamesClient.STATUS_OK:
Log.e("status", "STATUS_OK");
break;
case GamesClient.STATUS_REAL_TIME_MESSAGE_SEND_FAILED:
Log.e("status", "STATUS_REAL_TIME_MESSAGE_SEND_FAILED");
break;
case GamesClient.STATUS_REAL_TIME_ROOM_NOT_JOINED:
Log.e("status", "STATUS_REAL_TIME_ROOM_NOT_JOINED");
break;
}
}
}, msgBuf, roomId, p.getParticipantId());
Don't use anonymous listeners like that. The API uses weak references to listeners, so it often happens that these listeners will get garbage-collected before they are called. Please try again using a non-anonymous listener, that is, a listener that you hold a reference to. One easy way to do that is make the Activity the listener (i.e. add "implements RealTimeReliableMessageSentListener" to the Activity class).
Another option is to hold an explicit reference to the listener as a member variable in your Activity.
I am trying to make a simple server thread in QT to accept a connection, however although the server is listening (I can connect with my test app) I can't get the newConnection() signal to be acted on.
Any help as to what I'm missing here would be much appreciated!
class CServerThread : public QThread
{
Q_OBJECT
protected:
void run();
private:
QTcpServer* server;
public slots:
void AcceptConnection();
};
void CServerThread::run()
{
server = new QTcpServer;
QObject::connect(server, SIGNAL(newConnection()), this, SLOT(AcceptConnection()));
server->listen(QHostAddress::Any, 1000); // Any port in a storm
exec(); // Start event loop
}
void CServerThread::AcceptConnection()
{
OutputDebugStringA("\n***** INCOMING CONNECTION"); // This is never called!
}
First of all I can say that your server lives in new thread while CServerThread instance lives in another thread (in the thread this instance was created). Signal/slot connection you are creating is inderect and uses thread save event delivery between events loops of two different threads. It actually can cause such problem if thread where you creating CServerThread doesn't have Qt event loop running.
I suggest you to create some MyServer class which creates QTcpServer and calls listen and connects QTcpServer::newConnection() signal to its own slot. Then rewrite your server thread run method to something like this:
void CServerThread::run() {
server = new MyServer(host,port);
exec(); // Start event loop
}
In this approach both QTcpServer and newConnection processing object lives in the same thread. Such situation is easier to handle.
I have one really simple working example:
Header: http://qremotesignal.googlecode.com/svn/tags/1.0.0/doc/html/hello_2server_2server_8h-example.html
Source: http://qremotesignal.googlecode.com/svn/tags/1.0.0/doc/html/hello_2server_2server_8cpp-example.html