I have extended EventNotiferSupport, and set the isEnable() to respond True for all events. I have a notify() that logs what events I receive and the corresponding Exchange ID for the event.
I have added my ExchangeMessageNotifier with this.context.getManagementStrategy().addEventNotifier(this.exchangeMessageNotifier);
I run my program under basically no load, sending 1 message at a time 1 second delay between messages into Camel to send out. Everything works the way I expect. I receive my events everything looks good.
I decrease the delay between messages to 0 milliseconds, and I find that 1 out of approximately 20 messages I fail to receive one of the Events, (Often the Completed event).
Add a second thread sending at the same rate and I don't get any events for any messages.
What am I missing? I've done searches and I don't find anything that I need to do differently. Is there something I am missing?
I am using Apache Camel 2.16.3, and moved to 2.18.1 still see the same behavior.
Well found my own answer. Part of the fun of inheriting code without any informaiton.
In your implementation of the EventNotifierSupport you need to override the doStart() method and configure the EventNotifierSupport for what events you wish to receive.
protected void doStart() throws Exception {
// filter out unwanted events
setIgnoreCamelContextEvents(true);
setIgnoreServiceEvents(true);
setIgnoreRouteEvents(true);
setIgnoreExchangeCreatedEvent(true);
setIgnoreExchangeCompletedEvent(false);
setIgnoreExchangeFailedEvents(true);
setIgnoreExchangeRedeliveryEvents(true);
setIgnoreExchangeSentEvents(false);
}
This is in addition to doing the following:
public boolean isEnabled(EventObject event) {
return true;
}
Which enables you to determine if you want a particular event, out of the selected groups you had set in the doStart().
Once these changes were in I was receiving consistent events.
Related
Case
Clients are ReplyingKafkaTemplate instances.
Server is a ConcurrentMessageListenerContainer created using #KafkaListener and #SendTo annotations on a method.
ContainerFactory uses ContainerStoppingErrorHandler.
Request topic has only 1 partition.
Group ids are static. eg. test-consumer-group.
Requests are sent with timeouts.
Due to an exception thrown, server goes down
but the client keeps dispatching requests which queue up on the
request topic.
Current Behavior
When the server comes back up it continues processing old requests which would have timed out.
Desired Behavior
Instead, it would be better to continue with the last message; thereby skipping past even unprocessed messages as corresponding requests would timeout and retry.
Questions
What is the recommended approach to achieve this?
From the little that I understand, it looks like I'll have to manually set the initial offset. What's the simplest way to implement this?
Your #KafkaListener class must extends AbstractConsumerSeekAware and do something like this:
#Override
public void onPartitionsAssigned(Map<TopicPartition, Long> assignments, ConsumerSeekCallback callback) {
super.onPartitionsAssigned(assignments, callback);
callback.seekToEnd(assignments.keySet());
}
So, every time when your consumer joins the group it is going to seek all the assigned partitions to the end skipping all the old records.
I'm reading docs but I'm not sure if this is possible on spring-cloud-stream using binder for kinesis.
I want to wait consuming messages from stream with some delay or configuration or by positive condition.
For example, I want wait 30 minutes after consumer process the message.
First aproximation is use condition with SPEL based on header message and current time, but the condition is created on startup. Then new Date is always the same.
I know that condition in below code is invalid.
#StreamListener(StreamProcessor.MY_STREAM, condition="#{headers['creation-date'] + 30minutes < new java.util.Date().getTime()}")
public void checkOut(Message<String> myMessage) {
//Do something
}
Do you know if is this possible without sleeping threads?
All you need is use Polled Consumer, this way you have full control over frequency, acks etc.
This is a bit of a conceptual question, so let me know if it's off topic.
I'm looking at writing yet another library to process messages off a queue - in this case an Azure storage queue. It's pretty easy to create an observable and throw a message into it every time a message is available.
However, there's a snag here that I'm not sure how to handle. The issue is this: when you're done processing the message, you need to call an API on the storage queue to actually delete the message. Otherwise the visibility timeout will expire and the message will reappear to be dequeued again.
As an example, here's how this loop looks in C#:
public event EventHandler<string> OnMessage;
public void Run()
{
while(true)
{
// Read message
var message = queue.GetMessage();
if (message != null)
{
// Run any handlers
OnMessage?.Invoke(this, message.AsString);
// Delete off queue when done
queue.DeleteMessage(message);
}
else
{
Thread.Sleep(2500);
}
}
}
The important thing here is that we read the message, trigger any registered event handlers to do things, then delete the message after the handlers are done. I've omitted error handling here, but in general if the handler fails we should NOT delete the message, but instead let it return to visibility automatically and get redelivered later.
How do you handle this kind of thing using Rx? Ideally I'd like to expose the observable for anyone to subscribe to. But I need to do stuff at the end of processing for that message, whatever the "end" happens to mean here.
I can think of a couple of possible solutions, but I don't really like any of them. One would be to have the library call a function supplied by the consumer, that takes in the source observable, hooks up whatever it wants, then returns a new observable that the library can then subscribe on to do the final cleanup. But that's pretty limiting, as consumers basically only have one shot to hook up to the messages, which seems pretty limiting.
I guess I could put the call to delete the message after the call to onNext, but then I don't know if the processing succeeded or failed unless there's some sort of back channel in that api I don't know about?
Any ideas/suggestions/previous experience here?
Try having a play with this:
IObservable<int> source =
Observable
.Range(0, 3)
.Select(x =>
Observable
.Using(
() => Disposable.Create(() => Console.WriteLine($"Removing {x}")),
d => Observable.Return(x)))
.Merge();
source
.Subscribe(x => Console.WriteLine($"Processing {x}"));
It produces:
Processing 0
Removing 0
Processing 1
Removing 1
Processing 2
Removing 2
I've been working with the Message Hub sample code found at this link: https://github.com/ibm-messaging/message-hub-samples
In particular, I've been trying to increase the throughput of the producer with the Kafka Java console example. I noticed the documentation in this snippet of code:
// Synchronously wait for a response from Message Hub / Kafka on every message produced.
// For high throughput the future should be handled asynchronously.
RecordMetadata recordMetadata = future.get(5000, TimeUnit.MILLISECONDS);
producedMessages++;
I've already turned off the thread sleep found later in the code which also helped increase the throughput, but I was hoping I could get some help on implementing the future asynchronously in this block. Thanks in advance!
you have two basic options for handling the outcome of a produce request asynchronously
1) use the overloaded send with a completion callback argument, which will be invoked asynchronously:
public Future<RecordMetadata> send(ProducerRecord<K, V> record, Callback callback);
if using the callback you may ignore the future.
2) pass the Future to some other thread you have created, and have it inspect the future for completion, while leaving the thread that calls send free to carry on.
I'm using ReactiveUI's RegisterScheduler like this:
MessageBus.Current.RegisterScheduler<string>(RxApp.MainThreadScheduler, someContract);
I want some of the message subscribers (who register this message) handle it in UI thread, but at the same time, some other subscribers handle it in backgroud thread.
Here is the use case: Some of the message's subscribers will handle UI related data which must happen in UI thread. But I want to reduce the UI thread usage as minimum as possible, so I would like to have other operations (not related to UI) happen in background thread. Is it possible for ReactiveUI?
My current solution is to use 2 different contract string, which in fact can be considered 2 different messages. It works, but both messages should be send out together, which makes me feel reduntent.
Any ideas?
Fei
Perhaps ignore the scheduler, or set maybe set your default appropriately (the default is CurrentThreadScheduler) and just use ObserveOn to define which scheduler the handler will be executed with when you need to.
To handle on the UI thread:
MessageBus.Current.Listen<string>()
.ObserveOn(RxApp.MainThreadScheduler)
.Subscribe(message => { /* ... */ });
To handle on a thread pool thread:
MessageBus.Current.Listen<string>()
.ObserveOn(RxApp.TaskpoolScheduler)
.Subscribe(message => { /* ... */ });