I've started to explore NetMQ 3.3.0.11 and ran into an issue with the use of Poller.
I try to achieve that the poller polls for about 1s and then stops and allows something else to be done before it resumes polling for 1s and so on.
I have the following code:
var poller = new Poller (client) { PollTimeout = 10 };
while (true)
{
for (var poll = 0; poll < 100; poll++)
{
poller.PollOnce ();
}
do_something;
}
The problem I'm facing is that during that polling period the Client.ReceiveReady event is not fired even though a message is ready to be picked up. And a InvalidOperationException stating Poller is started is raised.
Any idea what I'm doing wrong?
First try to work with version 3.3.0.12-rc1, it fixes a lot of issues, probably also the one you are suffering from.
Also regarding the do_something, I suggest working with NetMQTimer instead of PollOnce (use PollTillCanceled instead). You can also use NetMQScheduler for the do something stuff.
Related
I am working on a project to read from our existing ElasticSearch instance and produce messages in Pulsar. If I do this in a highly multithreaded way without any explicit synchronization, I get many occurances of the following log line:
Message with sequence id X might be a duplicate but cannot be determined at this time.
That is produced from this line of code in the Pulsar Java client:
https://github.com/apache/pulsar/blob/a4c3034f52f857ae0f4daf5d366ea9e578133bc2/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java#L653
When I add a synchronized block to my method, synchronizing on the pulsar template, the error disappears, but my publish rate drops substantially.
Here is the current working implementation of my method that sends Protobuf messages to Pulsar:
public <T extends GeneratedMessageV3> CompletableFuture<MessageId> persist(T o) {
var descriptor = o.getDescriptorForType();
PulsarPersistTopicSettings settings = pulsarPersistConfig.getSettings(descriptor);
MessageBuilder<T> messageBuilder = Optional.ofNullable(pulsarPersistConfig.getMessageBuilder(descriptor))
.orElse(DefaultMessageBuilder.DEFAULT_MESSAGE_BUILDER);
Optional<ProducerBuilderCustomizer<T>> producerBuilderCustomizerOpt =
Optional.ofNullable(pulsarPersistConfig.getProducerBuilder(descriptor));
PulsarOperations.SendMessageBuilder<T> sendMessageBuilder;
sendMessageBuilder = pulsarTemplate.newMessage(o)
.withSchema(Schema.PROTOBUF_NATIVE(o.getClass()))
.withTopic(settings.getTopic());
producerBuilderCustomizerOpt.ifPresent(sendMessageBuilder::withProducerCustomizer);
sendMessageBuilder.withMessageCustomizer(mb -> messageBuilder.applyMessageBuilderKeys(o, mb));
synchronized (pulsarTemplate) {
try {
return sendMessageBuilder.sendAsync();
} catch (PulsarClientException re) {
throw new PulsarPersistException(re);
}
}
}
The original version of the above method did not have the synchronized(pulsarTemplate) { ... } block. It performed faster, but generated a lot of logs about duplicate messages, which I knew to be incorrect. Adding the synchronized block got rid of the log messages, but slowed down publishing.
What are the best practices for multithreaded access to the PulsarTemplate? Is there a better way to achieve very high throughput message publishing?
Should I look at using the reactive client instead?
EDIT: I've updated the code block to show the minimum synchronization necessary to avoid the log lines, which is just synchronizing during the .sendAsync(...) call.
Your usage w/o the synchronized should work. I will look into that though to see if I see anything else going on. In the meantime, it would be great to give the Reactive client a try.
This issue was initially tracked here, and the final resolution was that it was an issue that has been resolved in Pulsar 2.11.
Please try updating the Pulsar 2.11.
I have an IIB message flow that runs for a few hours each evening, using a java loop to perform some actions once per minute.
During that time, if the multi-instance broker that the flow is running on fails over, the failover hangs until this message flow ends its processing (potentially hours later).
Is there any kind of hook I can use in Java to say "if the broker is stopping or failing over, then cancel this processing to let it happen"?
Edit
I have now tried the following code as a test, but even when a request is made to stop the execution group/flow, the booleans all remain as true
Boolean egIsRunning = true;
Boolean aIsRunning = true;
Boolean msgFlowIsRunning = true;
while (egIsRunning && aIsRunning && msgFlowIsRunning)
{
Thread.sleep(1000);
ExecutionGroupProxy e = ExecutionGroupProxy.getLocalInstance();
egIsRunning = e.isRunning();
ApplicationProxy a = e.getApplicationByName("SANDBOX.APP");
aIsRunning = a.isRunning();
MessageFlowProxy m = a.getMessageFlowByName("SANDBOX_MSGFLOW");
msgFlowIsRunning = m.isRunning();
}
So, I don't think the Integration API is going to help here? Or is there some ".isTryingToStop" method that I'm missing?
I recommend to look at the Integration API.
From within your Java Compute Node you can connect with BrokerProxy.getLocalInstance() (don't forget to disconnect(), otherwise you will run out of memory eventually). Then I would try MessageFlowProxy.isRunning() as an exit criteria for your loop.
Should isRunning not work, there are other options like AdministeredObjectListener to figure out what is happening with your flow.
I'm using ktor for server side development with websockets.
Documentations shows us this example of using incoming channel:
for (frame in incoming.mapNotNull { it as? Frame.Text }) {
// some
}
But mapNotNull is marked as deprecated in favor of Flow. How should I use this API and what problems could be there? For example, the Flow is a cold stream. It means that the producer function will be called on each collect. How does it work in context of websocket. Will it be reopened on second collect call, or maybe old messages will be delivered once after the next collect? How can I collect N messages, then stop collecting, then collect again?
Thanks in advance :)
How should I use this API and what problems could be there?
What I am using and what I have seen in one of the examples somewhere in the docs is the consumeAsFlow() method called on ReceiveChannel. Here is the entire snippet:
webSocket("/websocket") { //this: DefaultWebSocketServerSession
incoming
.consumeAsFlow()
.map { receive(it) }
.collect()
}
Haven't seen major issues with this approach. One thing you should be aware of (but that goes for the non-flow approach as well) is that if you throw inside your flow, then it will break the WebSocket connection, which is usually not something you'd like to do. It might be worth considering wrapping the entire thing in a try-catch.
Will it be reopened on second collect call, or maybe old messages will be delivered once after the next collect?
You open the websocket before you even start consuming the messages from the flow. You can see that inside webSocket() {} you are in the context of DefaultWebSocketServerSession. This is your connection management. Inside your flow you are simply receiving messages one by one as they arrive (after the connection has been established). If the connection breaks, then you're out of the flow. It needs to be re-established before you can process your messages. This establishing bit is done by the Route.webSocket() method. I do recommend taking a look at its Javadoc.
If you wish to add some clean up after the connection is closed you can add a finally block like so:
webSocket("/chat") {
try {
incoming
.consumeAsFlow()
.map { receive(it, client) }
.collect()
} finally {
// cleanup
}
}
In short: collect is called once per received message. If there is no connection (or it was broken) then collect won't be called.
How can I collect N messages, then stop collecting, then collect again?
What is the use case for this? I don't think you should be doing this with any flow. You can of course take(n) items from a flow, but you won't be able to take any more from it again.
I am writing an application for my own purposes that aims to get play pause events no matter what is going on in the system. I have gotten this much working
let commandCenter = MPRemoteCommandCenter.shared()
commandCenter.togglePlayPauseCommand.isEnabled = true
commandCenter.togglePlayPauseCommand.addTarget { (MPRemoteCommandEvent) -> MPRemoteCommandHandlerStatus in
print("Play Pause Command")
return .success
}
commandCenter.nextTrackCommand.isEnabled = true
commandCenter.nextTrackCommand.addTarget { (MPRemoteCommandEvent) -> MPRemoteCommandHandlerStatus in
print("NextTrackCommand")
return .success
}
commandCenter.previousTrackCommand.isEnabled = true
commandCenter.previousTrackCommand.addTarget { (MPRemoteCommandEvent) -> MPRemoteCommandHandlerStatus in
print("previousTrackCommand")
return .success
}
commandCenter.playCommand.isEnabled = true
commandCenter.playCommand.addTarget { (MPRemoteCommandEvent) -> MPRemoteCommandHandlerStatus in
print("playCommand")
return .success
}
MPNowPlayingInfoCenter.default().playbackState = .playing
Most of those methods are there because apparently you will not get any notifications without having nextTrackCommand or previousTrackCommand or playCommand implemented.
Anyways my one issue is that as soon as you open another application that uses audio these event handlers stop getting called and I cant find a way to detect and fix this.
I would normally try doing AVAudioSession things to state this as a background application however that does not seem to work. Any ideas on how I can get playpause events no matter what state the system is in?
I would like to be able to always listen for these events OR get an indication of when someone else has taken control of the audio? Perhaps even be able to re-subscribe to these play pause events.
There's an internal queue in the system which contains all the audio event subscribers. Other applications get on top of it when you start using them.
I would like to be able to always listen for these events
There's no API for that but there's a dirty workaround. If I understand your issue correctly, this snippet:
MPNowPlayingInfoCenter.default().playbackState = .paused
MPNowPlayingInfoCenter.default().playbackState = .playing
must do the trick for you if you run it in a loop somewhere in your application.
Note that this is not 100% reliable because:
If an event is generated before two subsequent playbackState state changes right after you've switched to a different application, it would still be catched by the application in the active window;
If another application is doing the same thing, there would be a constant race condition in the queue, with unpredictable outcome.
References:
Documentation for playbackState is here;
See also a similar question;
See also a bug report for mpv with a similar
issue (a pre-MPRemoteCommandCenter one, but still very valuable)
OR get an indication of when someone else has taken control of the audio
As far as I know there's no public API for this in macOS.
This is a bit of a conceptual question, so let me know if it's off topic.
I'm looking at writing yet another library to process messages off a queue - in this case an Azure storage queue. It's pretty easy to create an observable and throw a message into it every time a message is available.
However, there's a snag here that I'm not sure how to handle. The issue is this: when you're done processing the message, you need to call an API on the storage queue to actually delete the message. Otherwise the visibility timeout will expire and the message will reappear to be dequeued again.
As an example, here's how this loop looks in C#:
public event EventHandler<string> OnMessage;
public void Run()
{
while(true)
{
// Read message
var message = queue.GetMessage();
if (message != null)
{
// Run any handlers
OnMessage?.Invoke(this, message.AsString);
// Delete off queue when done
queue.DeleteMessage(message);
}
else
{
Thread.Sleep(2500);
}
}
}
The important thing here is that we read the message, trigger any registered event handlers to do things, then delete the message after the handlers are done. I've omitted error handling here, but in general if the handler fails we should NOT delete the message, but instead let it return to visibility automatically and get redelivered later.
How do you handle this kind of thing using Rx? Ideally I'd like to expose the observable for anyone to subscribe to. But I need to do stuff at the end of processing for that message, whatever the "end" happens to mean here.
I can think of a couple of possible solutions, but I don't really like any of them. One would be to have the library call a function supplied by the consumer, that takes in the source observable, hooks up whatever it wants, then returns a new observable that the library can then subscribe on to do the final cleanup. But that's pretty limiting, as consumers basically only have one shot to hook up to the messages, which seems pretty limiting.
I guess I could put the call to delete the message after the call to onNext, but then I don't know if the processing succeeded or failed unless there's some sort of back channel in that api I don't know about?
Any ideas/suggestions/previous experience here?
Try having a play with this:
IObservable<int> source =
Observable
.Range(0, 3)
.Select(x =>
Observable
.Using(
() => Disposable.Create(() => Console.WriteLine($"Removing {x}")),
d => Observable.Return(x)))
.Merge();
source
.Subscribe(x => Console.WriteLine($"Processing {x}"));
It produces:
Processing 0
Removing 0
Processing 1
Removing 1
Processing 2
Removing 2