Is it possible (and if yes, how), to publish a stream of entity updates directly (without any additional topic) as Source returned from a service like this?
public ServiceCall<NotUsed, Source<EntityPublicEvent, ?>> entityUpdates()
What I'm trying to achieve is to consume those events in Angular 5 over websocket.
#Override
public ServiceCall<String, Source<EntityPublicEvent, ?>> getUpdateNotifications() {
return request -> CompletableFuture.completedFuture(updatesTopic.subscriber().map(this::convertEvent));
}
I can see in Angular gets connected and the service is also called to return the Source but no updates are sent as messages to Angular.
There is also this line appearing in logs, which I suppose could be the reason why I cant get any messages in Angular:
[info] akka.actor.RepointableActorRef
[sourceThread=currency-impl-application-akka.actor.default-dispatcher-3,
akkaTimestamp=06:59:55.778UTC,
akkaSource=akka://currency-impl-application/system/distributed
PubSubMediator, sourceActorSystem=currency-impl-application] - Message
[tcs.currency.impl.CurrencyEvent$Created] without sender to
Actor[akka://currency-impl-application/system/distributedPubSubMediator#128487187]
was not delivered. [1] dead letters encountered. This logging can be
turned off or adjusted with configuration settings
'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
Solved, in this case my Angular component was creating the connection but not listening for messages. I was trying to start with the code provided here https://tutorialedge.net/typescript/angular/angular-websockets-tutorial/, which is an echo service and not just a simple listener.
With this simple code I'm now able to receive the update messages:
let subject = Observable.webSocket('ws://localhost:9000/ws/api/currency/updates');
subject
.retry()
.subscribe(
(msg) => console.log('message received: ' + msg),
(err) => console.log(err),
() => console.log('complete')
);
Related
Can some one help how to configure NATS jet stream subscription in spring boot asynchronously example: looking for an equivalent annotation like #kafkalistener for Nats jetstream
I am able to pull the messages using endpoint but however when tried to pull messages using pushSubscription dispatcherhandler is not invoked. Need to know how to make the listener to be active and consume messages immediately once the messages are published to the subject.
Any insights /examples regarding this will be helpful, thanks in advance.
I don't know what is your JetStream retention policy, neither the way you want to subscribe. But I have sample code for WorkQueuePolicy push subscription, wish this will help you.
public static void subscribe(String streamName, String subjectKey,
String queueName, IMessageHandler iMessageHandler) throws IOException,
InterruptedException, JetStreamApiException {
long s = System.currentTimeMillis();
Connection nc = Nats.connect(options);
long e = System.currentTimeMillis();
logger.info("Nats Connect in " + (e - s) + " ms");
JetStream js = nc.jetStream();
Dispatcher disp = nc.createDispatcher();
MessageHandler handler = (msg) -> {
try {
iMessageHandler.onMessageReceived(msg);
} catch (Exception exc) {
msg.nak();
}
};
ConsumerConfiguration cc = ConsumerConfiguration.builder()
.durable(queueName)
.deliverGroup(queueName)
.maxDeliver(3)
.ackWait(Duration.ofMinutes(2))
.build();
PushSubscribeOptions so = PushSubscribeOptions.builder()
.stream(streamName)
.configuration(cc)
.build();
js.subscribe(subjectKey, disp, handler, false, so);
System.out.println("NatsUtil: " + durableName + "subscribe");
}
IMessageHandler is my custom interface to handle nats.io received messages.
First, configure the NATS connection. Here you will specify all your connection details like server address(es), authentication options, connection-level callbacks etc.
Connection natsConnection = Nats.connect(
new Options.Builder()
.server("nats://localhost:4222")
.connectionListener((connection, eventType) -> {})
.errorListener(new ErrorListener(){})
.build());
Then construct a JetStream instance
JetStream jetStream = natsConnection.jetStream();
Now you can subscribe to subjects. Note that JetStream consumers can be durable or ephemeral, can work according to push or pull logic. Please refer to NATS documentation (https://docs.nats.io/nats-concepts/jetstream/consumers) to make the appropriate choice for your specific use case. The following example constructs a durable push consumer:
//Subscribe to a subject.
String subject = "my-subject";
//queues are analogous to Kafka consumer groups, i.e. consumers belonging
//to the same queue (or, better to say, reading the same queue) will get
//only one instance of each message from the corresponding subject
//and only one of those consumers will be chosen to process the message
String queueName = "my-queue";
//Choosing delivery policy is analogous to setting the current offset
//in a partition for a consumer or consumer group in Kafka.
DeliverPolicy deliverPolicy = DeliverPolicy.New;
PushSubscribeOptions subscribeOptions = ConsumerConfiguration.builder()
.durable(queueName)
.deliverGroup(queueName)
.deliverPolicy(deliverPolicy)
.buildPushSubscribeOptions();
Subscription subscription = jetStream.subscribe(
subject,
queueName,
natsConnection.createDispatcher(),
natsMessage -> {
//This callback will be called for incoming messages
//asynchronously. Every subscription configured this
//way will be backed by its own thread, that will be
//used to call this callback.
},
true, //true if you want received messages to be acknowledged
//automatically, otherwise you will have to call
//natsMessage.ack() manually in the above callback function
subscribeOptions);
As for the declarative API (i.e. some form of #NatsListener annotation analogous to #KafkaListener from Spring for Apache Kafka project), there is none available out of the box in Spring. If you feel like you absolutely need it, you can write one yourself, if you are familiar with Spring BeanPostProcessor-s or other extension mechanism that can help to do that. Alternatively you can refer to 3rd party libs, it looks like a bunch of people (including myself) felt a bit uncomfortable when switching from Kafka to NATS, so they tried to bring the usual way of doing things with them from the Kafka world. Some examples can be found on github:
https://github.com/linux-china/nats-spring-boot-starter,
https://github.com/dstrelec/nats
https://github.com/amalnev/declarative-nats-listeners
There may be others.
When a subscriber of a topic throws an exception non-handled message lands in {subscribername}_error queue.
Given the example:
const string subsriberName = "AnotherSubscriber";
cfg.SubscriptionEndpoint<AnotherThingHappened>(host, subsriberName, configurator =>
{
configurator.Handler<AnotherThingHappened>(context =>
{
Console.WriteLine(context.Message.AnotherThingType);
if (Random.NextDouble() < 0.1)
{
throw new Exception("Oups, I failed :(");
}
return Task.CompletedTask;
});
});
It created "AnotherSubscriber" subscription on topic ObjectCreatedA. But when it fails the message goes to the queue anothersubscriber_error. It makes it harder to diagnose, monitor and replay messages. Because from ASB perspective this is just an ordinary queue.
How do I route failures to the DLQ of topic ObjectCratedA/AnotherSubscriber instead of **_error one?
Thanks in advance.
This is now possible as of MassTransit 6.2, see the related GitHub issue.
Your configuration will now need to look something like:
cfg.SubscriptionEndpoint(
"my-subscription",
"my-topic",
e =>
{
e.ConfigureConsumer<MyConsumer>(provider);
// Send failures to built-in Azure Service Bus Dead Letter queue
e.ConfigureDeadLetterQueueDeadLetterTransport();
e.ConfigureDeadLetterQueueErrorTransport();
});
I am trying to implement publish/subscribe architecture using Web API and Rabbit MQ message broker.
I have two projects in my solution: Publisher and Subscriber.
Publishing is implementing successfully but I cannot find place in my
subscriber project to read published message from the queue.
Both of my projects are .Net Core ASP WEB API
Thanks in advance
Register rabbitMq as HostedService using the AddSingleton method in ConfigureServices Method. IHostedService internally calls ApplicationGetStarted event. So the rabbit starts listening there
public void ConfigureServices(IServiceCollection services)
{
services.AddMassTransit(x =>
{
x.UsingRabbitMq();
});
// OPTIONAL, but can be used to configure the bus options
services.AddOptions<MassTransitHostOptions>()
.Configure(options =>
{
// if specified, waits until the bus is started before
// returning from IHostedService.StartAsync
// default is false
options.WaitUntilStarted = true;
// if specified, limits the wait time when starting the bus
options.StartTimeout = TimeSpan.FromSeconds(10);
// if specified, limits the wait time when stopping the bus
options.StopTimeout = TimeSpan.FromSeconds(30);
});
}
}
I've started working recently with spring webflux and Rabbitmq along with cassandra reactive repository. What I've noticed is that the message is acknowledged even saving in cassandra didn't succued for some element. I propagete exception thrown during saving but even though the message is take down from queue. I'm wondering what I should do to let Rabbitmq know that this message should be consider as failed (I want to reject message to send it to dead letter queue )
#RabbitListener(queues = Constants.SOME_QUEUE, returnExceptions = "true")
public void receiveMessage(final List<ItemList> itemList) {
log.info("Received message from queue: {}", Constants.SOME_QUEUE);
itemService.saveAll(itemList)
.subscribe(
item-> log.info("Saving item with {}", item.getId()),
error -> {
log.error("Error during saving item", error);
throw new AmqpRejectAndDontRequeueException(error.getMessage());
},
() -> log.info(Constants.SOME_QUEUE+
" queue - {} items saved", itemList.size())
);
}
Reactive is non-blocking; the message will be acked as soon as the listener thread returns to the container. You need to somehow block the listener thread (e.g. with a Future<?>) and wake it up when the cassandra operation completes, exiting normally if successful, or throwing an exception on failure so the message will be redelivered.
I solved my problem by sending explicitly acknowledge/reject message to rabbitmq. It caused that I was forced to wrote a little more code but now at least it works and I have full controll what's happening.
I have a saga setup in a fork/join configuration.
Events defined on the saga
FileMetadataMsg
FileReadyMsg
SomeOtherMsg
Process starts off when a file comes in on a separate listener.
Publishes SagaStart(correlationId)
Publishes FileSavedToMsg(correlationId, fileLoc)
Publishes FileMetadataMsg(correlationId, metadata)
Publishes FileReadyMsg(correlationId, fileLoc)
Downstream endpoint of does some work on the file
Consumer<FileSavedToMsg>
Publishes SomeOtherMsg(GotTheFileMsg.correlationId, data)
I am getting a FileSavedToMsg in the saga_skipped queue. I can only assume it's due to having a correlationId on the FileSavedToMsg because the saga itself is not using FileSavedToMsg in its state machine and does not have an Event<FileSavedToMsg>.
If this is the reason why...should I be passing the correlationId along in a field other than the CorrelationId, so the saga doesn't see it? I need it somewhere so I can tag SomeOtherMsg with it.
Here is how the saga endpoint is defined
return Bus.Factory.CreateUsingRabbitMq(cfg =>
{
var host = cfg.Host(new Uri("rabbitmq://localhost/"), h =>
{
h.Username("guest");
h.Password("guest");
});
cfg.ReceiveEndpoint(host, "study_saga", epCfg =>
{
epCfg.StateMachineSaga(machine, repository);
});
});
Here is how the worker endpoint is defined
return Bus.Factory.CreateUsingRabbitMq(x =>
{
var host = x.Host(new Uri("rabbitmq://localhost/"), h =>
{
h.Username("guest");
h.Password("guest");
});
x.ReceiveEndpoint(host, "study_3d_volume_worker", c =>
{
c.PrefetchCount = 1;
c.Instance(_studyCreatedMsgConsumer);
});
});
These are running on the same machine, but in seperate Console/Topshelf applications.
If you are getting messages on a queue that are not consumed by a consumer on that receive endpoint, it might be that you either previously were consuming that message type and removed it from the consumer (or saga, in your case) or you were using the queue from some other purpose and it consumed that message type.
Either way, if you go into the RabbitMQ management console and look for the queue, you can expand the Bindings chevron, click to go to the exchange of the same name (that's a standard MassTransit convention), and then expand the bindings of the exchange to see which message types (the exchanges named like .NET type names) are bound to that exchange.
If you see one that is not consumed by the endpoint, that's the culprit. You can Unbind it using the UI, after which messages published will no longer be sent to the queue.