I have a pub/sub scenario, where I create and save to DB something in one service, publish it to a SNS topic, subscribe with SQS listener, and handle the message and save it to DB in another service. So far so good.
In one of the scenarios I create a user and subscribe it to a site. Then I send the new user to its topic, the user-site relation to another topic, and the subscribed service updates its own DB tables.
private void publishNewUserNotifications(UserEntity userEntity, List<SiteEntity> sitesToAssociateWithUser) {
iPublisherService.publishNewUserNotification(userEntity);
if (sitesToAssociateWithUser != null || !sitesToAssociateWithUser.isEmpty()) {
List<String> sitesIds = sitesToAssociateWithUser.stream().map(SiteEntity::getSiteId).collect(Collectors.toList());
iPublisherService.publishSitesToUserAssignment(userEntity.getId(), new ArrayList<>(), sitesIds);
}
}
The problem is that sometimes I have a thread race and handle the user-site relation before I created the user in the second service, get an empty result from DB when loading the User object, and fail to handle the user-site relation.
#Override
#Transactional
public void handle(UsersSitesListNotification message) {
UsersSitesNotification assigned = message.getAssigned();
List<UserEntity> userEntities = iUserRepository.findAllByUserIdIn(CollectionUtils.union(assigned.getUserIds()));
List<SiteEntity> siteEntities = iSiteRepository.findAllByIdIn(CollectionUtils.union(assigned.getSiteIds()));
List<UserSiteAssignmentEntity> assignedEntities = fromUsersSitesNotificationToUserSiteAssignmentEntities(assigned, userEntities, siteEntities);
Iterable<UserSiteAssignmentEntity> saved = iUserSiteAssignmentRepository.saveAll(assignedEntities);
}
Because of that, I consider using SQS in a synchronic way. The problem is that in order to use SQS I need to import the "spring-cloud-aws-messaging" package, and the SQS configuration inside it uses the Async client.
Is there a way to use SQS in a synchronic way? What should I change? How should I override the Async configuration that I need in the package/get some other package?
Any idea will help, tnx.
Related
I have a producer, which send more than 1000 messages in a minute to a specific endpoint. I’m using Microsoft DI and I’ve configured the send Endpoint as described here https://masstransit-project.com/usage/producers.html#send .
// Masstransit setup
serviceCollection.AddMassTransit(mt =>
{
mt.UsingAzureServiceBus((ctx, cfg) =>
{
cfg.Host(massTransitSettings.TestServiceBusConnectionString);
cfg.ReceiveEndpoint("mytestmessage", e =>
{
e.MaxDeliveryCount = 3; //How many times the transport will redeliver the message on negative acknowledgment
});
});
});
serviceCollection.AddTransient<ITestMessageProducer, TestMessageProducer>();
// Producer setup
public class TestMessageProducer : ITestMessageProducer
{
private readonly ISendEndpointProvider _testEndpoint;
public TestMessageProducer(ISendEndpointProvider testEndpoint)
{
_testEndpoint = testEndpoint;
}
public async Task SendTestMessage(ITestMessage testmessage)
{
var endpoint = await _testEndpoint.GetSendEndpoint(new Uri("queue:mytestmessage"));
await endpoint.Send(testmessage);
}
}
Query:
The SendTestMessage function has been called very frequently as mention above. Will it be ok to call “GetSendEndpoint” everytime? I have read somewhere that GetSendEndpoint creates a new instance of ISendEndpoint everytime.
Will the MaxDeliveryCount still be worked on my sendendpoint?
Thank you.
Send endpoints are cached by address, only a single instance will be created.
MaxDeliveryCount is a receive endpoint concern, but you should not configure a receive endpoint without consumers as all messages will be moved to the _skipped queue.
Can some one help how to configure NATS jet stream subscription in spring boot asynchronously example: looking for an equivalent annotation like #kafkalistener for Nats jetstream
I am able to pull the messages using endpoint but however when tried to pull messages using pushSubscription dispatcherhandler is not invoked. Need to know how to make the listener to be active and consume messages immediately once the messages are published to the subject.
Any insights /examples regarding this will be helpful, thanks in advance.
I don't know what is your JetStream retention policy, neither the way you want to subscribe. But I have sample code for WorkQueuePolicy push subscription, wish this will help you.
public static void subscribe(String streamName, String subjectKey,
String queueName, IMessageHandler iMessageHandler) throws IOException,
InterruptedException, JetStreamApiException {
long s = System.currentTimeMillis();
Connection nc = Nats.connect(options);
long e = System.currentTimeMillis();
logger.info("Nats Connect in " + (e - s) + " ms");
JetStream js = nc.jetStream();
Dispatcher disp = nc.createDispatcher();
MessageHandler handler = (msg) -> {
try {
iMessageHandler.onMessageReceived(msg);
} catch (Exception exc) {
msg.nak();
}
};
ConsumerConfiguration cc = ConsumerConfiguration.builder()
.durable(queueName)
.deliverGroup(queueName)
.maxDeliver(3)
.ackWait(Duration.ofMinutes(2))
.build();
PushSubscribeOptions so = PushSubscribeOptions.builder()
.stream(streamName)
.configuration(cc)
.build();
js.subscribe(subjectKey, disp, handler, false, so);
System.out.println("NatsUtil: " + durableName + "subscribe");
}
IMessageHandler is my custom interface to handle nats.io received messages.
First, configure the NATS connection. Here you will specify all your connection details like server address(es), authentication options, connection-level callbacks etc.
Connection natsConnection = Nats.connect(
new Options.Builder()
.server("nats://localhost:4222")
.connectionListener((connection, eventType) -> {})
.errorListener(new ErrorListener(){})
.build());
Then construct a JetStream instance
JetStream jetStream = natsConnection.jetStream();
Now you can subscribe to subjects. Note that JetStream consumers can be durable or ephemeral, can work according to push or pull logic. Please refer to NATS documentation (https://docs.nats.io/nats-concepts/jetstream/consumers) to make the appropriate choice for your specific use case. The following example constructs a durable push consumer:
//Subscribe to a subject.
String subject = "my-subject";
//queues are analogous to Kafka consumer groups, i.e. consumers belonging
//to the same queue (or, better to say, reading the same queue) will get
//only one instance of each message from the corresponding subject
//and only one of those consumers will be chosen to process the message
String queueName = "my-queue";
//Choosing delivery policy is analogous to setting the current offset
//in a partition for a consumer or consumer group in Kafka.
DeliverPolicy deliverPolicy = DeliverPolicy.New;
PushSubscribeOptions subscribeOptions = ConsumerConfiguration.builder()
.durable(queueName)
.deliverGroup(queueName)
.deliverPolicy(deliverPolicy)
.buildPushSubscribeOptions();
Subscription subscription = jetStream.subscribe(
subject,
queueName,
natsConnection.createDispatcher(),
natsMessage -> {
//This callback will be called for incoming messages
//asynchronously. Every subscription configured this
//way will be backed by its own thread, that will be
//used to call this callback.
},
true, //true if you want received messages to be acknowledged
//automatically, otherwise you will have to call
//natsMessage.ack() manually in the above callback function
subscribeOptions);
As for the declarative API (i.e. some form of #NatsListener annotation analogous to #KafkaListener from Spring for Apache Kafka project), there is none available out of the box in Spring. If you feel like you absolutely need it, you can write one yourself, if you are familiar with Spring BeanPostProcessor-s or other extension mechanism that can help to do that. Alternatively you can refer to 3rd party libs, it looks like a bunch of people (including myself) felt a bit uncomfortable when switching from Kafka to NATS, so they tried to bring the usual way of doing things with them from the Kafka world. Some examples can be found on github:
https://github.com/linux-china/nats-spring-boot-starter,
https://github.com/dstrelec/nats
https://github.com/amalnev/declarative-nats-listeners
There may be others.
With nestjs microservices, you can send messages and receive the result using request/response-based approach. That is implemented with a combination of #MessagePattern and client.send('my_pattern', myData). For an example see the nest docs: https://docs.nestjs.com/microservices/basics#request-response and https://docs.nestjs.com/microservices/basics#sending-messages.
How do I receive the result in the event-based approach?
Suppose you have a user microservice and an auth microservice. Whenever a user is created you want an auth subject to be created as well (saving the username and a hash of the password, so that the user may login with an api request to the auth microservice instead of the user service).
auth/auth.controller.ts
#EventPattern('EVT_USER_CREATED')
public async handleUserCreated(data: any): Promise<AuthSubject> {
if (!data.username || !data.password) {
throw new RpcException('Auth subject must supply username and password');
}
const newSubject: CreateAuthSubject = {
username: data.username,
password: data.password,
email: data.email ?? '',
};
const sub = await this.subjectService.create(subject);
return sub;
}
user/user.controller.ts
#Post('')
#ApiBody({ type: CreateUser })
#ApiCreatedResponse({ type: User })
public async create(#Body() user: CreateUser): Promise<User> {
const newUser = await this.userService.create(user);
this.userQueue
.emit<any, CreateUser>('EVT_USER_CREATED', user)
.subscribe((res) => {
console.log(res); // undefined
});
return newUser;
}
To verify that there is no error in my setup, I changed #EventPattern to #MessagePattern and this.userQueue.emit<... to this.userQueue.send<.... It worked, i.e. res was a valid auth subject with username and password as expected. However, with the event-based approach outlined in this question res is always undefined (whether or not the auth controllers handleUserCreated returns or throws).
Ultimately I would like to achieve the following: If another microservice needs to process 'EVT_USER_CREATED' events, I just add a #EventPattern('EVT_USER_CREATED') public async handleUserCreated method to its controller. The observable this.userQueue.emit<any, CreateUser>('EVT_USER_CREATED', user) would then receive both results: Once for each of the microservices consuming the user created event.
So suppose I add a third microservice: the customer microservice that is responsible for saving payment information, the history of orders, etc. Like the auth service it subscribes to 'EVT_USER_CREATED'.
customer/customer.controller.ts
#EventPattern('EVT_USER_CREATED')
public async handleUserCreated(data: any): Promise<AuthSubject> {
const customer = await this.customerService.create(data);
return customer ;
}
Now with the above setup the microservices auth and customer will alternate in receiving the events: If the user microservices emits the creation of a user, only the auth service will react to it and create an auth subject from hat user. No customer will be created for that user. For the next user that is created in the user microservice, only a customer will be created but not an auth subject. The third created user will again be consumed by the auth microservice but not by the customer microservice. And so on.
-- auth microservice
/
user microservice --- message broker ---
\
-- customer microservice
To summarize: How do I achive the messaging architecture shown in the diagram, such that I only need one emit(...) call in the user.controller.ts and, such that I reveive both responses in the subscrption to the emit(...) call?
It may be a bit late but I leave my contribution for future users.
For this type of architecture strongly based on events, it is advisable to use a messaging broker such as Kafka, with which different services can be subscribed to different topics, and even different services can listen to the same topic, in this way you can react in different ways to the same event.
Kafka also offers many advantages that will be useful when you want to scale your microservices, and as an extra, it has support in Nest.
https://docs.nestjs.com/microservices/kafka
What you should keep in mind is that when messaging systems are used, communication is usually completely asynchronous, this is an important detail since, following the case you present, not only is it enough for you to send a message through kafka to another microservice to validate the credentials of a user, but it is necessary to return a response to the client. For this case, the tools offered by nest are limited since using the #MessagePattern decorator, we can receive messages from kafka in our controller, but we cannot wait for a response (of the same topic or another) with the confirmation of success to respond to the Customer request. In this case, it would be convenient for you to create your own transporter or your own kafka client with Kafkajs (https://kafka.js.org/) so that if necessary, you can keep the user's request until you receive a confirmation for another topic.
A solution that I have seen in certain forums is to save an object in memory (key / value) with the "Response" object of a request associated with the id of a user, in this way you can send a message by kafka or any other broker with a specific action, and receive the confirmation by another controller, retrieve the user's "Request" object and then, send the response (Request.send (...)). It is not the ideal solution but it works for some cases.
For example:
#Controller('users')
class MyController {
constructor(private kafkaClient: KafkaClient) {}
private users: new Map<string, Request>() // Or private users: any = {}
onModuleInit() {
// By default, NestJs create topic "users-topic.reply"
this.kafkaClient.subscribeToResponseOf('users-topic')
}
#Post('auth')
authUser(#Req req: Request, #Res res: Response): void {
const userData = req.body;
const { id } = userData;
// Save request object in private variable;
this.users.set(id, res); // or this.users[id] = res;
// Send kafka message to validate user data
this.kafkaClient.emit('users-topic')
}
#MessagePattern('users-topic.reply')
authUser(#Payload() data: any): any {
if (data.message === 'success') {
const res:Request = this.users.get(data.id); // or this.users[data.id];
// Clean users object
this.users.remove(data.id); // or this.users[data.id] = null;
// Response client request
res.send('Authentication successfully')
}
}
}
I want to create an interview bot using azure bot service and want to use cosmos db for interview questions can it possible?Need help and suggestions for this.
It is unclear what is your architecture to actually get the bot working and if there is any limitation, but I assume you are using C# as your language and hosting the Bot in a C# Web application.
You can use this article as base Bot conversation history with Azure Cosmos DB.
It not only shows how to store UserData, but also how to store the State in Cosmos DB (this actually is better because you get the performance benefits of Cosmos DB and you also go over the 32Kb limit that the Bot Framework State has).
Following that article, you will be storing in Cosmos DB:
User Data Store: To store data specific to a user.
Conversation Store: To store data specific to a conversation.
Private Conversation Store: To store data specific to a user in a conversation
If you want to store the chat lines, it's not done by default by the Bot Framework. You have to create a class that implements IActivityLogger and let the user know that you are storing the chat.
public class CosmosDBActivityLogger : IActivityLogger
{
private readonly DocumentClient _client;
private readonly string _collectionUri;
public ServiceBusActivityLogger(DocumentClient client, string databaseName, string collectionName)
{
this._client = DocumentClient;
// This is the collection where you want to store the chat
this._collectionUri = UriFactory.CreateDocumentCollectionUri(databaseName, collectionName);
}
public async Task LogAsync(IActivity activity)
{
var message = activity.AsMessageActivity();
// At this point you might want to handle your own Activity schema or leave the default
// Not handling errors for simplicity's sake, but you should
this._client.CreateDocumentAsync(this._collectionUri, message);
}
}
Then you have to add the logger wherever you are declaring your Bot Container, for example, in Global.asax:
protected void Application_Start()
{
var builder = new ContainerBuilder();
builder.RegisterType<CosmosDBActivityLogger>().AsImplementedInterfaces().InstancePerDependency();
builder.Update(Conversation.Container);
GlobalConfiguration.Configure(WebApiConfig.Register);
}
More info on how to register the middleware here.
I've been trying to come up with a demo of a website that uses MassTransit with RabbitMQ to post messages to a service running on Service Fabric as a Stateful service.
Everything was going fine, my client would post a message:
IBusControl bus = BusConfigurator.ConfigureBus();
Uri sendToUri = new Uri($"{RabbitMqConstants.RabbitMqUri}" + $"{RabbitMqConstants.PeopleServiceQueue}");
ISendEndpoint endPoint = await bus.GetSendEndpoint(sendToUri);
await endPoint.Send<ICompanyRequest>(new {CompanyId = id });
My consumer in my service fabric service was defined like:
IBusControl busControl = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
IRabbitMqHost host = cfg.Host(new Uri(RabbitMqConstants.RabbitMqUri), h =>
{
h.Username(RabbitMqConstants.UserName);
h.Password(RabbitMqConstants.Password);
});
cfg.ReceiveEndpoint(host, RabbitMqConstants.PeopleServiceQueue, e =>
{
e.Consumer<PersonInformationConsumer>();
});
});
busControl.Start();
This does allow me to consume the message in my class and I can process it fine. The problem comes when we want to use IReliableDictonary or IReliableQueue or anything that needs to reference the context that is run from the RunAsync function in the service fabric service.
So my question is, how can I configure (is it possible) MassTransit to work within a Stateful Service Fabric Service which knowledge of the service context itself?
Many thanks in advance.
Mike
Update
Ok, I've made some progress on this, if I point the register routines to my message consumer class (eg):
ServiceRuntime.RegisterServiceAsync("ServiceType", context => new PersonInformationConsumer(context)).GetAwaiter().GetResult();
ServiceEventSource.Current.ServiceTypeRegistered(Process.GetCurrentProcess().Id, typeof(PersonInformationConsumer).Name);
Then in my consumer class for my messages I can do the following:
internal sealed class PersonInformationConsumer : StatefulService, IConsumer<ICompanyRequest>
{
private static StatefulServiceContext _currentContext;
#region Constructors
public PersonInformationConsumer(StatefulServiceContext serviceContext) : base(serviceContext)
{
_currentContext = serviceContext;
}
public PersonInformationConsumer() : base(_currentContext)
{
}
I can now successfully call the service message:
ServiceEventSource.Current.ServiceMessage(this.Context, "Message has been consumed, request Id: {0}", context.Message.CompanyId);
The problem I have now is trying to store something on the IReliableDictionary, doing this causes as "Object reference not set to an instance of an object" error :( ... any ideas would be appreciated (although may not read until new year now!)
public async Task Consume(ConsumeContext<ICompanyRequest> context)
{
ServiceEventSource.Current.ServiceMessage(this.Context, "Message has been consumed, request Id: {0}", context.Message.CompanyId);
using (ITransaction tx = StateManager.CreateTransaction())
{
try
{
var myDictionary = await StateManager.GetOrAddAsync<IReliableDictionary<string, long>>("myDictionary");
This is causing the error.... HELP! :)
You'll need to do a bit more to get MassTransit and stateful services working together, there's a few issues to concern yourself here.
Only the master within a stateful partition (n masters within n partitions) will be able to write/update to the stateful service, all replicas will throw exceptions when trying to write back any state. So you'll need to deal with this issue, on the surface it sounds easy until you take in to consideration the master can move around the cluster due to re-balancing the cluster, the default for general service fabric applications is to just turn off the processing on the replicas and only run the work on the master. This is all done by the RunAsync method (try it out, run 3 stateful services with something noddy in the RunAsync method, then terminate the master).
There is also partitioning of your data to consider, due to stateful services scale with partitions, you'll need to create a way to distributing data to separate endpoint on your service bus, maybe have a separate queue that only listens to a given partition range? Say you have a UserCreated message, you might split this on country UK goes to partition 1, US goes to partition 2 etc...
If you just want to get something basic up and running, I'd limit it to one partition and just try putting your bus creation within the the RunAsync and shutdown the bus once a cancelation is requested on the cancelation token.
protected override async Task RunAsync(CancellationToken cancellationToken)
{
var busControl = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
IRabbitMqHost host = cfg.Host(new Uri(RabbitMqConstants.RabbitMqUri), h =>
{
h.Username(RabbitMqConstants.UserName);
h.Password(RabbitMqConstants.Password);
});
cfg.ReceiveEndpoint(host, RabbitMqConstants.PeopleServiceQueue, e =>
{
// Pass in the stateful service context
e.Consumer(c => new PersonInformationConsumer(Context));
});
});
busControl.Start();
while (true)
{
if(cancellationToken.CancellationRequested)
{
//Service Fabric wants us to stop
busControl.Stop();
cancellationToken.ThrowIfCancellationRequested();
}
await Task.Delay(TimeSpan.FromSeconds(1));
}
}