I have a User Microservice, that publishes events when a user is created, modified, or deleted. In addition, I have a Calendar Microservice that needs the user data. The Calendar service, subscribes to the user events and keeps a read only copy of the needed data.
I have just added an Account Microservice that needs the existing user data and I have used the same model as above. What is the best way to get the existing user data into the Account Microservice?
When I did the first Microservice I republished the user-modified event for every user. It was simple, because nothing else was using this event. If I republish now, for the new Account Microservice, the existing Calendar Microservice will also get the events. My logic is idempotent, but this is a lot of wasted work and will only get worse as I get more services.
I've loaded data a lot in the past. I know how to make it work, but I am looking for a best-practice and a way to do it with minimal coordination and dependencies with other services.
When I did the first Microservice I republished the user-modified event for every user. It was simple, because nothing else was using this event. If I republish now, for the new Account Microservice, the existing Calendar Microservice will also get the events. My logic is idempotent, but this is a lot of wasted work and will only get worse as I get more services.
What you probably want here is a design in which you pull, rather than push, copies of the events to new systems: see Greg Young, Polyglot Data.
In broad strokes - the subscriber keeps track of the high water mark, and asks for more events after some mark, and then the event store answers that query with zero or more new events, in order. The twitter timeline api can give you a sense for how that might work -- sadly, the images that explained the ideas seem to have been removed from the docs, but some of them have been captured here at stack overflow, and by the wayback machine
Related
I've manage to send notification to Teams users through the Chatbot.
In order to send notification, you need to persist the conversation in order to have a channel to push the message or anything else. The persistence of the conversation are done in memory, so, after i restart the chatbot, i'm not able anymore to notify any user until they come back to chat with the bot and store the conversation again.
It's possible to save the conversation on a storage like CosmosDB or BlobStorage using this library https://github.com/microsoft/botbuilder-js/tree/main/libraries/botbuilder-azure?
What i understand about this library, it has been developed to persist some state of the conversation, not the conversation itself.
Maybe there is another way to retrieve conversations (or create from scratch with some stored data)?
The samples you see online store the data in memory because they are -just- samples, and it's much easier to keep the sample self-contained if it doesn't have a dependency on you setting up a database just to get the sample running.
In the real world the data -must- be kept somewhere else, for all practical purposes, such as Cosmos, Blob or other. In addition, you actually don't need the -entire- conversation reference - see here a more detailed answer I wrote a few months ago with some further references: Sending proactive messages from an outside process to organizational users via Teams chat bot
Also worth noting, the samples and docs show to get the data you need in the conversationUpdate from the user (i.e. when they install your bot), which is definitely easiest, but if you've lost it now (because your bot has reset), you get the same data from the user every time they send a message to your bot - you can save it at that time if you haven't saved it already.
I'm currently working on a project that is a sort of an extension to an existing project.
Existing Project:
A product that is already there and has a pub/sub system of its own. It uses GraphQL.
Let's take an example.
There's a group (GROUP_1) and some users are a part of that group.
Let's say user A calls a mutation, the other users' sessions subscribed to that mutation get an update regarding the same(that A made these changes). The changes are reflected in their frontend accordingly.
What do we need?
Instead of user A making the change, we want to update when a third party service notifies us to make the change.
The thing is that the third-party service uses MQTT protocol and will give us a topic (for GROUP_1) to which it will publish the messages and we need to subscribe to it. Upon receiving the message if the message satisfies our condition we will call the mutation on behalf of the user the message specifies.
Problem
So we need the client to listen to that topic forever.
And there can be an addition to the topics provided. New groups can be added in the future. So we will have different topics for different groups(GROUP_1, GROUP_2, etc)
Research that I did
To have a client that listens to a topic permanently and we can add to that client new topics as we generate new GROUPS.
And then let this client update the GraphQL API eventually reflecting changes in the session of the users in that GROUP.
I think that's not the way MQTT should be used, we should make changes in the existing project but that is not in our hands as of now.
Constraint
We can't change the existing workflow of GraphQL mutation and subscription that the existing project has.
Questions
Is there a better approach to do this with the given constraint?
If not then how should we go about creating a client that listens permanently. Shall we have a server that's up for the same or is there a better way to do this?
Maybe I'm not understanding your post, but the solution seems pretty simple: Create a new client that subscribes to what the "third party service" publishes to, and then publish to the Topic that the existing clients subscribe to, all the info that was passed in by the 3rd party. The beauty of a Pub/Sub architecture is that adding functionality to an existing system is done by just adding a new set of clients...subscribers, publishers, or both. The existing system doesn't have to change at all, if you can reuse the existing Topics and data formats. So you create a "Proxy" that takes in the new info, and publishes out in the old format.
Suppose that I have a microservice for messaging. The microservice knows how to send emails. The service have templates of emails that have some sort of "template engine" like pugjs, and can replace data in the body of the message.
I have an user service (used for authentication/authorization for example), and a bank account service (each user have one). Between the User microservice and Bank Account microservice it's clear that we don't have to duplicate any data than de user's uuid.
But I want now to send every day a message to each user with their account statement. The Messaging microservice needs data from the User microservice and the Bank Account microservice.
Okay... This is a small case of the real world. Now I know that to have the benefits of decoupled microservices I must follow some rules:
I can't share databases between microservices
I can't make synchronous requests between microservices
Okay... I can use a broker and each time a new user is created/updated the Messaging microservice can store that data. But really, this is a stupid thing:
I don't want to have inconsistency with this data, and keeping things sync is hard
The development time and complexity of the Messaging Microservice must now consider: listen and extract the relevant data from events, keep data consistent about other domains/services, managing the saved data on database
And think about a Messaging microservice. Really I must store all the data needed to parse the templates?
I read a lot about microservices and people creating rules for their simple examples. But I never really saw a good explanation and real-world examples like above.
So how to have the microservices above without data duplication?
In your domain example I would not let the message service know anything about bank or user details. Instead the message service should just receive instructions to send messages to recipients along with the given content. I would use a dedicated scheduled job (maybe implemented as an account notification service) that performs the work of acquiring the user and account data from the corresponding services, compiles the information for the message service and instructs it to actually send the messages. This introduces another "higher level, business purpose entity/service" but allows you to keep a clear separation of concerns.
In general it will happen frequently that your "basic" domain services are used by another service that represents a specific business purpose and requires their data. Dependency in itself is not a bad thing as long as concerns are seperated clearly and interfaces versioned, changes communicated etc.
Don't forget the whole idea of microservices is for allowing teams to have dedicated responsibilities with clear interfacing. It is about organization as much as it is about architecture.
Is there a way to avoid race conditions when inserting events into Google Calendar using the API? I have an experimental app that needs to insert events into user's calendar. However, the user should not be double booked at a particular time.
The only true way of guaranteeing something like this is if Google provides a way. Other solutions are prone to race conditions (example, some other app that the user has updating it while my app is ignorant). The options as far as I can tell:
E-Tags with If-Match option - But this option is not valid for inserts according to the docs (https://developers.google.com/google-apps/calendar/v3/version-resources).
Important: There is no support for conditional modifications for
insert operations. Instead, it is guaranteed that if you are allowed
to provide a resource ID, then the operation will only succeed if no
existing entry has that ID.
So if I provide an if-match with the calendar etag and try to insert, it always fails regardless of the fact that the etag is correct.
Using "watch" for a brief period while the event is about to be inserted (https://developers.google.com/google-apps/calendar/v3/reference/events/watch). However, this solution is prone to race conditions due to the time take to get notified. One user inserting while my app gets notified of the update is problematic.
Sync the whole calendar with server periodically and consider the source of truth as the synced version and avoid double booking. Less optimal because new events added won't get synced fast enough, but the most likely will work.
What other solutions come to mind? I think #3 is the most suitable to guarantee consistency and integrity of data to avoid double booking.
I found a solution to this on this blog which states that you'll be using Calendar Resource API instead of the Google Calendar. I think this guide is what you're looking for.
Unlike a regular calendar, resources:
-Do not allow conflicts
-Can be shared across everyone on your Google Apps domain
-Can be invited (multiple at a time) to events, just like a person
-Their ability to share availability with everyone makes resources great for managing rooms schedules. Especially in a shared work
environment like your office.
FYI: This guide assumes you already have Google Apps. Currently only
paid accounts have access to resource calendars. If you’re
grandfathered in or using a personal Gmail account, this process won’t
make much sense.
I am developing a feature in my app where I need to store the state of a particular series of actions between 2 users, just like an ajax based chat service.
The scenario is as follows:
A user can is able to see which other users are online and then challenge one of them. The other user receives the challenge and accepts it. Now both the users are given 5 questions each and the match starts simultaneously for them (almost simultaneously) Then as a user navigates thru the question or solves it, the state is updated on the other user screen as well.
Essentially this is very similar to a chat system like say facebook chat. I am able to see who all are online. I send a chat msg to my friend who can then respond to that and that response is seen on my chat window.
I believe all this can be achieved by using ajax. I can easily make ajax calls to .asmx webservices and retrieve objects for a particular user as Session can be accessed there. However, I am wondering where to maintain state because session is for a particular user and I want my particular object to be accessible for two particular users.
Where do I store the state? Or taking the example of Ajax based chat, where should I store what message User1 entered and when Uer1 entered how is it shown to user2?
I was thinking application object but read it is not recmmended.
What do you recommend for such a thing?
If you're trying to do "near real-time" message passing you might want to take a look at HTTP polling (a.k.a. long polling). I won't use SQL for temporal message passing and short-term state transitions like I've seen in the past. If running on a single web server just keep the state in session or in the ASP.NET Cache. If running on multiple web servers take a look at distributed caches like memcached, Velocity (Win 2008), or NCache. Then serve the cached data to the AJAX requests which are sitting and waiting (because of long-polling). The key design issue is the design of the cache keys (no pun intended), which would need to include the user's ID for user-specific event data.
P.S. There are frameworks for large-scale, near real-time message distribution that solve scaling issues which are hit when hundreds of clients are participating in long-polling at the same time. The broad name for those frameworks are 'Comet', and they are most useful when broadcasting the same messages to many clients.