I'm trying to design a system in an event-driven architecture style, trying also to expose REST API to send commands/queries. I decided to use Kafka as a message broker.
The choreography I'm trying to design is the following:
The part that is very obscure to me is how to implement event joins:
billing-service should start creating the user only when it receives the user creation event (1) and the account has been created (2)
api-gateway should return the result to the client only when both account and billing service have finished their processing (2 and 3)
I know I could use other protocols on the client side (e.g. WebSockets) but I prefer not doing that because I will need to expose such API to 3rd party. I could also do an async client call and poll to check if the request has been completed but it appears very complex to manage.
What is the suggested way of implementing such an interaction?
p.s. I'm using Spring Boot and Spring Cloud Stream.
Request/reply messaging on the client side is possible with spring-cloud-stream, but it's a bit involved because it wasn't designed for that, it's intended for unidirectional stream processing.
You would be better off using spring-kafka (ReplyingKafkaTemplate) or spring-integration-kafka (Outbound Gateway) for request/reply on the client side.
On the service side you can use a #StreamListener (spring-cloud-stream) or a #KafkaListener or a spring-integration inbound-gateway.
Related
We are architecting a new system where ESB is the centerline channel for service communication. as we don't want our HTML/js client to interact with a bus directly, so we decided to introduce Api (ReST-API) layer where the client will post request to Api than API will put a message in ESB and respective service will subscribe for that particular message and will process it.
so it will be like
UI -> Rest -> ESB -> Service
As I knew no other services will not be interested in this particular message. so just wondering cant we do below??
UI->Rest-> Service and keep ESB used for only inter-service communications not from UI to service path. considering there will be 1000's of UI to service communication.
It all depends on how many subscribers(services) would need this message that is coming from UI. As of now you have identified just one service that needs this data. Introducing an ESB layer would decouple the "Rest" and "Service" layers and in the future if there are more services needing this message, a little change in ESB layer would be sufficient to meet your future needs without disturbing the "Rest" and "Service". Without ESB, you end-up in a tightly coupled pattern. Also, if the service is enhanced in the future, ESB could play a role in data transformation.
I have a small stock-market application with Spring boot and if any product updated I want to serve an updated product to the clients in realtime
does it make sense to use message queues like RabbitMQ and Sse(Server Sent Events) for this, or is there a more sensible solution?
Solution
Publish your updated data to some channel
Your clients should subscribe to that channel to get updated feed in real-time.
Tools
Use in-house setup for RabbitMQ, ActiveMQ, Kafka or other open-source tools and implement WebSocket (For Front end applications)
Use commercial service like Google Cloud PubSub
Readymade and fully packaged solution with supported SDK for backend and frontend, https://www.pubnub.com/.
For this you can use either of
Spring Integration
Web Sockets
JMS
Spring Integration is an implementation of Enterprise Integration Patterns and is ideal for asynchronous processing data at realtime.
However, looking at your scope, it is only about publisher-subscriber pattern. Hence can be solved with JMS.
With JMS the subscribers/consumers can register/de-register dynamically. Also it provides ways to have fall-backs and tracking.
I am developing a series of microservices using Spring Boot and Kafka. For asynchronous communication, I am using Kafka which is working well.
I have a use case where I require synchronous communication between two microservices (a user registers a profile via the user profile service which needs to create an auth account in the auth microservice).
Should I just call the auth service directly (service to service communication) or should I use Kafka?
Any examples or best practise advice would be appreciated.
There are multiple factors that can drive your decision:
Required any acknowledgement from your Auth Service?
if yes:
For Immediate acknowledgement, use http
For not so immediate acknowledgement, Callback pattern can be implemented.
In your case, user profile sends request via Kafka to auth service and it calls
endpoint of user-profile to report status of the job.
if no:
Use queue one for better resiliency.
Error Handling
Think of auth service failure? What should be the reaction of user service ?
if on auth-service failure, user-service should also fail
Use http
if on auth-service failure, user service should not fails.
Use queue
Ideally in user creation and authentication realtime response is given to the client side but if it involves complex process or tasks post user creation queue should be preferred.
For multiple microservices synchronous interaction and to work on their API responses you can build a aggregator service which could serve as a communication medium between different services and work alongside your kafka queue consumer service.
There are a lot of tutorials and articles (including official site) promoting spring boot as a good tool for building microservices.
Let's say we have some rest api endpoint (User profile) which aggregates data from multiple services (User service, Stat service, Friends service).
To achieve this, user profile endpoint makes 3 http calls to those services.
But in Spring, requests are blocking and as I see, the server will quickly run out of available resources (threads) to serve request in such system.
So to me, it as quite inefficient way to build such systems (compared to non-blocking frameworks, like play! framework or node.js)
Do I miss something?
P.S.: I do not mean here spring 5 with its new webflux framework.
No one prevents you from building an asynchronous microservice architecture with Spring Boot :).
Something along these lines:
Instead of one service calling another synchronously, a service can put events to a queue (e.g. RabbitMQ). The events are delivered to services that subscribe to those events.
Using RabbitMQ and its "exchange" concept, the event producing service doesn't even need to the consumers of its events.
A blog post detailing this with Spring Boot code can be found here: https://reflectoring.io/event-messaging-with-spring-boot-and-rabbitmq/
This is not a limitation of Spring rather it is more to do with the Application Architecture.
For instance, the scenario that you have is commonly solved using Aggregate Design Pattern
While this solution is quite prevalent,it has the limitation of being synchronous, and thus blocking. Asynchronous behaviour in such scenarios should be implemented in an application specific way.
Having said that if you have to call other services in order to be able to serve a response to a request from a client(outside), this is typically an architectural problem. It really doesn’t matter if you are using HTTP or asynchronous message passing (with a request-reply pattern), the overall response time for the outside client will be bad
Also, I have seen quite a few applications which uses synchronous REST calls for external clients, but when communication is needed between internal MicroServices, it should always be asynchronous. You can read an interesting paper on this topic here MicroServices Messaging Patterns
I would like to know if using a JMS in the below scenario is feasible or not.
I am adding a feature of calling an API service which will dispatch the emails to the customer.
So i thought of implementing a JMS in my application where i would put the events or messages in the queue and write a listener in the same application which will process the message and call the rest API service call which will dispatch the message to the customers.
My question was is it good to have a JMS in between the rest call and our application ?
Or should i directly call the rest api to dispatch the messages to the customer ?
I think that depends on the availability and overhead of your rest service.
If you know there will be times that your service will be down, but don't want to impact the process using the API, then JMS queues make since.
Or if you feel the rest service is causing a bottle neck from the API service side and want to queue up the messages somewhere where they can survive an outage of your own, JMS with a provider that supports persistent messages makes since in this case.
Using JMS would also open the door for completely decoupling the two. Whatever application hosts the rest service could just as easily be converted to pull messages from the JMS queue without a need to make a rest call if that seemed more efficient.
Just a few examples of how you could justify using JMS in this scenario.