I'm thinking if Spring Cloud Stream can be a good fit for a specific system we're thinking to build ground up. There's currently a Monolith (ESB) which is currently in use but we are looking to get benefitted by the goodness of microservices (spring cloud ecosystem especially).
We receive request from the input source (JMS Queue, ActiveMQ to be specific) at the rate of 5 requests/second.
We will need to have different routing rules (based on the payload or some derived logic) and route the message to different output destinations (say A, B, C). The output destinations are JMS queues.
Finally, we'll have to receive the 3 responses from A,B,C (by listening to different set of queues) and mash up the final response. This response is finally dispatched to another output channel (which is anther JMS queue).
There are a few corner cases such as when the response for A takes more than '5' seconds, then we'll want to mash up the responses of 'B' and 'C' and an error object for 'A'. Same goes for 'B' and 'C' too.
Also, the destinations 'A','B' and 'C' are dynamic. We could have more target systems 'D', 'E' etc in the future. We're looking at not having to change the main orchestration layer if a new system is introduced.
Is Spring Cloud Stream the right choice? I'm looking for more specific pointers in case of Aggregating the responses from multiple JMS queues (with timeouts) and mashing up the response.
What you are talking about is fully sufficient for the Aggregator EIP or its more powerful friend Scatter-Gather .
Both of them are available in Spring Integration:
Aggregator
Scatter-Gather
So, you will need to have some correlationKey to be able to gather all the responses to the same group to aggregate in the end.
Also there is group-timeout option which allows you to release group when there is no all replies after some time.
Related
I'm looking for the best approach as to how I can go about doing validation of a message as its enqueued in async messaging based systems.
Scenario:
Let's say we have a two services A and B where they need to interact with each other asynchronously. And we have a queue between them lets say SQS which will receive the message from A, which will be then polled by service B.
Ask:
How can I validate the message like doing schema validation as its enqueued to SQS since currently SQS doesnt have any in-built schema validation functionality like we have for JMS
Couple of options I can think of:
Have a validation layer maybe a small service sitting between A and SQS queue but not sure how feasible this will be
Use some sort of MOM like AWS Eventbridge between A and SQS queue as it has functionalities to validate schemas as well as it could act as a central location to store all the schemas
Have a rest endpoint in B that'll do the validation and have SQS sitting behind B but then this removes the async communication b/w A and B
Would appreciate any inputs on the above ask and how it could be resolved via best practices.
I'd recommend to read about the Mediator Topology of Event-Driven architecture style. From the details that you shared, it sounds to me that putting a "Mediator Service" called M for example, which will get messages from A, make the required validations, and then will send the message to SQS on its way to B - will achieve what you want.
Validation of the message payloads can occur on the "way in" or the "way out" depending on your use case and scaling needs. Most scenarios will aim to prevent invalid data getting too far downstream i.e. you will validate before putting data into SQS.
However, there are reasons you may choose to validate the message payload while reading from the queue. For example, you may have many services adding messages, those messages may have multiple "payload versions" over time, different teams could be building services (frontend and backend) etc. Don't assume everything and everyone is consistent.
Assuming that the payload data in SQS is validated and can be processed by a downstream consumer without checking could cause lots of problems and/or breaking scenarios. Always check your data in these scenarios. In my experience it's either the number one reason, or close to it, for why breaking changes occur.
Final point: with event-driven architectures the design decision points are not just about the processing/compute software services but also about the event data payloads themselves which also have to be designed properly.
We are working on an IOT platform, which ingests many device parameter
values (time series) every second from may devices. Once ingested the
each JSON (batch of multiple parameter values captured at a particular
instance) What is the best way to track the JSON as it flows through
many microservices down stream in an event driven way?
We use spring boot technology predominantly and all the services are
containerised.
Eg: Option 1 - Is associating UUID to each object and then updating
the states idempotently in Redis as each microservice processes it
ideal? Problem is each microservice will be tied to Redis now and we
have seen performance of Redis going down as number api calls to Redis
increase as it is single threaded (We can scale this out though).
Option 2 - Zipkin?
Note: We use Kafka/RabbitMQ to process the messages in a distributed
way as you mentioned here. My question is about a strategy to track
each of this message and its status (to enable replay if needed to
attain only once delivery). Let's say a message1 is being by processed
by Service A, Service B, Service C. Now we are having issues to track
if the message failed getting processed at Service B or Service C as
we get a lot of messages
Better approach will be using Kafka instead of Redis.
Create a topic for every microservice & keep moving the packet from
one topic to another after processing.
topic(raw-data) - |MS One| - topic(processed-data-1) - |MS Two| - topic(processed-data-2) ... etc
Keep appending the results to same object and keep moving it down the line, untill every micro-service has processed it.
We have multiple microservices which are processing the same input in parallel.
We have to get the responses from these microservices and consolidate into a single response for bulk volumes. We thought of using hazelcast for doing this but I am just trying to understand the best and the most performant way to do this. Can someone please suggest some approach or tool for the same. Just a pointer would suffice. I just am looking for the right direction and I can do the rest. Thanks in advance.
I would use a messaging queue (e.g. Kafka or SQS) for asynchronous communication between microservices, mainly because it's probably the best way to scale and keep microservices decoupled.
In your example, you could have 3 topics (X, Y and Z) in your messaging queue. Your microservice "A" posts a message w/ inputs in topics X and Y. Microservices "B" and "C" consumes messages from X and Y, do what they need to do and post back messages in topic Z. Microservice "A" is also a consumer and reads messages/outputs from topic Z, doing the proper consolidation.
Note that, outputs from "B" and "C" could be embedded in the message, or the message could contain metadata that points to the data/output located somewhere else (e.g. database).
Hope it helps!
Maybe Spring-WebFlux is an option for your service. It allows for dispatching the request to the different services in parallel, process results and give the response to the client.
https://docs.spring.io/spring/docs/current/spring-framework-reference/web-reactive.html
Let's say I have a stream with 3 applications - a source, processor, and sink.
I need to retain the order of my the messages I received from my source. When I receive messages A,B,C,D, I have to send them to sink as A,B,C,D. (I can't send them as B,A,C,D).
If I have just have 1 instance of each application, everything will run sequentially and the order will be retained.
If I have 10 instances of each application, the messages A,B,C,D might get processed at the same time in different instances. I don't know what order these messages will wind up in.
So is there any way I can ensure that I retain the order of my messages when using multiple instances?
No; when you scale out (either by concurrency in the binder or by deploying multiple instances), you lose order. This is true for any multi-threaded application, not just spring-cloud-stream.
You can use partitioning so that each instance gets a partition of the data, but ordering is only retained within each partition.
If you have sequence information in your messages, you can add a custom module using a Spring Integration Resequencer to reassemble your messages back into the same sequence - but you'll need a single instance of the resequencer before a single sink instance.
I have multiple clients(referring to them as channels) accessing a service on a WebSphere message broker.
The service is a likely to be a SOAP based webservice (can possibly be RESTful too).
Prioritizing requests for MQ/JMS can be handled by WMB using the header info (priority).
The SOAP or HTTP Nodes do not seem to have an equivalent property. Wondering how we cna achieve priority for requests from a specific client channel.
Can I use multiple execution groups(EG) to give higher priortiy for a specific channel. In other words, I am thinking of using EG to give a bigger pipe for a specific channel which should translate to requests being processed faster compared to the other channels.
Thanks
the end points
If you have IIB v9 you can use the "workload management" feature described here:
http://pic.dhe.ibm.com/infocenter/wmbhelp/v9r0m0/topic/com.ibm.etools.mft.doc/bj58250_.htm
https://www.youtube.com/watch?v=K11mKCHMRxo
The problem with this is that it allows you to cap different classes of messages at max rates, it won't allow you to run low priority work at full speed when there is no high priority work for example.
So a better approach might be to create multiple EGs using the maxThreads property on the EG level HTTP connector and the number of additional instances configured on each flow to give relative priority to the different classes of traffic.