What exactly is a consumer saga, and how is it different from Automatonymous? I know that Automatonymous is a separate library that is used by MassTransit.
Consumer sagas, for lack of a better name, are the original sagas implemented by MassTransit when it was created 13 years ago. They were consumers with state and used variants of IConsumer<T> to direct messages to saga instances. Consumer sagas implement one or more interfaces to consume correlated saga events. This support is included so that it is easy to move applications from other saga implementations to MassTransit.
State Machine Sagas, which use Automatonymous, provide a powerful state machine syntax to create sagas. They are more flexible in terms of event correlation, and have a fluent syntax for defining state and behavior. They also work nicely with dependency injection via the creation of custom activities which are resolved at run-time for each message.
Automatonymous was written separately to enable its use out of MassTransit, but it by the same author (me).
Related
When developing inbound channel adapters, I couldn’t find any place that mentions the differences between AbstractMessageSource and MessageProducerSupport in Spring Integration. I’m asking this question in the context of reactive streams, so I’m actually looking at AbstractReactiveMessageSource, but I guess it doesn’t matter for my question. I also wonder whether MessageProducerSupport supports project reactor and doesn’t have an equivalent to AbstractReactiveMessageSource.
There is some documentation about these types of components: https://docs.spring.io/spring-integration/docs/current/reference/html/overview.html#finding-class-names-for-java-and-dsl-configuration
The inbound message flow side has its own components, which are divided into polling and listening behaviors.
So, the MessageProducerSupport is for those protocols which provide a listening callback for us. So, we can hook up into that, build message and produce it into a channel provided by the MessageProducer. So, it is self-eventing component which really has everything to listen to the source system and produce messages from the callback. This type of channel adapters called event-driven and samples of them are JMS, AMQP, HTTP, IMAP, Kinesis etc.
From here it is wrong to try to compare the MessageProducerSupport with an AbstractMessageSource because the are not relevant. The one you should look into is a SourcePollingChannelAdapter. Exactly this one is that kind of flow beginning endpoint which is similar to MessageProducerSupport. With only the problem that it is based on a periodic scheduled task to request for messages in the provided MessageSource. This type of component is for those protocols which don't provide listening callback, e.g. local file system, (S)FTP, JDBC, MongoDb, POP3, S3 etc.
You probable would expect something similar to MessageSource for the MessageProducer level, but there is not that kind of layer because everything single event-driven protocol has its own specifics, therefore we cannot extract some common abstraction like in case of polling protocols.
If your source system provides for you a reactive Publisher, you don't need to look into a SourcePollingChannelAdapter and MessageSource. You just need a MessageProducerSupport and call its subscribeToPublisher(Publisher<? extends Message<?>> publisher) from the start() implementation.
There is no need in the reactive implementation for the polling since Publisher is not pollable by itself it is event-driven. Although it has its own back-pressure specifics which is out of MessageProducerSupport scope.
There is also some explanation in this section of the doc: https://docs.spring.io/spring-integration/docs/current/reference/html/reactive-streams.html#source-polling-channel-adapter. And see a couple next paragraphs.
I am investigating solution to implement microservice Saga pattern in platform hosted in K8S in GCP.
There are 2 options: Eventulate Tram and Axon. However, these frameworks seem not to support message broker managed by cloud provider such as google-cloud-Pubsub whereas I do not want to deploy either Kafka or RabbitMQ to K8S since GCP support PubSub already.
So is there any way to integrate either Eventulate or Axon to use google cloud PubSub?
Thanks
Uncertain about Eventuate's angle on this, but Axon works with extensions as message brokers other than Axon Server. Throughout Axon's lifecycle (read: last 10 years), some of these have been provided, but none are currently used for all types of messages defined by Axon Framework. So, you wouldn't be able to use Kafka for sending commands in Axon for example.
Reasoning for this? Commands, events and queries have different routing requirements which should be reflected by using the right tool for the job.
To be a bit more specific on Axon's side, the following extensions can be used for distributing your messages:
AMQP -> for Events
Kafka -> for Events
JGroups -> for Commands
Spring Cloud Discovery -> for Commands
As you can tell, there currently is no Pub/Sub extension out there to allow you to distribute your messages. Added on top of that, my gut would tell me if it was available, then it would likely only be used for Event messages due to Pub/Sub's intent when it comes to being a message broker.
Luckily this actually makes it rather straightforward to create just such a extension yourself. Going into all the details to build this would be a little much, so I would recommend to have a look at Axon's AMQP extension first when it comes to achieving this. Hints on the matter are that for publication, you should add a component to handle Axon's events and publish them on Pub/Sub. For handling events, you are required to build a StreamableMessageSource or SubscribableMessageSource. These interfaces are used respectively by the TrackingEventProcessor and SubscribingEventProcessor, which in turn are the component in charge of dealing with the technical aspect of handling events.
By the way, if you would be building such an extension and you need a hand, it would be best to request this at AxonIQ forum, which you can find here.
Last note, and rather important I'd say, is the argument that such a connector would not be able to deal with all types of messages. If you would require a more full fledged Axon application to run in a distributed fashion, I would highly recommend to give Axon Server a try prior to building your own solution from the ground up.
What is the actual difference between SAGA design and SEDA (Staged event driven) design? Mostly the search I did, points to example of SAGA usage between micro services and SEDA example generally points to dividing event processing into multiple stages. SEDA does not talk about usage between two micro services or between components in same micro services. If I think of SEDA stage processing where messages are exchanged between component via queue, then I could think of SEDA is generic version of SAGA whereas SAGA talks about how to achieve event processing between micro services. SAGA also used to implement local transaction when business process involves across multiple micro services with the help of events. I am interested to know your thoughts on these patterns
I'm fairly new to Microservices...
I've taken an interest in learning more about two main patterns like service discovery and circuit breaker and I have conducted research on how these could be implemented.
As a Java Developer, I'm using Spring Boot. From what I understand, these patterns are useful if microservices communicate via HTTP.
One of the topics I've recently seen is the importance of event-driven architecture, which makes use of an event message bus that services would use to send messages to for other services, which subscribe to the bus
and process the message.
Given this event-driven nature, how can service-discovery and circuit breakers be achieved/implemented, given that these are commonly applicable for services communicating via HTTP?
From what I understand, these patterns are useful if microservices communicate via HTTP.
It is irrelevant that the communication is HTTP. The circuit breaker is useful in prevention of cascade failures that are more probable to occur in the architectures that use a synchronous communication style.
Event-driven architectures are in general asynchronous so cascade failure is less probable to occur.
Service discovery is used in order for the microservices to discover each other but in Event-driven architectures microservices communicate only to the messaging infrastructure (i.e. the Event store in Event sourcing) so discoverability could be used only at the infrastructure level.
I. circuit breaker and service discovery are patterns. When we say Pattern they can be implemented with any programming language. 'HTTP' protocol is for transfer of data.
circuit breaker can be implemented within Java. You can find many implementations (of course, with varying capabilities and interpretation of pattern) on github.
Some of the well-known, built for purpose implementations are :
Hysterix from NetflixOSS For using Hysterix: You can follow Spring Guide - Spring Circuit Breaker
Apache Polygene - which has example of JMX circuit breaker
Resilience4j
II. About,
Given this event-driven nature, how can service-discovery and circuit
breakers be achieved/ implemented, given that these are commonly
applicable for services communicating via HTTP?
It seems you need bit more research on topic of Microservices interactions.
There are two ways to which microservices interactions are possible. You have to choose one over the other. You can/should not mix both.
Orchestration: An interaction style that has an intelligent controller that dispatches events to processes. Please note the word 'processes' which is representing business processes here. Orchestration style was preferred in old SOA implementations as well.
Choreography: An interaction style that allows processes to subscribe to events and handle them independently or through integration with other processes without the need for a central controller.
These topics are greatly covered under
Orchestration vs. Choreography
Need of Service Discovery:
With choreography, two or more microservices can coordinate their activities and processes to share information and value.
But, these microservices may not be aware of each other's existence i.e. There are no hard-coded or service references of dependency endpoints configured or coded into them. Why we do this, is for avoiding any kind of coupling between services. So, the question remains is how one service, if required will find another services' endpoint? This is where service discovery mechanism is used.
Another perspective is, with microservices deployment with containers etc, microservices endpoints will not be even tied to any hosts etc. [due to spin-up and spin-down of containers]. So, for this case as well, we need 'service discovery' mechanism.
So, In service discovery mechanism, a centralized service discovery tool helps services to register themselves and to discover other services via a DNS or HTTP interface.
Service discovery can be implemented with
1. Server-side service discovery
2. Client Side service discovery
Consul,etcd, zookeeper are some of the key-tools names within service discovery space.
Spring Boot integrates well with Spring Cloud. And Spring Cloud provides Eureka (for service discovery) as well as Hystrix (for circuit breaker patterns). Also, Spring Cloud Stream to provide event driven patterns
Very easy to use with Spring Boot
I believe there is a misunderstanding in the question in that you assume that event-driven architectures cannot be implemented on top of HTTP.
An event-driven architecture may be implemented in many different ways and (when the architecture is that of a distributed system), on top of many different protocols.
It can be implemented using a message broker (i.e. Kafka, RabbitMQ, ActiveMQ, etc) as you suggested it too. However, this is just a choice and certainly not the only way to do it.
For example, the seminal book Building Microservices by Sam Newman, in Chapter 4: Integration, under Implementing Asynchronous Event-Based Collaboration says:
“Another approach is to try to use HTTP as a way of propagating
events. ATOM is a REST-compliant specification that defines semantics
(among other things) for publishing feeds of resources. Many client
libraries exist that allow us to create and consume these feeds. So
our customer service could just publish an event to such a feed when
our customer service changes. Our consumers just poll the feed,
looking for changes. On one hand, the fact that we can reuse the
existing ATOM specification and any associated libraries is useful,
and we know that HTTP handles scale very well. However, HTTP is not
good at low latency (where some message brokers excel), and we still
need to deal with the fact that the consumers need to keep track of
what messages they have seen and manage their own polling schedule.
I have seen people spend an age implementing more and more of the
behaviors that you get out of the box with an appropriate message
broker to make ATOM work for some use cases. For example, the
Competing Consumer pattern describes a method whereby you bring up
multiple worker instances to compete for messages, which works well
for scaling up the number of workers to handle a list of independent
jobs. However, we want to avoid the case where two or more workers see
the same message, as we’ll end up doing the same task more than we
need to. With a message broker, a standard queue will handle this.
With ATOM, we now need to manage our own shared state among all the
workers to try to reduce the chances of reproducing effort. If you
already have a good, resilient message broker available to you,
consider using it to handle publishing and subscribing to events. But
if you don’t already have one, give ATOM a look, but be aware of the
sunk-cost fallacy. If you find yourself wanting more and more of the
support that a message broker gives you, at a certain point you might
want to change your approach.”
Likewise, if your design uses a message broker for the event-driven architecture, then I'm not sure if a circuit breaker is needed, because in that case the consumer applications control the rate at which event messages are being consumed from the queues. The producer application can publish event messages at its own pace, and the consumer applications can add as many competing consumers as they want to keep up with that pace. If the server application is down the client applications can still continue consuming any remaining messages in the queues, and once the queues are empty, they will just remain waiting for more messages to arrive. But that does not put any burden on the producer application. The producer and the consumer applications are decoupled in this scenario, and all the work the circuit breaker does in other scenarios would be solved by the message broker application.
Somewhat similar can be said of the service discovery feature. Since the producer and the consumer do not directly talk to each other, but only through the message broker, then the only service you need to discover would be the message broker.
I need a messaging or event framework in my Spring project.
Basic requirements:
Single producer/sender, which will create the messages/events
Global channel/queue/etc where the producer will send messages into
Multiple components should be able to register within this channel/queue, so they can receive the messages/events
All components should be able to receive all messages - every message would be visible to all receivers, NOT just to one (first one for example). So single consumer cannot make a message to disappear and be not visible to others
Messages should be distributed across all consumers asynchronous way, so all of them can receive messages at the same time, not just each after other
What would the best fit for my needs?
I think your requirements meet the features of Spring Integration.
http://www.springsource.org/spring-integration
Spring has native support for Observer pattern (Look at Events section). Check if it meets your needs. If it doesn't then you can use Spring's JMS support + ActiveMQ.
Guava EventBus could work for you. It allows publish-subscribe-style communication between components without the need for them to explicitly know of each other. Here's a nice article explaining it further with some examples.