newbie here.
Reading the docs I understand we can use an incoming HTTP request as a trigger to wake up a suspended activity.
In my case, the business trigger is the arrival of a message on a bus (from another system)…..
I thought of building out dedicated hosted service that just listens to messages arriving on the bus and invoke / trigger the respective activities....
Would I be following the suggested patterns if I do that ? It feels wrong as I'd be writing some custom external code rather than relying on the declarative approach usually described in the ELSA docs...
Any thoughts welcome..
This is a great question. Both patterns are great and in fact, the declarative approach depends on supporting infrastructure (such as hosted services).
For example, let's take the HttpEndpoint and AzureServiceBusMessageReceived activities.
Both of them require supporting infrastructure:
HttpEndpoint depends on ASP.NET Core middleware to trigger workflows as HTTP requests come in
AzureServiceBusMessageReceived depends on a hosted service that contains message workers to trigger the appropriate workflows.
For your case, you don't have to write your own hosted service if you can use one of the existing messaging activities, since it's already done for you.
At the same time, it's perfectly OK to just have your own hosted service that consumes messages and trigger workflows yourself. You could make it even a bit fancier by having your hosted service trigger business-specific activities.
For example, rather than triggering some low-level "message received" activity, you could trigger a "order created" activity if that is what the message is all about.
More details about implementing these types of activities can be found https://elsa-workflows.github.io/elsa-core/docs/guides/guides-blocking-activities.
As you already discovered, there are also examples in the repository https://github.com/elsa-workflows/elsa-core/tree/master/src/samples.
I was only considering the Elsa Guides, but just discovered a whole list of additional samples in the Elsa-Core project itself. In particular, there are several examples that seem to handle my use case (example Elsa.Samples.RabbitMqWorker)....
Related
I have multiple microservices. Each microservice has REST endpoints.
I want to orchestrate microservices for creating a workflow. Is it possible with Activiti? Will it be possible without writing any code?
Assume each microservice gets some input and returns some output (which can be used as input for the next microservice), can Activiti provide a way to pass output from one microservice to another microservice directly?
Activiti, like most other BPM platforms can certainly be used for microservice orchestration. But there are many caveats.
Long running microservices (i.e. that don't immediately return results) can potentially block the BPM process which may not be ideal, services may require authentication and
other times a microservice will place results in a repository or queue which will need to be picked up by the BPM process using a service task.
There is a good article that discusses the synchronous vs asynchronous invocation at the following Camunda blog post : https://camunda.com/blog/2013/11/bpmn-service-synchronous-asynchronous/
While this is a Camunda post, the same pattern can be easily handled by Activiti.
You other question was about whether you could do this without writing code. The answer is it depends. The most trivial orchestration of synchronous ReST calls can likely be handled without writing code, but once you start using asynchronous services where results may not be placed into the ReST response, or long running service calls, or even if the service calls have something other than basic authentication then it is possible you will need to write some Java Delegate service tasks.
Yes, any process engine can be used in this way. You most likely write a little glue code for data mapping / connectors.
Benefits of covering the orchestration part with a BPMN2 process engine include for instance ootb support for:
state management and long running processes / persistence for data
versioning (!)
retries and error handling
tooling to modify state and date in case something went wrong
timeouts, parallel execution (if necessary)
scalability of the orchestration
graphical process model
audit trail
and end to end visibility in monitoring tools based on BPMN 2 model
ability to include business rules tasks (DMN) for more complex rules
combination of push and pull communication pattern and a/sync communication
business-IT alignment via BPMN 2
support for the various BPMN 2 events
standardization (skills, security, software quality, features) ...
This is a great related article about the WHY:
https://blog.bernd-ruecker.com/3-common-pitfalls-in-microservice-integration-and-how-to-avoid-them-3f27a442cd07
This is about the important design consideration in case you go with a process engine: https://blog.bernd-ruecker.com/the-microservice-workflow-automation-cheat-sheet-fc0a80dc25aa
Edit v1:
I have been going through some system design videos and learnt about microservice architecture using message queues and event-driven architecture.
But I don't seem to find any substantial point of difference between the two.
Both have different components/services publishing or subscribing to eventBus/messagingQueues and performing the tasks associated with the published event.
Is microservice architecture with messaging queues a subset of event driven architecture or is there something more to it that I need to figure out.
Original V0:
I have been going through some system design videos and learnt about microservice architecture and event-driven architecture.
But I don't seem to find any substantial point of difference between the two.
Both have different components/services publishing or subscribing to eventBus/messagingQueues and performing the tasks associated with the published event.
Is microservice architecture a subset of event driven architecture or is there something more to it that I need to figure out.
Event Driven Architecture is a system design concept where we use "techniques" to achieve synchronous or asynchronous communication in our system. More likely than not we want asynchronous communication.
Such techniques can be pub/sub, long polling, Queueing, websockets and etc.
Microservice is an approach to designing our system where we make our services decoupled to one another or at least we try our best to. For example, Facebook's newsfeed service is independent of other services like Profile, photos and messaging. One benefit of this is "separation of concerns", so for example if newsfeed goes down we can still continue to upload photos and chat our friends. If FB was "monolith", one service going down could have taken down the whole site. Another benefit of microservice is deployability, the smaller the service the faster to test and deploy.
Let's take pizza for example, deciding whether to cut it in squares or triangular, or how big/small the slices are is thinking microservices. Which one to eat first and next is thinking event-driven. Do you go for the larger slices, mixed, small ones or meatier ones? Just like how our systems can decide intelligently what events to trigger next.
Just remember that these are concepts to help you understand an existing system, or help you decide how you would build your system. In the real-world when you onboard to a new company you'll find yourself asking questions like
How service-oriented is the system?
How eventful is the flow of data?
Short answer to your question... they're not necessarily related but inevitably we implement them together when scaling one or the other.
For example given this microservice architecture
[checkout service] ---> [email service]
Let's say the user waits very long for checkout and email to finish. 90% of the wait is coming from the email service. In reality the user should be able to continue browsing the other pages while they wait for the email.
In this example we solved the long wait time by adding Queue
[checkout service] ---> [Queue] ---> [email service]
We've improved user experience by making our microservice more eventful. When the user clicks the checkout button, a response is returned immediately allowing the user to continue browsing while the "email event" is dispatched to the queue.
Short answer: No, these are not the same and not subsets.
Both have different components/services publishing or subscribing to eventBus/messagingQueues and performing the tasks associated with the published event.
This is wrong. Microservices are not necessary about events and publishing/subscribing.
In this case Wikipedia tackles this very question.
From a formal perspective, what is produced, published, propagated,
detected or consumed is a (typically asynchronous) message called the
event notification, and not the event itself, which is the state
change that triggered the message emission. Events do not travel, they
just occur. However, the term event is often used metonymically to
denote the notification message itself, which may lead to some
confusion. This is due to Event-Driven architectures often being
designed atop message-driven architectures, where such communication
pattern requires one of the inputs to be text-only, the message, to
differentiate how each communication should be handled.
https://en.wikipedia.org/wiki/Event-driven_architecture
I'll be honest, I treated them the same when designing and writing code. But I guess technically there is a difference, as per the paragraph quoted above.
Technically we cannot use the word "Same" in this case. Below will give a clear relation between these artifacts:
Event-driven Microservices rely on message queues (to store/forward messages) to send/receive events, wrapped in messages.
Event-driven architectures usually leverage messaging technology in order to transport the information that something has happened in the past from one place to another or many other places.
Message queues can also be used in a non-event driven architecture, for instance, to perform asynchronous request/response communication.
In addition, when using an event-driven approach the information that is transmitted is usually different, it is just indicating what (business) event has happened with usually fewer information than provided by normal messages.
For instance, you can send a message to create a new order in an online shop system and the message could contain all the information the receiver needs to process it. The important thing is also that there is dedicated receiver of the message.
In the event-driven approach some component would rather send some order checkout requested event (or similar) without knowing what other component or what other components (think of publish/subscribe mechanisms) will "listen* to that event and perform corresponding actions. In such an architecture it would also make sense to send other events before the actual checkout happens, such as new shopping cart created or item added to cart event.
So the event-driven approach also implies some kind of choreography between your Microservices where more complex business operations can include lots of events published and processed by different components without having one central component which orchestrates who gets what information in what order.
Of course, from my experience, it makes perfect sense to combine event-driven choreography and non-event driven orchestration in a Microservices architecture depending on the use cases.
We are designing a reporting system using microservice architecture. All the services are supposed to be subscribers to the event bus and they communicate by raising events. We also decided to expose each of our services using REST api. Now the question is , is it a good idea to create our services as web api [RESTful] applications which are also subscribers to the event bus? so basically there are 2 ponits of entry to each service - api and events. I have a feeling that we should separate out these 2 as these are 2 different concerns. Any ideas?
Since Microservices architecture are Un-opinionated software design. So you may get different answers on this questions.
Yes, REST and Event based are two different things but sometime both combined gives design to achieve better flexibility.
Answering to your concerns, I don't see any harm if REST APIs also subscribe to a queue as long as you can maintain both of them i.e changes to message does not have any impact of APIs and you have proper fallback and Eventual consistency mechanism in place. you can check discussion . There are already few project which tried it such as nakadi and ponte.
So It all depends on your service's communication behaviour to choose between REST APIs and Event-Based design Or Both.
What you do is based on your requirement you can choose REST APIs where you see synchronous behaviour between services
and go with Event based design where you find services needs asynchronous behaviour, there is no harm combining both also.
Ideally for inter-process communication protocol it is better to go with messaging and for client-service REST APIs are best fitted.
Check the Communication style in microservices.io
REST based Architecture
Advantage
Request/Response is easy and best fitted when you need synchronous environments.
Simpler system since there in no intermediate broker
Promotes orchestration i.e Service can take action based on response of other service.
Drawback
Services needs to discover locations of service instances.
One to one Mapping between services.
Rest used HTTP which is general purpose protocol built on top of TCP/IP which adds enormous amount of overhead when using it to pass messages.
Event Driven Architecture
Advantage
Event-driven architectures are appealing to API developers because they function very well in asynchronous environments.
Loose coupling since it decouples services as on a event of once service multiple services can take action based on application requirement. it is easy to plug-in any new consumer to producer.
Improved availability since the message broker buffers messages until the consumer is able to process them.
Drawback
Additional complexity of message broker, which must be highly available
Debugging an event request is not that easy.
I'm in the middle of a project where we will migrate a major software system based on a larger set of custom made technologies to be based on OSGi services. For this we will likely need a some sort of message bus that plays nice with OSGi services.
Sync and ASync delivery
Point-to-point only
Guaranteed delivery - preferable with persistence via files
Strict message ordered from the same client (Async mode), but necessarily from different clients
Support for process-to-process and node-to-node nice but not strictly required
Open source solutions will be preferred, but not required.
I have looked at eventbus (as recommended in https://stackoverflow.com/a/1953453/796559), but that does not seem to work well.
So the question is, which technologies match the above?
Tonny,
Having just come from a very similar, and successful project, please let me share my experience with you to save you some time and your company some money. First and foremost, ESB's were a really good idea 8 years ago when they were proposed. And, they solved an important problem: how do you define a business problem in a way that those pesky coders will understand? The goal was to develop a system that would allow a business person to create a software solution with little or no pesky developer interaction needed that would suck up money better spent on management bonuses.
To answer this the good folks at many organizations came up with JBI, BPMN and a host of other solutions that let business folks model the business processes they wanted to "digitize". But really, they were all flawed at a very critical level: they addressed business issues, but not integration issues. As such, many of these implementations were unsuccessful unless done by some high-priced consultant, and even then your prospects were sketchy.
At the same time, some really smart folks in the very late 90's published a book called "Enterprise Integration Patterns" which identified over 60 design patterns used to solve common integration problems. Many of the folks performing ESB stuff realized that their problem wasn't one of business modelling. Rather the problem was how to integrate thier existing applications. To help solve this Michael Strachan and some really smart guys started the Apache Software Foundation Project "Camel". Camel is a strict implementation of Enterprise Integration Patterns in addition to a huge number of components designed to allow folks like you and I to hook stuff together.
So, if you think of your business process as simply a need to send data from one application to another to another, with the appropriate data transformations between, then Camel is your answer. Now, what if you want to base the "route" (a specified series of application endpoints you want to send data thorugh) off of a set of configurable rules in a database? Well, Camel can do that too! There's an endpoint for that! Anyhow, dont' do the traditional ESB, its old and busted. And Absolutely do the camel thing.
Please let me know if this helps.
The OSGi specification defines a component "Event Admin" which is a lightweight pub-sub event subsystem.
From the RFC0157:
Event Admin specifies a means for an event source to send events to
event listeners. Event sources can create events with a topic and
properties and request Event Admin to deliver the events to event
listeners which have declared interest in specific event topics and/or
property values. The event source can request synchronous (and
unordered) delivery or asynchronous (and ordered) delivery.
Compared to your requirements, it would score as follows:
Sync and ASync delivery: Check
Point-to-point only: No. Pub-Sub
Guaranteed delivery - preferable with persistence via files: NO
Strict message ordered from the same client (Async mode): YES
Support for process-to-process: if (process == OSGi service) -> Yes
Support for node-to-node: Not yet. The guys of
Distributed OSGi have been working on this, but I've not seen
anything concrete.
I like the concept of Camel, but recently decided to go for the (lighter) Event Admin as my requirements are limited. +1 to Mike on the Camel motivation. I'd look into it and then compare options before deciding.
Aren't you looking for an ESB? ServiceMix is a:
flexible, open-source integration container that unifies the features and functionality of Apache ActiveMQ, Camel, CXF, ODE, Karaf into a powerful runtime platform you can use to build your own integrations solutions. It provides a complete, enterprise ready ESB exclusively powered by OSGi.
iPOJO Event Admin Handlers are a nicer-to-use way to access the Event Admin service mentioned by #maasg.
looks like you are talking about an ESB here. If its the case, then you might have look at wso2 ESB. It is powered by apache synapse. it uses OSGi as the modular framework, so that you can add/remove features according to your requirement. There is a whole product stack from wso2 like message brokers, Business process servers (ODE), etc based on the same OSGi core platform.
disclaimer : I work for wso2.
This is more a theoretical question than a practical one, but given I undestand the principles of SOA I am still a bit unsure about if this can be applied to any app.
The usual example is where a client wants to know something from a server thus we implement a service that can provide that information given a client request, it can be stateless or statefull, etc.
But what happens when we want to be notified when something happens on the server, maybe we call a service to register a search and want to be notified when a new item arrives to the server that matches or search.
Of course that can be implemented using polling and leverage that using long timeouts, but I can not see a way in the usual protocols to receive events from the server without making a call to ask.
If you can point me to an example, or tell me an architecture that could support then you have made my day.
Have you considered pub-sub (ie; WS-Eventing, WS-Notification)? These are the usual means to pushing "stuff" to interested consumers/subscribers.
You want to use a Publish-Subscribe design. If you are using WCF checkout Programming WCF by Juval Lowery. In the appdendix he shows how to build a Pub-Sub system that is actually fully Per-Call. It doesn't even rely on CallbackContracts and keeping long running Channels open and so doesn't require any reconnection logic when communication is broken...let alone the need for any polling.