I am working on a program in golang, which I am sructuring based on Hexagonal Architecture. I think I have my head wrapped mostly around the idea, but there is something I just can't figure out.
The function of the program is to monitor multiple IP cameras for alarm events, which a receiver can receive a live stream of alarm events over a HTTP2.0 PUSH REQUEST. (Just in-case thats not the technical term, my service establishes a TCP/HTTP connection from a GET request and keeps it open, and when the cameras triggers an alarm event, the camera pushes it back to the services)
Layers of Architecture
Adaptors
HTTP Handler
In-memory JSON Store
Ports
DeviceService Interface
EventService Interface
DeviceRepo Interface
EventRepo Interface
Services
DeviceService
EventService
Domain
DeviceDomain
EventDomain
The user adds a device to the system via API, the request includes the desired monitoring schedule (When the receiver should start and stop daily) and url.
A scheduler is responsible to periodically checking if a receiver is meant to be started based on its schedule. If it's meant to be running for a device it starts a receiver for that device.
The receiver establishes connection to the IP camera and loops over the alarm event stream processing the alarm events and passing them to the EventService.
The EventService receives the event, and is responsible for handling the event, based on the domain logic, and decides to send an email or ignore it. It also saves all events to the eventrepo.
The two parts of code i'm not sure where they sit is the scheduler and receiver. So should they be;
a. Both in the same package and placed at the Adaptors layer
b. The receiver in the Adaptors layer and the scheduler in the Service layer
c. Both scheduler and receivers in the Service layer?
I am just confused, as the receiver isn't started by the user directly, but started by a running loop which continually checks a condition. But I also might have different receivers for different brands of cameras. Which is an implementation detail, which means the receiver should be in the Adaptors layer. Which makes me think option b is best.
I'm possibly over thinking it, but let me know what you all think the best option is or suggest a better one.
If it can help you, my design would be as follow:
Driver actors:
Human User: Interacts with the app using a driver port: "for adding devices"
Device (IP camera): Sends alarm events to the app using another driver port: "for receiving alarm events"
Driven actors:
Device (IP camera): The app interacts with the device using the driven port "for checking device", in order to start and stop it daily, according to the schedule of the device.
Warning Recipients: The app sends an email to them when an alarm event is received and it is not ignored.
Alarm Event Store: For persisting the alarm events the app receives.
The app ("Alarm Monitor") does the following business logic:
Maintains a collection of devices it has to monitor ("for adding devices").
It has a "worker" (the scheduler) that periodically checks the devices status and starts/stops them according to the schedule of the device.
It handles alarm events received from the devices. When an alarm event is received, the app either sends an email or ignore it. And stores the event in a repository.
So for me:
The scheduler is part of the business logic.
The receiver is the adapter of a device. It deels with http stuff.
Here is the picture:
"A scheduler is responsible to periodically checking if a receiver is meant to be started based on its schedule"
Ultimately it doesn't really matter to the application whether a human presses an "autoStartReceivers" button peridically or it's done by a scheduling process. Therefore that's an infrastructure concern and the scheduler is a driver adapter. You'd probably have a ReceiverService.autoStartReceivers service command that would be invoked by the scheduler periodically.
Now for the Receiver I'd say it depends on the implementation. If the Receiver doesn't know about infrastructure/vendor-specific details, but only does coordination then it may belong to the application/service layer.
For instance perhaps the receiver works with an abstract EventSource (HTTP, WebSockets, etc.) and uses an EventDecoder (vendor-specific) to adapt events and then relays them to an EventProcessor then it really only is doing orchestration. The EventSource & EventDecoder would be adapters. However if the Receiver knows about specific infrastructure details then it becomes an adapter.
Ultimately all the above is supporting logic for your core domain of event processing. The core domain logic wouldn't really care how events were captured and probably wouldn't care either how resulting actions are carried on. Therefore, your core domain in it's most simplistic form is probably actions = process(event) pure functions.
a. Both in the same package and placed at the Adaptors layer
b. The receiver in the Adaptors layer and the scheduler in the Service layer
c. Both scheduler and receivers in the Service layer?
The receiver and scheduler are both adapters. I don't think that they must be placed in the same package, but you can do that. So a is the best answer for me, because...
The receiver connects your application with an external device - the ip camara. Thus the receiver is an adapter for the EventService port.
The scheduler indirectly manages the lifecycle of the receiver through the DeviceService port. It enables or disables an ip camara and this leads to a connect and disconnect of the receiver.
From the perspective of your application core the scheduler is just another adapter that tells the DeviceService port to enable or disable some ip camara. This could also be done by a user who clicks on a button in the UI. The scheduler is just a technical assistance for the user which executes tasks that the user wants based on a schedule. Thus the scheduler is also an adapter.
Related
tldr Why don't I receive PBT_APMRESUMEAUTOMATIC, PBT_APMRESUMESUSPEND, and PBT_APMSUSPEND as the payload to service events of type SERVICE_ACCEPT_POWEREVENT?
I'm trying to detect when a windows device as woken back up from sleep. I have a constellation of processes that interact via IPC, which includes both UI applications with an event handler function provided to RegisterClassEx and services using RegisterServiceCtrlHandlerExW.
My preference is to receive these events in a service. My understanding is that I can get SERVICE_ACCEPT_POWEREVENT in dwControlsAccepted, and can then distinguish specific kinds of power event by looking at the dwEventType parameter, as per these docs https://learn.microsoft.com/en-us/windows/win32/api/winsvc/nc-winsvc-lphandler_function_ex. However, I only ever receive PBT_APMPOWERSTATUSCHANGE, corresponding to fiddling with the power cord on the laptop. I expected to also receive some combination of PBT_APMRESUMEAUTOMATIC, PBT_APMRESUMESUSPEND, and PBT_APMSUSPEND.
When testing on the UI side, I do get WM_POWERBROADCAST events of any kind. Obviously I've missed some part of setup there. Again, the process that actually needs this info is a service, so I would have to IPC the event to a service if this is what ended up working.
For full credit, I also experimented with SERVICE_CONTROL_CONTINUE and SERVICE_CONTROL_PAUSE (enabled via SERVICE_ACCEPT_PAUSE_CONTINUE), but never receive these events at all. I had expected those to correlate with sleeping the laptop but apparently not.
When building an application before scaling to multiple micro services. You have a codebase consisting of services that are decoupled. IE a services no longer depends on another service, not even loosely via a interface. It receives input from a service via a message buss. It has a method receivePaymentRequest but its callee is not the Order service. Its invoked via the message bus, perhaps in the future on another server. But imagine theres no need to run multiple servers at this point.
a order services posts to the message bus payment-request event
the payment services picks up on this message
payment is completed
payment service send a payment-complete event message to the message bus
the order service picks up this message
I"m not thinking about the patterns that enable this to be fault tolerant. But instead when to use this approach since it adds a lot of complexity. So please ignore what i've left out with regards to this
Correct? Is it stupid to implement it like such before scaling to microservices. How does this. Is SOA the step before actual microservices?
When should a class receive/publish on the message buss and when should it depend on a service as a class (even injected via a interface) ?
Problem:
Suppose there are two services A and B. Service A makes an API call to service B.
After a while service A falls down or to be lost due to network errors.
How another services will guess that an outbound call from service A is lost / never happen? I need some another concurrent app that will automatically react (run emergency code) if service A outbound CALL is lost.
What are cutting-edge solutions exist?
My thoughts, for example:
service A registers a call event in some middleware (event info, "running" status, timestamp, etc).
If this call is not completed after N seconds, some "call timeout" event in the middleware automatically starts the emergency code.
If the call is completed at the proper time service A marks the call status as "completed" in the same middleware and the emergency code will not be run.
P.S. I'm on Java stack.
Thanks!
I recommend to look into patterns such as Retry, Timeout, Circuit Breaker, Fallback and Healthcheck. Or you can also look into the Bulkhead pattern if concurrent calls and fault isolation are your concern.
There are many resources where these well-known patterns are explained, for instance:
https://www.infoworld.com/article/3310946/how-to-build-resilient-microservices.html
https://blog.codecentric.de/en/2019/06/resilience-design-patterns-retry-fallback-timeout-circuit-breaker/
I don't know which technology stack you are on but usually there is already some functionality for these concerns provided already that you can incorporate into your solution. There are libraries that already take care of this resilience functionality and you can, for instance, set it up so that your custom code is executed when some events such as failed retries, timeouts, activated circuit breakers, etc. occur.
E.g. for the Java stack Hystrix is widely used, for .Net you can look into Polly .Net to make use of retry, timeout, circuit breaker, bulkhead or fallback functionality.
Concerning health checks you can look into Actuator for Java and .Net core already provides a health check middleware that more or less provides that functionality out-of-the box.
But before using any libraries I suggest to first get familiar with the purpose and concepts of the listed patterns to choose and integrate those that best fit your use cases and major concerns.
Update
We have to differentiate between two well-known problems here:
1.) How can service A robustly handle temporary outages of service B (or the network connection between service A and B which comes down to the same problem)?
To address the related problems the above mentioned patterns will help.
2.) How to make sure that the request that should be sent to service B will not get lost if service A itself goes down?
To address this kind of problem there are different options at hand.
2a.) The component that performed the request to service A (which than triggers service B) also applies the resilience patterns mentioned and will retry its request until service A successfully answers that it has performed its tasks (which also includes the successful request to service B).
There can also be several instances of each service and some kind of load balancer in front of these instances which will distribute and direct the requests to an available instance (based on regular performed healthchecks) of the specific service. Or you can use a service registry (see https://microservices.io/patterns/service-registry.html).
You can of course chain several API calls after another but this can lead to cascading failures. So I would rather go with an asynchronous communication approach as described in the next option.
2b.) Let's consider that it is of utmost importance that some instance of service A will reliably perform the request to service B.
You can use message queues in this case as follows:
Let's say you have a queue where jobs to be performed by service A are collected.
Then you have several instances of service A running (see horizontal scaling) where each instance will consume the same queue.
You will use message locking features by the message queue service which makes sure that as soon one instance of service A reads a message from the queue the other instances won't see it. If service A was able to complete it's job (i.e. call service B, save some state in service A's persistence and whatever other tasks you need to be included for a succesfull procesing) it will delete the message from the queue afterwards so no other instance of service A will also process the same message.
If service A goes down during the processing the queue service will automatically unlock the message for you and another instance A (or the same instance after it has restarted) of service A will try to read the message (i.e. the job) from the queue and try to perform all the tasks (call service B, etc.)
You can combine several queues e.g. also to send a message to service B asynchronously instead of directly performing some kind of API call to it.
The catch is, that the queue service is some highly available and redundant service which will already make sure that no message is getting lost once published to a queue.
Of course you also could handle jobs to be performed in your own database of service A but consider that when service A receives a request there is always a chance that it goes down before it can save that status of the job to it's persistent storage for later processing. Queue services already address that problem for you if chosen thoughtfully and used correctly.
For instance, if look into Kafka as messaging service you can look into this stack overflow answer which relates to the problem solution when using this specific technology: https://stackoverflow.com/a/44589842/7730554
There is many way to solve your problem.
I guess you are talk about 2 topics Design Pattern in Microservices and Cicruit Breaker
https://dzone.com/articles/design-patterns-for-microservices
To solve your problem, Normally I put a message queue between services and use Service Discovery to detect which service is live and If your service die or orverload then use Cicruit Breaker methods
I've been aware of event sourcing, CQRS, DDD and micro services for a little while and I'm now at that point where I want to try and start implementing stuff and giving something a go.
I've been looking into the technical side of CQRS and I understand the DDD concepts in there. How both the write side handles commands from the UI and publishes events from it, and how the read side handles events and creates projections on them.
The difficulty I'm having is the communication & a handling events from service-to-service (both from a write to read service and between micro services).
So I want to focus on eventstore (this one: https://eventstore.com/ to be less ambiguous). This is what I want to use as I understand it is a perfect for event sourcing and the simple nature of storing the events means I can use this for a message bus as well.
So my issue falls into two questions:
Between the write and the read, in order for the read side to receive/fetch the events created from the write side, am i right in thinking something like a catch up subscription can be used to subscribe to a stream to receive any events written to it or do i use something like polling to fetch events from a given point?
Between micro services, I am having an even harder time... So when looking at CQRS tutorials/talks etc... they always seem to talk with an example of an isolated service which receives commands from the UI/API. This is fine. I understand the write side will have an API attached to it so the user can interact with it to perform commands. E.g. create a customer. However... say if I have two micro services, e.g. a order micro service and an shipping micro service, how does the shipping micro service get the events published from the order micro service. Specifically, how does those customer events, translate to commands for the shipping service.
So let's take a simple example of: - Command created from the order's API to place an order. - A OrderPlacedEvent is published to the event store. How does the shipping service listen and react to this is it need to then DispatchOrder and create ain turn an OrderDispatchedEvent.
Does the write side of the shipping microservice then need to poll or also have a catch up subscription to the order stream? If so how does an event get translated to an command using DDD approach?
something like a catch up subscription can be used to subscribe to a stream to receive any events written to it
Yes, using catch-up subscriptions is the right way of doing it. You need to keep the stream position of your subscription persisted somewhere as well.
Here you can find some sample code that works. I am not posting the whole snippet since it is too long.
The projection service startup flow is:
Load the checkpoint (first time ever it would be the stream start)
Subscribe to the stream from that checkpoint
The runtime flow will then be:
The subscription will then call the function you provide when it receives an event. There's some plumbing there to do, like if you subscribe to $all, you need to filter out system events (it will be easier in the next version of Event Store)
Project the event
Store the new checkpoint
If you make your projections idempotent, you can store the checkpoint from time to time and save some IO.
how does the shipping micro service get the events published from the order micro service
When you build a brand new system and you have a small team working on all the components, you can make a shortcut and subscribe to domain events from another service, as you'd do with projections. Within the integration context (between the boxes), ordering should not be important so you can use persistent subscriptions so you won't need to think about checkpoints. Event Store will do it for you.
Be aware that it introduces tight coupling on the domain event schema of the originating service. Your contexts will have the Partnership relationship or the downstream service will be a Conformist.
When you move forward with your system, you might decide to decouple those contexts properly. So, you introduce a stable event API for the service that publishes events for others to consume. The same subscription that you used for integration can now instead take care of translating domain (internal) events to integration (external) events. The consuming context would then use the stable API and the domain model of the upstream service will be free in iterating on their domain model, as soon as they keep the conversion up-to-date.
It won't be necessary to use Event Store for the downstream context, they could just as well use a message broker. Integration events usually don't need to be persisted due to their transient nature.
We are running a webinar series about Event Sourcing at Event Store, check our web site to get on-demand access to previous webinars and you might find interesting to join future ones.
The difficulty I'm having is the communication & a handling events from service-to-service (both from a write to read service and between micro services).
The difficulty is not your fault - the DDD literature is really weak when it comes to discussing the plumbing.
Greg Young discusses some of the issues of subscription in the latter part of his Polygot Data talk.
Eventide Project has documentation that does a decent job of explaining the principles behind how the plumbing fits things together.
Between micro services, I am having an even harder time...
The basic idea: your message store is fundamentally a database; when the host of your microservice wakes up, it queries the message store for messages after some checkpoint, and then feeds them to your domain logic (updating its own local copy of the checkpoint as needed).
So the host pulls a document with events in it from the store, and transforms that document into a stream of handle(Event) commands that ultimately get passed to your domain component.
Put another way, you build a host that polls the database for information, parses the response, and then passes the parsed data to the domain model, and writes its own checkpoints.
I understand process managers (aska as sagas) consume events (and commands) and produce/send commands on the read side of CQRS.
I have three questions:
How do process managers generally get the events in CQRS implementations? Do they usually subscribe to something like an eventbus OR are they directly sent the events?
Does this delivery mechanism (no matter what it is) need to be reliable (at least once) delivery? It would seem a problem if events were to be missed (e.g. due to a crash).
Are there any examples where a process manager listens for events from ARs that it has not directly or indirectly sent a command to? E.g. just listening for specific events.
I am asking these question about event distribution with regards to the write side / domain model in CQRS, not the read / query side.
Thanks,
Ashley.
Yes, they register for the events they like to receive on the event bus.
Yes, it needs to be reliable. For, the event bus usually has a "at least once" delivery guarantee, i.e., it guarantees that each event is delivered to each registered endpoint at least once. The process managers also need to make sure that their state changes and the commands they send to the command bus are also stored reliably when they acknowledge the receipt of the event. In our own CQRS implementation, we use JEE transactions for that part.
Sure, I would even say this is the standard. A very simple example is some notification system (e.g., notification mails) when something in the domain happened.