Could anybody link me to any example of a client javascript example for subscribing to custom events of Hyperledger Sawtooth, raised from a custom transaction processor? I could not find any.
An application can react to each event and store event data for further analysis.
For Example :
A state delta processor could store state data in a reporting database for analysis and processing, which provides access to state information without the delay of requesting state data from the validator.
For more GitHub examples follow these :
The Sawtooth Supply Chain
The Sawtooth Marketplace repository
Related
Looking for some help on an application design. I am using spring framework and hosting application in AWS.
I am working on an enterprise Java Web application that is suppose to handle events when their trigger time is reached. For example, consumers can set an event to begin on 12/20/22 at 07:35 AM, and system is suppose to send a notification when that time is reached.
I can store these events in a database along with their trigger time and setup a Spring scheduler (#Scheduler) to run every minute and process events whose trigger time is reached. My only concern with this approach is, there could be hundreds/thousands of event to trigger at any minute, and it cannot be processed within one minute.
Is there any alternate way to design this? I don't know if Spring offers a feature where I could create these Event, and Frameworks trigger these events when trigger time is reached. In that way, I can stay away from managing Scheduling and Triggering part.
I am using AWS to host this applications, so another option I'm thinking towards is creating an AWS lambda for every such Event, and let AWS manage the triggering part. In that way, I can stay away from managing the triggers.
Let me know your views? Or If you came across similar problems and how you resolved that?
You can consider using spring-cloud-dataflow to manage this as tasks and streams.
You create a custom batch application that will use #Scheduled to check the your database when events are dure and then send events to a stream. You can use Spring Integration APIs to interact with RabbitMQ or Kafka topics.
The event should contain enough information needed to process the event.
You then have a stream application that produces the content and send via email or pass it on to a separate stream app that sends the email.
https://dataflow.spring.io/docs/stream-developer-guides/programming-models/
The flow will look something like:
:mail_events | message-processor | message-sender
You will configure property for mail_events to match the topic created and configured for you mail-event-batch application.
You can use Spring Cloud Data Flow to manage the mail-event-batch application as well.
You can scale each application https://dataflow.spring.io/docs/recipes/scaling/
I have discovery service: https://github.com/Naresh-Chaurasia/API-MicroServices-Kafka/tree/master/Microservices-CQRS-SAGA-Kafka/DiscoveryService
I have Product Service: https://github.com/Naresh-Chaurasia/API-MicroServices-Kafka/tree/master/Microservices-CQRS-SAGA-Kafka/ProductsService
I have API gateway: https://github.com/Naresh-Chaurasia/API-MicroServices-Kafka/tree/master/Microservices-CQRS-SAGA-Kafka/ApiGateway
Product Service and API gateway are registered with discovery service. I use API Gateway to access the Product Service.
I am following a course to implement CQRS for products service.
Under ProductService, I have src/main/java/com/appsdeveloperblog/estore/ProductsService/command/ProductAggregate.java
Here the ProductAggregate is Command of CRQS.
It has the following methods (Please refer to GitHub for more details):
#CommandHandler
public ProductAggregate(CreateProductCommand createProductCommand) throws Exception {
...
}
#EventSourcingHandler
public void on(ProductCreatedEvent productCreatedEvent) {
...
}
It also has src/main/java/com/appsdeveloperblog/estore/ProductsService/query/ProductEventsHandler.java, which persist the product in H2 db.
I have also implemented src/main/java/com/appsdeveloperblog/estore/ProductsService/query/ProductsQueryHandler.java, which is used to query the db.
Here the ProductsQueryHandleris Query of CRQS.
My Question is as follows
What i am failing to understand that how and when is the Publish Event generated, and when the message is put in Messaging queue.
Also, is it possible that after the data is persisted to Event Store, it is not stored in Read DB. If yes, then how can we synchronize the Read DB.
What i am failing to understand that how and when is the Publish Event generated, and when the message is put in Messaging queue.
It happens after the events are published into the event store.
There are lots of possible designs that you might use to copy events from the event store to the event handler on the query side. These would include
Having the application code copy the event onto the message queue, which the event handler subscribes to
Having the event handler pull batches of events from the event store on a schedule
Having the event handler pull events from the event store, but using the message queue to announce that there are new messages to pull.
is it possible that after the data is persisted to Event Store, it is not stored in Read DB.
Yes. How common that is will depend on... well, really it mostly depends on how much you invest in reliability.
This is why the pull model tends to be popular - the read process can keep track of which events it has seen, and ask for the next batch of messages after X - where X is a time stamp, or a sequence number, or something.
Warning: if you are trying to roll your own event store, getting these details right can be tricky. Unless the details of the event store are part of your competitive advantage, you really want to buy reliability rather than trying to build it.
I've been aware of event sourcing, CQRS, DDD and micro services for a little while and I'm now at that point where I want to try and start implementing stuff and giving something a go.
I've been looking into the technical side of CQRS and I understand the DDD concepts in there. How both the write side handles commands from the UI and publishes events from it, and how the read side handles events and creates projections on them.
The difficulty I'm having is the communication & a handling events from service-to-service (both from a write to read service and between micro services).
So I want to focus on eventstore (this one: https://eventstore.com/ to be less ambiguous). This is what I want to use as I understand it is a perfect for event sourcing and the simple nature of storing the events means I can use this for a message bus as well.
So my issue falls into two questions:
Between the write and the read, in order for the read side to receive/fetch the events created from the write side, am i right in thinking something like a catch up subscription can be used to subscribe to a stream to receive any events written to it or do i use something like polling to fetch events from a given point?
Between micro services, I am having an even harder time... So when looking at CQRS tutorials/talks etc... they always seem to talk with an example of an isolated service which receives commands from the UI/API. This is fine. I understand the write side will have an API attached to it so the user can interact with it to perform commands. E.g. create a customer. However... say if I have two micro services, e.g. a order micro service and an shipping micro service, how does the shipping micro service get the events published from the order micro service. Specifically, how does those customer events, translate to commands for the shipping service.
So let's take a simple example of: - Command created from the order's API to place an order. - A OrderPlacedEvent is published to the event store. How does the shipping service listen and react to this is it need to then DispatchOrder and create ain turn an OrderDispatchedEvent.
Does the write side of the shipping microservice then need to poll or also have a catch up subscription to the order stream? If so how does an event get translated to an command using DDD approach?
something like a catch up subscription can be used to subscribe to a stream to receive any events written to it
Yes, using catch-up subscriptions is the right way of doing it. You need to keep the stream position of your subscription persisted somewhere as well.
Here you can find some sample code that works. I am not posting the whole snippet since it is too long.
The projection service startup flow is:
Load the checkpoint (first time ever it would be the stream start)
Subscribe to the stream from that checkpoint
The runtime flow will then be:
The subscription will then call the function you provide when it receives an event. There's some plumbing there to do, like if you subscribe to $all, you need to filter out system events (it will be easier in the next version of Event Store)
Project the event
Store the new checkpoint
If you make your projections idempotent, you can store the checkpoint from time to time and save some IO.
how does the shipping micro service get the events published from the order micro service
When you build a brand new system and you have a small team working on all the components, you can make a shortcut and subscribe to domain events from another service, as you'd do with projections. Within the integration context (between the boxes), ordering should not be important so you can use persistent subscriptions so you won't need to think about checkpoints. Event Store will do it for you.
Be aware that it introduces tight coupling on the domain event schema of the originating service. Your contexts will have the Partnership relationship or the downstream service will be a Conformist.
When you move forward with your system, you might decide to decouple those contexts properly. So, you introduce a stable event API for the service that publishes events for others to consume. The same subscription that you used for integration can now instead take care of translating domain (internal) events to integration (external) events. The consuming context would then use the stable API and the domain model of the upstream service will be free in iterating on their domain model, as soon as they keep the conversion up-to-date.
It won't be necessary to use Event Store for the downstream context, they could just as well use a message broker. Integration events usually don't need to be persisted due to their transient nature.
We are running a webinar series about Event Sourcing at Event Store, check our web site to get on-demand access to previous webinars and you might find interesting to join future ones.
The difficulty I'm having is the communication & a handling events from service-to-service (both from a write to read service and between micro services).
The difficulty is not your fault - the DDD literature is really weak when it comes to discussing the plumbing.
Greg Young discusses some of the issues of subscription in the latter part of his Polygot Data talk.
Eventide Project has documentation that does a decent job of explaining the principles behind how the plumbing fits things together.
Between micro services, I am having an even harder time...
The basic idea: your message store is fundamentally a database; when the host of your microservice wakes up, it queries the message store for messages after some checkpoint, and then feeds them to your domain logic (updating its own local copy of the checkpoint as needed).
So the host pulls a document with events in it from the store, and transforms that document into a stream of handle(Event) commands that ultimately get passed to your domain component.
Put another way, you build a host that polls the database for information, parses the response, and then passes the parsed data to the domain model, and writes its own checkpoints.
Currently I'm using Socket.io / SignalR to emit an event from my backend message queue system, whenever new data is incoming. That way I can setup an event handler in my React application and update the relay cache from within the event handler.
It does not seem like the most Graphql ish way to do things, so I was playing a bit around with pre-RFC live queries implementations, where you observed data changes in reactive data stores pushed it to the graphql server, and further to the client using websockets... with some rather complex custom code... obviously graphql is not ready for real live queries (not polling)
A few lines further down it says:
When building event-based subscriptions, the problem of determining what should trigger an event is easy, since the event defines that explicitly. It also proved fairly straight-forward to implement atop existing message queue systems.
Which leads me to my question. How can you (in a graphql way) best trigger graphql subscriptions when a new event is incoming to your backend message queue application and you need to reflect this new data in the ui in realtime - let's say each second? I'm not talking about triggering the event in the frontend/client or polling ever x seconds like you usually see when talking about subscriptions.
Not sure it's relevant but I'm using Relay Modern as my preferred graphql client.
Here's some ideas that might work if I get a little help to understand in general how to trigger/call a subscription without a mutation.
Backend worker / message queue "A" receives new incoming event with some device data. It uses either SignalR, or other pubsub (redis/socket.io/?) to notify the graphql server "B" (which subscribes to the event) about a new event has happened. The graphql server then trigger/execute the subscription and the frontend react relay application "C" automatically updates, since it has a relay subscription defined. This would be ideal, right? but how to trigger subscription on the graphql server?
Simply use Socket.io/SignalR to emit events from backend worker / message queue "A" on incoming data, subscribe and handle the event in the frontend "B", and then programically calling the subscription from within the Socket.io/SignalR event handler (if such a thing, directly calling a subscription, is even possible?). But then the only improvement from using subscriptions, instead of pure Socket.io/SignalR will be that I have moved the updating of the relay cache/store from the handler to the subscription. Not a big improvement, if any. But the manual update of the cache/store is really cumbersome, although not that hard :/
How do people handle real streaming live (device) data with signalr, and why is all realtime articles/examples just repeating the same old simple chat application, where the ui just updates after a user makes a click event? Is graphql not suited yet for dealing with a stream of frequently incoming device data in realtime? I understand why live queries was delayed after playing with implementing them myself, but without them, REAL realtime data updates and push it from the server to the frontend?
If I want to develop a node.js application for a Hyperledger Fabric Composer business network, it is necessary to define (in the model file) events that are emitted, whenever a transaction takes place. Otherwise, the node.js application is not "informed" about those transactions (see https://hyperledger.github.io/composer/latest/business-network/publishing-events.html).
Defining the events (in the model file ... and emitting them in the respective transaction processor functions) makes it possible for the node.js application to subscribe to those events (and therefore to be informed about transactions happening.)
So far I understand it.
My question is the following:
When I use the composer rest server (i.e. the automatically generated node.js application) instead of developing my "own" node.js application, do I still have to define the events for the transactions defined in the model file (.cto)?
Or is this not necessary because the composer rest server does not use those events anyway?
You would still need to define events in your model, then publish them in your transaction code (and subsequently consume them (subscribe) them from a client - whether composer-client or websockets etc. So regardless of whether you're using the REST APIs, Composer client APIs or even the CLI.
So if you POST a transaction from your REST client (eg. browser)- which sends it to the REST server - you must have defined an event (in your model) AND furthermore, your transaction logic would have to emit that event - for any listener to process it.