I'm working on an application for managing bank credit cards.
CQRS and Event Sourcing architecture was chosen for the app.
The most important aggregate in the app is CreditCard which controls the credit card lifecycle.
It looks something like:
class CreditCard {
private int status;
public void activate() {...}
public void deactivate () {...}
...
}
Its activate and deactivate methods protects credit card invariants and publish CardActivatedEvent and CardDeactivatedEvent, respectively, if the invocation of the method succeeds.
We store these events in the event store for later aggregate reconstruction on the command side.
We apply these events to various views.
We use these events to notify other third party systems.
All good for now.
Recently, we got a new requirement to charge all active credit cards on monthly basis.
My first instinct was, ok we can add charge method to the same CreditCard aggregate.
This method can check some invariants relevant to charging. Like, is the card in correct status for charging, was it charged already, etc.
On successful invocation, this method can publish CardChargedEvent.
Then we can create some process manager which will once per month query view side for active credit cards to get their IDs.
Having these IDs, the process manager can issue multiple charge commands (one per credit card aggregate) to the command side.
For each charge command received, the command side will reconstruct CreditCard aggregate object and call it's charge method.
The only problem is that this approach looks quite inefficient. Especially regarding database roundtrips on the command side (one read and one write per aggregate instance).
If we take into the equation that we can easily have 100k plus credit cards in our app, this roundtrip overhead starts looking to me as a bit of a problem.
Does anyone have any experience with batch operations on CQRS/ES systems?
Is my concern valid?
What to do in such cases?
How you implement batches in CQRS systems?
One alternative that pops to my mind is that for charging use case I ditch CQRS/ES/DDD principles, and implement the whole thing using stored procedures on one of our view databases. This procedure can search for suitable credit cards in the credit card view table and populate the "to be charged queue" table with records found. Then I can have some external process that reads this second table and do whatever it needs to do.
Recently, we got a new requirement to charge all active credit cards on monthly basis. My first instinct was, ok we can add charge method to the same CreditCard aggregate.
I think this is where the design flaw happens.
Your CreditCard aggregate was designed with a specific use case in mind:
The most important aggregate in the app is CreditCard which controls the credit card lifecycle.
Charging the credit card is not part of the credit card lifecycle. Whether it happens or not depends on the credit card state, but charging (successfully) the credit card will not change the state of your domain object. It should not interact with the CreditCard domain aggregate, as the purpose of your aggregate is to enforce business rules when changing your state. You should ask yourself: what aggregate is changed when charging a credit card ?
The answer to this question depends on the rest of your domain model and business cases, but it has more to do to with stuff like account balance or credit authorization than the card itself. You could implement like this:
A batch process working monthly would query your CreditCard aggregate for active cards, then try to charge all customers for their monthly fees by sending a command to the AccountBalance aggregate ;
The AccountBalance aggregate would raise a BalanceChangedEvent if the customer has enough money, or a CreditAuthorizationRequiredEvent if not, temporarily freezing the account until the credit was authorized or rejected ;
A CreditAuthorization aggregate could either allow or deny the credit, based on credit allowance business rules, raising events accordingly ;
The AccountBalance aggregate would unfreeze the account, changing the balance or not based on the outcome, eventually raising or not the BalanceChangedEvent ;
The CreditCard aggregate would register to the CreditDeniedEvent to deactivate the credit card because the customer was not able to pay the fees ;
... and so on ...
Related
I am new to Event Sourcing and I have encountered an example which I am not quite sure the pros and cons of different approaches.
Let's say this is a bank example, I have three entities Account, Deposit and Transfer.
My idea is, when a use deposits, command bank.deposit will create two events:
deposit.created and account.deposited. Can I or should I include the deposit.created event uuid in account.deposited as a reference?
Taking to the next step, if later the bank has a transfer feature, should I made a separate event account.transfer_received or I should created a more general event account.credited to be used by both deposit and transfer?
Thanks in advance.
A good article to review is Nobody Needs Reliable Messaging. One key observation is that you often need identifiers at the domain level.
For instance, when I look at my bank account, and see that the account history includes a specific deposit, there is an identifier for the deposit that is reported in the view.
If you imagine it from an event sourced perspective, before the deposit the balance was X, and the history did not include deposit 12345; after processing the deposit, the balance was X+Y and deposit 12345 was in the account history.
(This means, among other things, that if a second copy of deposit 12345 were to appear, the domain model would know to ignore it even if the identifier for the event were different).
Now, there are reasons that you might want to keep various message ids around. See Hohpe's work on Enterprise Integration Patterns; in particular Correlation Identifier.
should I made a separate event
Usually. "Make the implicit, explicit". The fact that two events happen to have similar representations is not a reason to blur them when the ubiquitous language distinguishes the two.
It's somewhat analogous, in motivation, to providing a task based ui or eschewing the user of generic repositories.
I am learning to develop microservices using DDD, CQRS, and ES. It is HTTP RESTful service. The microservices is about online shop. There are several domains like products, orders, suppliers, customers, and so on. The domains built in separate services. How to do the validation if the command payload relates to other domains?
For example, here is the addOrderItemCommand payload in the order service (command-side).
{
"customerId": "CUST111",
"productId": "SKU222",
"orderId":"SO333"
}
How to validate the command above? How to know that the customer is really exists in database (query-side customer service) and still active? How to know that the product is exists in database and the status of the product is published? How to know whether the customer eligible to get the promo price from the related product?
Is it ok to call API directly (like point-to-point / ajax / request promise) to validate this payload in order command-side service? But I think, the performance will get worse if the API called directly just for validation. Because, we have developed an event processor outside the command-service that listen from the event and apply the event to the materalized view.
Thank you.
As there are more than one bounded contexts that need to be queried for the validation to pass you need to consider eventual consistency. That being said, there is always a chance that the process as a whole can be in an invalid state for a "small" amount of time. For example, the user could be deactivated after the command is accepted and before the order is shipped. An online shop is a complex system and exceptions could appear in any of its subsystems. However, being implemented as an event-driven system helps; every time the ordering process enters an invalid state you can take compensatory actions/commands. For example, if the user is deactivated in the meantime you can cancel all its standing orders, release the reserved products, announce the potential customers that have those products in the wishlist that they are not available and so on.
There are many kinds of validation in DDD but I follow the general rule that the validation should be done as early as possible but without compromising data consistency. So, in order to be early you could query the readmodel to reject the commands that couldn't possible be valid and in order for the system to be consistent you need to make another check just before the order is shipped.
Now let's talk about your specific questions:
How to know that the customer is really exists in database (query-side customer service) and still active?
You can query the readmodel to verify that the user exists and it is still active. You should do this as a command that comes from an invalid user is a strong indication of some kind of attack and you don't want those kind of commands passing through your system. However, even if a command passes this check, it does not necessarily mean that the order will be shipped as other exceptions could be raised in between.
How to know that the product is exists in database and the status of the product is published?
Again, you can query the readmodel in order to notify the user that the product is not available at the moment. Or, depending on your business, you could allow the command to pass if you know that those products will be available in less than 24 hours based on some previous statistics (for example you know that TV sets arrive daily in your stock). Or you could let the customer choose whether it waits or not. In this case, if the products are not in stock at the final phase of the ordering (the shipping) you notify the customer that the products are not in stock anymore.
How to know whether the customer eligible to get the promo price from the related product?
You will probably have to query another bounded context like Promotions BC to check this. This depends on how promotions are validated/used.
Is it ok to call API directly (like point-to-point / ajax / request promise) to validate this payload in order command-side service? But I think, the performance will get worse if the API called directly just for validation.
This depends on how resilient you want your system to be and how fast you want to reject invalid commands.
Synchronous call are simpler to implement but they lead to a less resilient system (you should be aware of cascade failures and use technics like circuit breaker to stop them).
Asynchronous (i.e. using events) calls are harder to implement but make you system more resilient. In order to have async calls, the ordering system can subscribe to other systems for events and maintain a private state that can be queried for validation purposes as the commands arrive. In this way, the ordering system continues to work even of the link to inventory or customer management systems are down.
In any case, it really depends on your business and none of us can tell you exaclty what to do.
As always everything depends on the specifics of the domain but as a general principle cross domain validation should be done via the read model.
In this case, I would maintain a read model within each microservice for use in validation. Of course, that brings with it the question of eventual consistency.
How you handle that should come from your understanding of the domain. Factors such as the length of the eventual consistency compared to the frequency of updates should be considered. The cost of getting it wrong for the business compared to the cost of development to minimise the problem. In many cases, just recording the fact there has been a problem is more than adequate for the business.
I have a blog post dedicated to validation which you can find here: How To Validate Commands in a CQRS Application
Scenario:
I have 2 Microservices (which both use CQRS + Event Sourcing internally)
Microservice 1 manages Contacts (= Aggregate Root)
Microservice 2 manages Invoices (= Aggregate Root)
The recipient of an invoice must be a valid contact.
CreateInvoiceCommand:
{
"content": "my invoice content",
"recipient": "42"
}
I now read lot's of times, that the write side (= the command handler) shouldn't call the read side.
Taking this into account, the Invoices Microservice must listen to all ContactCreated and ContactDeleted events in order to know if the given recipient id is valid.
Then I'd have thousands of Contacts within the Invoices Microservice, even if I know that only a few of them will ever receive an Invoice.
Is there any best practice to handle those scenarios?
The recipient of an invoice must be a valid contact.
So the first thing you need to be aware of - if two entities are part of different aggregates, you can't really implement "apply a change to this entity only if that entity satisfies a specification", because that entity could change between the moment you evaluate the specification and the moment you perform the write.
In other words - you can only get eventual consistency across an aggregate boundary.
The aggregate is the authority for its own state, but everything else (for example, the contents of the command message), it pretty much has to accept that some external authority has checked the data.
There are a couple approaches you can take here
1) You can blindly accept that the recipient specified in the command is valid.
2) You can try to verify the validity of the recipient from some external authority (aka: a read model of some other aggregate) between receiving it from the untrusted source and submitting it to the domain model.
3) You can blindly accept the command as described, but treat the invoice as provisional until the validity of the recipient is confirmed. That means there is a second command to run on the invoice that certifies the recipient.
Note - from the point of view of the model, these different commands are equivalent, but at the application layer they don't need to be -- you can restrict access to the command to trusted sources (don't make it part of the public api, require authorization that is only available to trusted sources, etc).
Approach #3 is the most microservicy, as the two commands can be separated in time -- you can accept the CreateInvoice command as soon as it arrives, and certify the recipient asynchronously.
Where would you put approach 4), where the Invoices Microservice has it's own Contacts Store which gets updated whenever there's a ContactCreated or ContactDeleted event? Then both entities are part of the same service and boundary. Now it should be possible to make things consistent, right?
No. You've made the two entities part of the same service, but the problem was never that they were in different services, but that they are in separate aggregates -- meaning we can be changing the entity states concurrently, which means that we can't ensure that they are immediately synchronized.
If you wanted immediate consistency, you need a model that draws your boundaries differently.
For instance, if the invoice entities were modeled as part of the Contacts aggregate, then the aggregate can ensure the invariant that new invoices require a valid recipient -- the domain model uses the copy of the state in memory to confirm that the recipient was valid when we loaded, and the write into the book of record verifies that the book of record hadn't changed since the load happened.
The write of the aggregate state is a compare-and-swap in the book of record; if some concurrent process had invalidated the recipient, the CAS operation would fail.
The trade off, of course, is that any change to the Contact aggregate would also cause the invoice to fail; concurrent editing of different invoices with the same recipient goes out the window.
Aggregates are all or nothing; they aren't separable.
Now, one out might be that your Invoice aggregate has a part that must be immediately consistent with the recipient, and another part where eventually consistent, or even inconsistent, is acceptable. In which case your goal is to refactor the model.
The recipient of an invoice must be a valid contact.
This is a business rule. The question should be asked, what does this business rule mean for my application? Who should take responsibility for implementing this rule, or can the responsibility be shared?
One possibility is that, yes, the business rule is about invoices so it should be the responsibility of the Invoices Service to implement it.
However, the business rule is really about the creation of invoices. And the owner of invoice creation in your architecture is, strangely, not the Invoices Service. The reason for this is that the name of the command is CreateInvoiceCommand.
Let's think about this - the Invoices Service will never just create an invoice on its own. It just provides the capability. In this architecture, the actual owner of invoice creation is the sender of the command.
Using this line of reasoning, if the business rule is saying that invoice creation cannot happen against an invalid recipient, then it becomes the responsibility of the command sender to ensure this business rule is implemented.
This would be a very different scenario if, rather than receiving a command, the Invoices Service subscribed to events. As an example, an event called WidgetSold. In this scenario, the owner of invoice creation clearly would be the Invoicing service, and so the business rule would be implemented there instead.
If the user clicks the create invoice for contact 42 button, it's the
user's responsibility to take care that contact 42 exists
Yes, that is correct. The user's intention is to create an invoice. The business rules around invoice creation should, therefore, be enforced at this point. How this happens (or whether this happens at all) is a different question.
But what if the user doesn't care? Then it would create an invoice
with an invalid recipient id.
Also correct. As you say, there are side-effects to this approach, one of which is that you can end up with inconsistent data across your system. That is one of the realities of SOA.
Isn't this somehow similar to this: The Invoice has a currencyCode
property, it's a String.
I don't know if I agree or not. Is asking is this a valid ISO currency? different to asking is entity 42 valid according to another system?. I would think so.
Isn't it kinda the same as given recipient is not null and is valid
according to my Contacts Database?
I agree that in reality, you could implement this validation in the service. I am just saying that I don't think it's the right place for it. If you wanted to do this, you would have to either call out the another service or store all contacts locally, as you framed your question originally. I think it's simpler to just do it outside of the service.
I think that the answer depends on how resilient you want the system to be, that is, how to handle the situation in wich the Contacts Microservice is down (not responding or very slow).
1. You want to be very resilient
If the Contacts Microservice is down, you want to be able to emit invoices for some (maybe most) of the contacts. In this case you listen to the ContactCreated and ContactDeleted and maintain a (eventually consistent) local list of valid contacts; they should be named accordingly to the Ubiquitous language in this bounded context, like Payers (or something like that). Then, in the Application layer, when building the CreateInvoiceCommand you check that Payer is valid and create the command.
2. You don't need to be resilient
If the Contacts Microservice is down, you refuse to generate invoices. In this case, when building the command you make a request to the Invoices Microservice API endpoint and verify that the Payer is valid.
In any case, you check for contact's validity before the command is dispatched.
I want to integrate recurring payment using Payeezy in codeigniter. I have implement the single time payment using curl and now i want to recurring payment with acknowledgement to update my DB.
I created a WordPress plugin for Payeezy that also handles recurring. You should be able to use the underlying PHP code for CodeIgniter.
https://wordpress.org/plugins/wp-payeezy-pay/
I can explain the process that will get you the least PCI compliance issues, and that's the token-based API.
1. Generate Token in Payment Form
So basically you'll use the Javascript API to generate your authorize token. An authorize token doesn't charge the card. It's for validating the card and returning a token for better PCI compliance. This API source code and explanation is here:
https://github.com/payeezy/payeezy_js
2. Post Form To Your Server for the Curl Call to FirstData
Then, once you have this token, you post it back to your controller file with a standard form post, but remove the name attribute on your credit card number and credit card CVC fields so that these do not post to your server. Note that you'll need to store this data (but not card number and CVC) because on refunds (and subscription cancellations) you'll need to reply back with the last purchase token, cardholder name, card type, card expiration date, amount spent, and currency code. You may wonder why FirstData/PayEezy is asking you to store cardholder name, card type, and card expiration date. Well, there's a perfectly good explanation for that. Your call center may need that detail for troubleshooting an issue over the phone with a customer. Also, you need that for refunds. And, most importantly, if you're doing a recurring subscription payment, your code needs to look at the expiration date ahead of time before charging because the API call will fail if the card is past expiration. Last, because you're not storing the credit card number and credit card validation (CVC) code, you're going to be in stronger PCI compliance.
From there, since you are already familiar with the Curl process for a single-purchase, it's just a minor single field change (transaction_type becomes 'recurring') in the Curl to do the recurring. For anyone not familiar with the Curl process, it's explained here:
https://developer.payeezy.com/payeezy-api/apis/post/transactions-4
Also, for those unfamiliar people, you'll need to read up on how FirstData/PayEezy wants you to send in the Curl request with a special header that includes Content-Type: application/json, apikey, token, Authorization, nonce, and timestamp. You can see more detail about that here:
https://github.com/payeezy/payeezy_direct_API/blob/master/payeezy_php/example/src/Payeezy.php
(What I did to make that code simpler was intercept the Curl calls from that script into a log file so that I could make it much more straightforward in a single function instead of breaking it up into all these little functions. That made it far easier to understand what was going on.)
3. Switching Curl Call for Recurring Payments
So, as you discovered in your Curl call, you saw how to do a one-time purchase by setting the transaction_type to 'purchase'. For doing recurring, you set transaction_type to 'recurring'. You have to do that from the start. So, if I'm selling something for $29.99 monthly, the very first month charge needs to still be set to type 'recurring', as would any subsequent month.
4. Your Responsibilities for Recurring Payments
Now, this is where everyone gets hung up because it's poorly documented unless you check the PayEezy Developer Support Forum. For subscriptions, PayEezy doesn't have a system for setting payment plans with varying durations, nor setting up automatic (set-it-and-forget-it) subscriptions for you. (I think I read that they have something experimental on Apple Pay, but nothing else yet.) So, to achieve this, you have 2 choices:
Use Chargify.com. Unfortunately, though, this increases CPA (Cost Per Acquisition) of your product or service. You'll have to factor that in if you want to use that. This basically is a SaaS service that you send the transaction to and they handle the automatic subscription plan for you against FirstData/PayEezy.
Roll your own cron job solution. To do this, you basically take the Curl code for a single transaction, and change the transaction type from 'purchase' to 'recurring'. (Do that from the start -- don't start with 'purchase' on a recurring charge.) From there, it's up to you with your own cron job to check for product or service expiration terms, and then send the API call back off to FirstData/PayEezy for charging that card again with the 'recurring' transaction_type.
On either of those options, the customer never gets asked to enter in credit card data past the first time unless their card expires or unless you have some problem billing that card (like insufficient funds).
Of course, doing your own cron job route for the recurring payment has implications you'll need to prepare for:
Add some failsafe code so that you prevent the possibility of duplicate transactions, such as a database field.
Add some failsafe code such that if you have cancelled a subscription, you won't charge them again.
Add some failsafe code such that if they cancel their subscription, yet purchase it again as a subscription at a later time, that you do charge them again and don't block it from your other failsafe code.
Add some sort of grace period on your product or service such that even if you "say" that the term expired, you have like a 2 day grace period so that your API has a chance to do a renewal.
It's probably a good idea to email the customer before their renewal period so that they can make certain they have money in their account and have a way to cancel that charge (like call your office or call center, or have a link to click where you provide a way to cancel).
If their card has expired before the renewal, and you detect that in the warning email that comes before renewal, then you'll want to let them know this.
If their card has been declined for any reason at the point of renewal, then you'll want to let them know this and give them a link to go through the cart again to buy it again, or some other way to save that transaction in your code.
How To Do Subscription Cancellations / Stop Recurring Payments
To stop a recurring payment, you treat it just like a refund on a single purchase, but use the transaction ID of the last purchase. This is documented with this Curl example here:
https://developer.payeezy.com/payeezy-api/apis/post/transactions/%7Bid%7D-0
Look under "Refund" and choose Token.
I have a requirement to be able to accept different forms of payment within the same order - ie not just the usual credit card or paypal for the whole thing, but perhaps paypal for one item, cheque for another. I know this sounds quite crazy, but there is a good business reason for the requirement so I can't just push back.
The best way I can think of implementing it at the moment is to have kind of a hub page, where you can "launch off" into multiple flows for each of the payments by opening new windows. I can't figure out a way of doing this in a linear flow as for example you can't guarantee that a user will come back from paypal, so you'd then lose the user completely.
Is there a neater way of doing this that anyone can think of, or can anyone point me to an example of a site that does somethign similar for inspiration?
Even when opening several windows at once, there is no guarantee that the user will complete all payment methods. So you are most probably going to lose a few users or payments. Be sure to send automated e-mail follow-ups for missing payments to minimize this problem. The e-mails could contain links to your payment providers for easy accesss to their outstanding payment operations.
This is a difficult problem, but how many payment processors do you have to go offsite for? Should only be paypal.
In any case, I'd give the user all their payment options on one page, and let them fill in the amount for each processor or payment type. Then the next page would list those they chose, the amount for each, and a link to "Complete this payment".
The link would open in a new window.
You'll have to have a good back end and javascript, as well as user warnings so that the payment page gets updated as each payment is processed. Consider using popup dialogs to show that a payment has completed, or that the order has sat idle for more than 10-30 minutes without complete payment.
Also, consider sending emails and letting the user complete the payments through links in the emails. Send a new email each time a payment is completed, and a final email if all payments are complete and the order is moving forward.
Send an email one hour, and one day later for uncompleted orders with remaining payments required, that also give them the option of choosing different payment options for the remainder.
Email isn't best (lose more orders that way due to changing minds) but it's good for the type of transactions you're thinking about.
Personally, I'd do it like this:
Let the user fill their basket in the ususal way
Allow them to add payment types and amounts to a list (2nd basket almost)
When the payments balance against the basket, start processing the payments
For external sites, try a frame which has a progress indicator at the top.
In an ideal world it wouldn't be linear. But a lot of users might lose a spawned window, or get confused by the parallelism.
Better to stick to established IxD principals and rely on good feedback instead. Give the user control from the outset and keep it transparent.
Lastly, start the payment process with the most immediate (e.g. paypal) to reduce users giving up. (COD should come last!)
Hope this helps,
Tom
If possible, just separate your order into separate smaller orders based off of the payment selections of the user.
And don't do it linearly. If anything you could open up each payment processor in a separate window so that you maintain presence.
I would take an approach where the whole order is broken down into sub-orders for each of the necessary payment methods. You can load the PayPal portion, the check portion, etc. and process them separately. It's important for the user to know how much is being charged to each of their payment methods, so it makes sense in this case to present the whole order as broken down by payment method (versus displaying as a unified order).
Implementation would be easiest if it's always a certain subset of items that is forced to any payment method. If this differs by user, or if it's when the order reaches a certain amount, the situation could become much more complicated. Can you be more specific about your approach?
Processing Multiple Order Payments
Give the user the option to make a payment for a pending order using any of your payment types.
Let the user specify an (Amount <= [Order Total] - [Payments Received]), if that is part of your requirement.
If the order is still pending after you process the payment (see how below), take them back to step 1 to rinse and repeat.
How to store and process each payment made:
Use a Payments table to store all order payments, the PaymentMethod used and its Amount with its CurrencyCode.
When a payment is received for an order, store the payment and sum all received amounts converted into your base currency as [Payments Received].
If [Payments Received] >= [Order Total], mark the order as Paid. Or, if dealing with double-converted foreign exchange rates, check if it is correct to within a small-enough margin, eg 0.5%.
Optionally, convert any overpayment into prepaid credit for the client.