I am working on a package to process incoming payments from external entities and match them against existing "bills".
Each payment has a reference to to bill_id, so matching is easy. Bills are then flagged as paid.
Problems is payment may bounce and the contra-payment will be received in the same or, more likely, in a subsequent file. Logic is that a contra-payment must be reversing an already paid bill, while a regular payment must be for a "non paid" bill. How would you implement and enforce such a validation in SSIS, considering that the contra-payment might be in the same file as the payment it is reversing or in a successive file. How would you structure your data-flow?
Also critical is the need to report on bounced payments, to assess a kind of "default rate".
I'd like to hear the opinion of the experts.
Many thanks
Related
I am new to Event Sourcing and I have encountered an example which I am not quite sure the pros and cons of different approaches.
Let's say this is a bank example, I have three entities Account, Deposit and Transfer.
My idea is, when a use deposits, command bank.deposit will create two events:
deposit.created and account.deposited. Can I or should I include the deposit.created event uuid in account.deposited as a reference?
Taking to the next step, if later the bank has a transfer feature, should I made a separate event account.transfer_received or I should created a more general event account.credited to be used by both deposit and transfer?
Thanks in advance.
A good article to review is Nobody Needs Reliable Messaging. One key observation is that you often need identifiers at the domain level.
For instance, when I look at my bank account, and see that the account history includes a specific deposit, there is an identifier for the deposit that is reported in the view.
If you imagine it from an event sourced perspective, before the deposit the balance was X, and the history did not include deposit 12345; after processing the deposit, the balance was X+Y and deposit 12345 was in the account history.
(This means, among other things, that if a second copy of deposit 12345 were to appear, the domain model would know to ignore it even if the identifier for the event were different).
Now, there are reasons that you might want to keep various message ids around. See Hohpe's work on Enterprise Integration Patterns; in particular Correlation Identifier.
should I made a separate event
Usually. "Make the implicit, explicit". The fact that two events happen to have similar representations is not a reason to blur them when the ubiquitous language distinguishes the two.
It's somewhat analogous, in motivation, to providing a task based ui or eschewing the user of generic repositories.
I am learning to develop microservices using DDD, CQRS, and ES. It is HTTP RESTful service. The microservices is about online shop. There are several domains like products, orders, suppliers, customers, and so on. The domains built in separate services. How to do the validation if the command payload relates to other domains?
For example, here is the addOrderItemCommand payload in the order service (command-side).
{
"customerId": "CUST111",
"productId": "SKU222",
"orderId":"SO333"
}
How to validate the command above? How to know that the customer is really exists in database (query-side customer service) and still active? How to know that the product is exists in database and the status of the product is published? How to know whether the customer eligible to get the promo price from the related product?
Is it ok to call API directly (like point-to-point / ajax / request promise) to validate this payload in order command-side service? But I think, the performance will get worse if the API called directly just for validation. Because, we have developed an event processor outside the command-service that listen from the event and apply the event to the materalized view.
Thank you.
As there are more than one bounded contexts that need to be queried for the validation to pass you need to consider eventual consistency. That being said, there is always a chance that the process as a whole can be in an invalid state for a "small" amount of time. For example, the user could be deactivated after the command is accepted and before the order is shipped. An online shop is a complex system and exceptions could appear in any of its subsystems. However, being implemented as an event-driven system helps; every time the ordering process enters an invalid state you can take compensatory actions/commands. For example, if the user is deactivated in the meantime you can cancel all its standing orders, release the reserved products, announce the potential customers that have those products in the wishlist that they are not available and so on.
There are many kinds of validation in DDD but I follow the general rule that the validation should be done as early as possible but without compromising data consistency. So, in order to be early you could query the readmodel to reject the commands that couldn't possible be valid and in order for the system to be consistent you need to make another check just before the order is shipped.
Now let's talk about your specific questions:
How to know that the customer is really exists in database (query-side customer service) and still active?
You can query the readmodel to verify that the user exists and it is still active. You should do this as a command that comes from an invalid user is a strong indication of some kind of attack and you don't want those kind of commands passing through your system. However, even if a command passes this check, it does not necessarily mean that the order will be shipped as other exceptions could be raised in between.
How to know that the product is exists in database and the status of the product is published?
Again, you can query the readmodel in order to notify the user that the product is not available at the moment. Or, depending on your business, you could allow the command to pass if you know that those products will be available in less than 24 hours based on some previous statistics (for example you know that TV sets arrive daily in your stock). Or you could let the customer choose whether it waits or not. In this case, if the products are not in stock at the final phase of the ordering (the shipping) you notify the customer that the products are not in stock anymore.
How to know whether the customer eligible to get the promo price from the related product?
You will probably have to query another bounded context like Promotions BC to check this. This depends on how promotions are validated/used.
Is it ok to call API directly (like point-to-point / ajax / request promise) to validate this payload in order command-side service? But I think, the performance will get worse if the API called directly just for validation.
This depends on how resilient you want your system to be and how fast you want to reject invalid commands.
Synchronous call are simpler to implement but they lead to a less resilient system (you should be aware of cascade failures and use technics like circuit breaker to stop them).
Asynchronous (i.e. using events) calls are harder to implement but make you system more resilient. In order to have async calls, the ordering system can subscribe to other systems for events and maintain a private state that can be queried for validation purposes as the commands arrive. In this way, the ordering system continues to work even of the link to inventory or customer management systems are down.
In any case, it really depends on your business and none of us can tell you exaclty what to do.
As always everything depends on the specifics of the domain but as a general principle cross domain validation should be done via the read model.
In this case, I would maintain a read model within each microservice for use in validation. Of course, that brings with it the question of eventual consistency.
How you handle that should come from your understanding of the domain. Factors such as the length of the eventual consistency compared to the frequency of updates should be considered. The cost of getting it wrong for the business compared to the cost of development to minimise the problem. In many cases, just recording the fact there has been a problem is more than adequate for the business.
I have a blog post dedicated to validation which you can find here: How To Validate Commands in a CQRS Application
Scenario:
I have 2 Microservices (which both use CQRS + Event Sourcing internally)
Microservice 1 manages Contacts (= Aggregate Root)
Microservice 2 manages Invoices (= Aggregate Root)
The recipient of an invoice must be a valid contact.
CreateInvoiceCommand:
{
"content": "my invoice content",
"recipient": "42"
}
I now read lot's of times, that the write side (= the command handler) shouldn't call the read side.
Taking this into account, the Invoices Microservice must listen to all ContactCreated and ContactDeleted events in order to know if the given recipient id is valid.
Then I'd have thousands of Contacts within the Invoices Microservice, even if I know that only a few of them will ever receive an Invoice.
Is there any best practice to handle those scenarios?
The recipient of an invoice must be a valid contact.
So the first thing you need to be aware of - if two entities are part of different aggregates, you can't really implement "apply a change to this entity only if that entity satisfies a specification", because that entity could change between the moment you evaluate the specification and the moment you perform the write.
In other words - you can only get eventual consistency across an aggregate boundary.
The aggregate is the authority for its own state, but everything else (for example, the contents of the command message), it pretty much has to accept that some external authority has checked the data.
There are a couple approaches you can take here
1) You can blindly accept that the recipient specified in the command is valid.
2) You can try to verify the validity of the recipient from some external authority (aka: a read model of some other aggregate) between receiving it from the untrusted source and submitting it to the domain model.
3) You can blindly accept the command as described, but treat the invoice as provisional until the validity of the recipient is confirmed. That means there is a second command to run on the invoice that certifies the recipient.
Note - from the point of view of the model, these different commands are equivalent, but at the application layer they don't need to be -- you can restrict access to the command to trusted sources (don't make it part of the public api, require authorization that is only available to trusted sources, etc).
Approach #3 is the most microservicy, as the two commands can be separated in time -- you can accept the CreateInvoice command as soon as it arrives, and certify the recipient asynchronously.
Where would you put approach 4), where the Invoices Microservice has it's own Contacts Store which gets updated whenever there's a ContactCreated or ContactDeleted event? Then both entities are part of the same service and boundary. Now it should be possible to make things consistent, right?
No. You've made the two entities part of the same service, but the problem was never that they were in different services, but that they are in separate aggregates -- meaning we can be changing the entity states concurrently, which means that we can't ensure that they are immediately synchronized.
If you wanted immediate consistency, you need a model that draws your boundaries differently.
For instance, if the invoice entities were modeled as part of the Contacts aggregate, then the aggregate can ensure the invariant that new invoices require a valid recipient -- the domain model uses the copy of the state in memory to confirm that the recipient was valid when we loaded, and the write into the book of record verifies that the book of record hadn't changed since the load happened.
The write of the aggregate state is a compare-and-swap in the book of record; if some concurrent process had invalidated the recipient, the CAS operation would fail.
The trade off, of course, is that any change to the Contact aggregate would also cause the invoice to fail; concurrent editing of different invoices with the same recipient goes out the window.
Aggregates are all or nothing; they aren't separable.
Now, one out might be that your Invoice aggregate has a part that must be immediately consistent with the recipient, and another part where eventually consistent, or even inconsistent, is acceptable. In which case your goal is to refactor the model.
The recipient of an invoice must be a valid contact.
This is a business rule. The question should be asked, what does this business rule mean for my application? Who should take responsibility for implementing this rule, or can the responsibility be shared?
One possibility is that, yes, the business rule is about invoices so it should be the responsibility of the Invoices Service to implement it.
However, the business rule is really about the creation of invoices. And the owner of invoice creation in your architecture is, strangely, not the Invoices Service. The reason for this is that the name of the command is CreateInvoiceCommand.
Let's think about this - the Invoices Service will never just create an invoice on its own. It just provides the capability. In this architecture, the actual owner of invoice creation is the sender of the command.
Using this line of reasoning, if the business rule is saying that invoice creation cannot happen against an invalid recipient, then it becomes the responsibility of the command sender to ensure this business rule is implemented.
This would be a very different scenario if, rather than receiving a command, the Invoices Service subscribed to events. As an example, an event called WidgetSold. In this scenario, the owner of invoice creation clearly would be the Invoicing service, and so the business rule would be implemented there instead.
If the user clicks the create invoice for contact 42 button, it's the
user's responsibility to take care that contact 42 exists
Yes, that is correct. The user's intention is to create an invoice. The business rules around invoice creation should, therefore, be enforced at this point. How this happens (or whether this happens at all) is a different question.
But what if the user doesn't care? Then it would create an invoice
with an invalid recipient id.
Also correct. As you say, there are side-effects to this approach, one of which is that you can end up with inconsistent data across your system. That is one of the realities of SOA.
Isn't this somehow similar to this: The Invoice has a currencyCode
property, it's a String.
I don't know if I agree or not. Is asking is this a valid ISO currency? different to asking is entity 42 valid according to another system?. I would think so.
Isn't it kinda the same as given recipient is not null and is valid
according to my Contacts Database?
I agree that in reality, you could implement this validation in the service. I am just saying that I don't think it's the right place for it. If you wanted to do this, you would have to either call out the another service or store all contacts locally, as you framed your question originally. I think it's simpler to just do it outside of the service.
I think that the answer depends on how resilient you want the system to be, that is, how to handle the situation in wich the Contacts Microservice is down (not responding or very slow).
1. You want to be very resilient
If the Contacts Microservice is down, you want to be able to emit invoices for some (maybe most) of the contacts. In this case you listen to the ContactCreated and ContactDeleted and maintain a (eventually consistent) local list of valid contacts; they should be named accordingly to the Ubiquitous language in this bounded context, like Payers (or something like that). Then, in the Application layer, when building the CreateInvoiceCommand you check that Payer is valid and create the command.
2. You don't need to be resilient
If the Contacts Microservice is down, you refuse to generate invoices. In this case, when building the command you make a request to the Invoices Microservice API endpoint and verify that the Payer is valid.
In any case, you check for contact's validity before the command is dispatched.
I've looked at this question, which didn't have a suitable answer (basically recommends tokenisation). I'm intrigued as to why the fields for delivery address are mandatory as I'm assuming this isn't validated against. The billing address makes sense as this will be the address linked to the card and therefore worth validating. However, the delivery address presumably provides very little.
I can see that it's provided in the response from SagePay in relation to PayPal payments, presumably to check whether this has been altered when getting to PayPal, but as PayPal payments are optional, then if this is the only reason surely it too should be optional?
The reason I ask is that when providing a service rather than a tangible physical product, what should go in this? I'm assuming in most cases this would simply be the same as the billing address, but in the scenario that a 3rd party pays for the service, should the delivery address now be the address of the person receiving the service?
Fraud screening. Sage Pay send all of that stuff to a third party, which then provides a fraud score.
Some of the criteria for that fraud score are based around delivery (known dodgy addresses, business rather than residential, delivery address doesn't match billing etc).
You can just submit billing addresses - you may have to tune your approach towards fraud scores accordingly.
I have a requirement to be able to accept different forms of payment within the same order - ie not just the usual credit card or paypal for the whole thing, but perhaps paypal for one item, cheque for another. I know this sounds quite crazy, but there is a good business reason for the requirement so I can't just push back.
The best way I can think of implementing it at the moment is to have kind of a hub page, where you can "launch off" into multiple flows for each of the payments by opening new windows. I can't figure out a way of doing this in a linear flow as for example you can't guarantee that a user will come back from paypal, so you'd then lose the user completely.
Is there a neater way of doing this that anyone can think of, or can anyone point me to an example of a site that does somethign similar for inspiration?
Even when opening several windows at once, there is no guarantee that the user will complete all payment methods. So you are most probably going to lose a few users or payments. Be sure to send automated e-mail follow-ups for missing payments to minimize this problem. The e-mails could contain links to your payment providers for easy accesss to their outstanding payment operations.
This is a difficult problem, but how many payment processors do you have to go offsite for? Should only be paypal.
In any case, I'd give the user all their payment options on one page, and let them fill in the amount for each processor or payment type. Then the next page would list those they chose, the amount for each, and a link to "Complete this payment".
The link would open in a new window.
You'll have to have a good back end and javascript, as well as user warnings so that the payment page gets updated as each payment is processed. Consider using popup dialogs to show that a payment has completed, or that the order has sat idle for more than 10-30 minutes without complete payment.
Also, consider sending emails and letting the user complete the payments through links in the emails. Send a new email each time a payment is completed, and a final email if all payments are complete and the order is moving forward.
Send an email one hour, and one day later for uncompleted orders with remaining payments required, that also give them the option of choosing different payment options for the remainder.
Email isn't best (lose more orders that way due to changing minds) but it's good for the type of transactions you're thinking about.
Personally, I'd do it like this:
Let the user fill their basket in the ususal way
Allow them to add payment types and amounts to a list (2nd basket almost)
When the payments balance against the basket, start processing the payments
For external sites, try a frame which has a progress indicator at the top.
In an ideal world it wouldn't be linear. But a lot of users might lose a spawned window, or get confused by the parallelism.
Better to stick to established IxD principals and rely on good feedback instead. Give the user control from the outset and keep it transparent.
Lastly, start the payment process with the most immediate (e.g. paypal) to reduce users giving up. (COD should come last!)
Hope this helps,
Tom
If possible, just separate your order into separate smaller orders based off of the payment selections of the user.
And don't do it linearly. If anything you could open up each payment processor in a separate window so that you maintain presence.
I would take an approach where the whole order is broken down into sub-orders for each of the necessary payment methods. You can load the PayPal portion, the check portion, etc. and process them separately. It's important for the user to know how much is being charged to each of their payment methods, so it makes sense in this case to present the whole order as broken down by payment method (versus displaying as a unified order).
Implementation would be easiest if it's always a certain subset of items that is forced to any payment method. If this differs by user, or if it's when the order reaches a certain amount, the situation could become much more complicated. Can you be more specific about your approach?
Processing Multiple Order Payments
Give the user the option to make a payment for a pending order using any of your payment types.
Let the user specify an (Amount <= [Order Total] - [Payments Received]), if that is part of your requirement.
If the order is still pending after you process the payment (see how below), take them back to step 1 to rinse and repeat.
How to store and process each payment made:
Use a Payments table to store all order payments, the PaymentMethod used and its Amount with its CurrencyCode.
When a payment is received for an order, store the payment and sum all received amounts converted into your base currency as [Payments Received].
If [Payments Received] >= [Order Total], mark the order as Paid. Or, if dealing with double-converted foreign exchange rates, check if it is correct to within a small-enough margin, eg 0.5%.
Optionally, convert any overpayment into prepaid credit for the client.