We have conducted a pen test from 3rd party vendor. One of the observations is that there are no data boundaries. We have millions of fields in our applications. There are no validations on String type fields apart from occasional business logic related constraints on few fields.
The pen test has revealed issues related to a lot of String fields where one can insert numeric values or negative values and the app is processing the same without any issues.
My question is, is this a valid test? Why would someone disallow a numeric value such as “100” in a String field unless specifically asked for by business?
If this is a valid issue, this means every single String attribute has to be tested for non-numeric values which is quite insane. What should be the right approach?
This answer is a bit late, but hopefully it's still helpful to you or someone else down the line.
This finding has to do with Input Validation, and is absolutely a valid test. This is one of the controls that is recommended for implementation by the OWASP Foundation.
The risk here typically comes when this unsanitized value is passed to a downstream component, such as a database, user interface, or vulnerable function.
Here's a hasty example:
async function chargeUser(userId: number, currency: string) {
const amount = (await User.getUser(userId)).amountDue;
return await api.chargeUser(user.id, amount + currency);
}
The above Typescript function takes a user ID and currency, then makes a call to an API to charge the user for their current subscriptions. Let's make a few assumptions:
The API expects the amount in the format of "100USD" or "100".
The API allows negative charges for refunds.
If the currency is not included, the API will assume USD.
If the currency is unknown, the API call will fail with an error.
The user currently has $100 due.
Under normal conditions, an employee would make a request from the UI, specifying the currency to charge the user in. They have no control over how much the user is charged:
{"userId": 1, "currency": "USD"}
This would result the user being charged 100 USD, as expected.
However, an attacker could leverage Javascript's loose typing and the lack of input validation to charge the user above the amount due:
{"userId": 1, "currency": 10000}
Since 100 + 10,000 is 10,100, and the API assumes the currency if it is omitted, this would result in the user being charged 10,100 USD.
Similarly, the attacker could instead do something such as:
{"userId": 1, "currency": -100100}
Since 100 - 100,1000 is -100,000 and the API supports negative amounts, this would result in the user having 100,000 USD deposited into their bank account.
Since penetration tests are time boxed and penetration testers generally don't have access to your backend code or possess the same level of in depth knowledge of your application as your team, it's not usually possible to determine each potential instance where this could occur, hence the broad recommendation to ensure proper input validation is implemented for all fields.
Any input from the user, for all fields, for all requests, should always be validated to the fullest extent possible.
Since you have a large number of fields that share similar validation logic, you may wish to implement a single shared function responsible for validation, possibly according to a schema. For possible solutions, see:
Open API
GraphQL
Spring Framework Documentation
Related
I send transactions programmatically and I need to know exactly how much the fee is going to be. I managed to figure out how to calculate fees for ordinary transaction ((transfer cost + receipt creation cost) * 2), but now I'm struggling with a case where I need all my funds out of the account without deleting it. As I understand, in this case there must be a storage rent left on the account. However, I can't really figure out how to calculate that rent. There is a value returned from 'EXPERIMENTAL_protocol_config' method that seem to be connected to rent - 'storage_amount_per_byte', which implies that each byte costs 10000000000000000000 yocto, and also I can get 'storage_usage' from 'query' method with request type 'view_account', which is supposedly indicated how many bytes my account uses (which is 182). But whenever I try to send a transaction, I get a 'NotEnoughBalance' error that states that transaction cost is higher than the balance, but just by 669547687500000000 yocto. Whatever I do, I can't understand where this number comes from. No combination of fees from aforementioned 'EXPERIMENTAL_protocol_config' method yields this number.
There seems to be little to no decent documentation on transaction fee calculation, except for some 'fixed' values for most used actions. If you have any info on fee/storage rent calculation - I'll be thankful for it.
Through a random chance, I managed to find out the name that the number '6695476875' is referred to as, 'Reserved for transactions', (in gas, not tokens) as in the official wallet (wallet.near.org). God knows why it is reserved, neither docs.near.org, nomicon.io nor wiki.near.org have any info regarding this 'reservation' and this number is never mentioned in any RPC API method. This number is also never mentioned in 'near-api-js' lib, so I really have no idea if devs are even aware of it.
Anyway, since the title of this problem is 'How to calculate storage rent', the answer is something like this:
You get account info from 'query' method of RPC API (here's the doc) and take the "storage_usage" value (this is the amount of bytes that your account takes up on the blockchain).
You get protocol info from 'EXPERIMENTAL_protocol_config' method of RPC API (here's the doc) and take the "storage_amount_per_byte" value.
You multiply the amount of bytes by the storage_amount_per_byte and add the magic 669547687500000000 number to it.
And the resulting number is the least amount of tokens that you must have at your account at any time.
I don't know why it is a common practice to make lives of developers harder in blockchain industry, but this is a good example of such practice.
I am learning to develop microservices using DDD, CQRS, and ES. It is HTTP RESTful service. The microservices is about online shop. There are several domains like products, orders, suppliers, customers, and so on. The domains built in separate services. How to do the validation if the command payload relates to other domains?
For example, here is the addOrderItemCommand payload in the order service (command-side).
{
"customerId": "CUST111",
"productId": "SKU222",
"orderId":"SO333"
}
How to validate the command above? How to know that the customer is really exists in database (query-side customer service) and still active? How to know that the product is exists in database and the status of the product is published? How to know whether the customer eligible to get the promo price from the related product?
Is it ok to call API directly (like point-to-point / ajax / request promise) to validate this payload in order command-side service? But I think, the performance will get worse if the API called directly just for validation. Because, we have developed an event processor outside the command-service that listen from the event and apply the event to the materalized view.
Thank you.
As there are more than one bounded contexts that need to be queried for the validation to pass you need to consider eventual consistency. That being said, there is always a chance that the process as a whole can be in an invalid state for a "small" amount of time. For example, the user could be deactivated after the command is accepted and before the order is shipped. An online shop is a complex system and exceptions could appear in any of its subsystems. However, being implemented as an event-driven system helps; every time the ordering process enters an invalid state you can take compensatory actions/commands. For example, if the user is deactivated in the meantime you can cancel all its standing orders, release the reserved products, announce the potential customers that have those products in the wishlist that they are not available and so on.
There are many kinds of validation in DDD but I follow the general rule that the validation should be done as early as possible but without compromising data consistency. So, in order to be early you could query the readmodel to reject the commands that couldn't possible be valid and in order for the system to be consistent you need to make another check just before the order is shipped.
Now let's talk about your specific questions:
How to know that the customer is really exists in database (query-side customer service) and still active?
You can query the readmodel to verify that the user exists and it is still active. You should do this as a command that comes from an invalid user is a strong indication of some kind of attack and you don't want those kind of commands passing through your system. However, even if a command passes this check, it does not necessarily mean that the order will be shipped as other exceptions could be raised in between.
How to know that the product is exists in database and the status of the product is published?
Again, you can query the readmodel in order to notify the user that the product is not available at the moment. Or, depending on your business, you could allow the command to pass if you know that those products will be available in less than 24 hours based on some previous statistics (for example you know that TV sets arrive daily in your stock). Or you could let the customer choose whether it waits or not. In this case, if the products are not in stock at the final phase of the ordering (the shipping) you notify the customer that the products are not in stock anymore.
How to know whether the customer eligible to get the promo price from the related product?
You will probably have to query another bounded context like Promotions BC to check this. This depends on how promotions are validated/used.
Is it ok to call API directly (like point-to-point / ajax / request promise) to validate this payload in order command-side service? But I think, the performance will get worse if the API called directly just for validation.
This depends on how resilient you want your system to be and how fast you want to reject invalid commands.
Synchronous call are simpler to implement but they lead to a less resilient system (you should be aware of cascade failures and use technics like circuit breaker to stop them).
Asynchronous (i.e. using events) calls are harder to implement but make you system more resilient. In order to have async calls, the ordering system can subscribe to other systems for events and maintain a private state that can be queried for validation purposes as the commands arrive. In this way, the ordering system continues to work even of the link to inventory or customer management systems are down.
In any case, it really depends on your business and none of us can tell you exaclty what to do.
As always everything depends on the specifics of the domain but as a general principle cross domain validation should be done via the read model.
In this case, I would maintain a read model within each microservice for use in validation. Of course, that brings with it the question of eventual consistency.
How you handle that should come from your understanding of the domain. Factors such as the length of the eventual consistency compared to the frequency of updates should be considered. The cost of getting it wrong for the business compared to the cost of development to minimise the problem. In many cases, just recording the fact there has been a problem is more than adequate for the business.
I have a blog post dedicated to validation which you can find here: How To Validate Commands in a CQRS Application
Scenario:
I have 2 Microservices (which both use CQRS + Event Sourcing internally)
Microservice 1 manages Contacts (= Aggregate Root)
Microservice 2 manages Invoices (= Aggregate Root)
The recipient of an invoice must be a valid contact.
CreateInvoiceCommand:
{
"content": "my invoice content",
"recipient": "42"
}
I now read lot's of times, that the write side (= the command handler) shouldn't call the read side.
Taking this into account, the Invoices Microservice must listen to all ContactCreated and ContactDeleted events in order to know if the given recipient id is valid.
Then I'd have thousands of Contacts within the Invoices Microservice, even if I know that only a few of them will ever receive an Invoice.
Is there any best practice to handle those scenarios?
The recipient of an invoice must be a valid contact.
So the first thing you need to be aware of - if two entities are part of different aggregates, you can't really implement "apply a change to this entity only if that entity satisfies a specification", because that entity could change between the moment you evaluate the specification and the moment you perform the write.
In other words - you can only get eventual consistency across an aggregate boundary.
The aggregate is the authority for its own state, but everything else (for example, the contents of the command message), it pretty much has to accept that some external authority has checked the data.
There are a couple approaches you can take here
1) You can blindly accept that the recipient specified in the command is valid.
2) You can try to verify the validity of the recipient from some external authority (aka: a read model of some other aggregate) between receiving it from the untrusted source and submitting it to the domain model.
3) You can blindly accept the command as described, but treat the invoice as provisional until the validity of the recipient is confirmed. That means there is a second command to run on the invoice that certifies the recipient.
Note - from the point of view of the model, these different commands are equivalent, but at the application layer they don't need to be -- you can restrict access to the command to trusted sources (don't make it part of the public api, require authorization that is only available to trusted sources, etc).
Approach #3 is the most microservicy, as the two commands can be separated in time -- you can accept the CreateInvoice command as soon as it arrives, and certify the recipient asynchronously.
Where would you put approach 4), where the Invoices Microservice has it's own Contacts Store which gets updated whenever there's a ContactCreated or ContactDeleted event? Then both entities are part of the same service and boundary. Now it should be possible to make things consistent, right?
No. You've made the two entities part of the same service, but the problem was never that they were in different services, but that they are in separate aggregates -- meaning we can be changing the entity states concurrently, which means that we can't ensure that they are immediately synchronized.
If you wanted immediate consistency, you need a model that draws your boundaries differently.
For instance, if the invoice entities were modeled as part of the Contacts aggregate, then the aggregate can ensure the invariant that new invoices require a valid recipient -- the domain model uses the copy of the state in memory to confirm that the recipient was valid when we loaded, and the write into the book of record verifies that the book of record hadn't changed since the load happened.
The write of the aggregate state is a compare-and-swap in the book of record; if some concurrent process had invalidated the recipient, the CAS operation would fail.
The trade off, of course, is that any change to the Contact aggregate would also cause the invoice to fail; concurrent editing of different invoices with the same recipient goes out the window.
Aggregates are all or nothing; they aren't separable.
Now, one out might be that your Invoice aggregate has a part that must be immediately consistent with the recipient, and another part where eventually consistent, or even inconsistent, is acceptable. In which case your goal is to refactor the model.
The recipient of an invoice must be a valid contact.
This is a business rule. The question should be asked, what does this business rule mean for my application? Who should take responsibility for implementing this rule, or can the responsibility be shared?
One possibility is that, yes, the business rule is about invoices so it should be the responsibility of the Invoices Service to implement it.
However, the business rule is really about the creation of invoices. And the owner of invoice creation in your architecture is, strangely, not the Invoices Service. The reason for this is that the name of the command is CreateInvoiceCommand.
Let's think about this - the Invoices Service will never just create an invoice on its own. It just provides the capability. In this architecture, the actual owner of invoice creation is the sender of the command.
Using this line of reasoning, if the business rule is saying that invoice creation cannot happen against an invalid recipient, then it becomes the responsibility of the command sender to ensure this business rule is implemented.
This would be a very different scenario if, rather than receiving a command, the Invoices Service subscribed to events. As an example, an event called WidgetSold. In this scenario, the owner of invoice creation clearly would be the Invoicing service, and so the business rule would be implemented there instead.
If the user clicks the create invoice for contact 42 button, it's the
user's responsibility to take care that contact 42 exists
Yes, that is correct. The user's intention is to create an invoice. The business rules around invoice creation should, therefore, be enforced at this point. How this happens (or whether this happens at all) is a different question.
But what if the user doesn't care? Then it would create an invoice
with an invalid recipient id.
Also correct. As you say, there are side-effects to this approach, one of which is that you can end up with inconsistent data across your system. That is one of the realities of SOA.
Isn't this somehow similar to this: The Invoice has a currencyCode
property, it's a String.
I don't know if I agree or not. Is asking is this a valid ISO currency? different to asking is entity 42 valid according to another system?. I would think so.
Isn't it kinda the same as given recipient is not null and is valid
according to my Contacts Database?
I agree that in reality, you could implement this validation in the service. I am just saying that I don't think it's the right place for it. If you wanted to do this, you would have to either call out the another service or store all contacts locally, as you framed your question originally. I think it's simpler to just do it outside of the service.
I think that the answer depends on how resilient you want the system to be, that is, how to handle the situation in wich the Contacts Microservice is down (not responding or very slow).
1. You want to be very resilient
If the Contacts Microservice is down, you want to be able to emit invoices for some (maybe most) of the contacts. In this case you listen to the ContactCreated and ContactDeleted and maintain a (eventually consistent) local list of valid contacts; they should be named accordingly to the Ubiquitous language in this bounded context, like Payers (or something like that). Then, in the Application layer, when building the CreateInvoiceCommand you check that Payer is valid and create the command.
2. You don't need to be resilient
If the Contacts Microservice is down, you refuse to generate invoices. In this case, when building the command you make a request to the Invoices Microservice API endpoint and verify that the Payer is valid.
In any case, you check for contact's validity before the command is dispatched.
I am currently in the process of planning & developing an integrated application using Stripe in addition to some other technologies (a combination of backbone & laravel). As a result I am using Stripejs in combination with the PHP stripe library.
I use stripejs to send Stripe their account info (for savings accounts I send country, routing and account numbers via stripes js lib, I then save off the created token in the response to the backend). I am attempting to transfer money to various recipients following charging accounts (so I have money in my stripe account) but I am a bit confused over exactly what approach to take.
My main questions are as follows.
If I am positive of the identities of the individual’s using the application, is verification required (5 – 10 people max will be using)?
If it is required what is the best approach to verifying the user using the current technique I am using (create a token, save off on the backend)?
From my experience with Stripe, you don't need to 'verify identities' each time you use a token, so long as a customer has been set up for recurring use. So as long as your application is properly secured so that only the 5-10 people can use it, and can only process payments for their own accounts, you are good to go. In other words, you obviously want to do your own user verification to make sure only valid people can use the application, but you don't need to do it as far as Stripe is concerned.
That said, if you are in essence transferring money between people, you might want to go the extra distance and make sure that the user has the correct card number or bank account number each time. In which case I would use something like:
Get user data using tokens stored in DB.
$cu = Stripe_Customer::retrieve("cus_4pn93XXXXXXXXXXX");
$card = $cu->cards->retrieve("card_14g7ZU4rLTyXXXXXXXXXXX");
If Stripe's returned data matches user-submitted data, you know you have the correct user. In particular, if the last 4 digits of the card and the expiry date match the last 4 returned by Stripe and the Stripe returned expiry date, you can be reasonably sure (albeit not certain) that the person actually possesses the card in question.
I have a requirement to be able to accept different forms of payment within the same order - ie not just the usual credit card or paypal for the whole thing, but perhaps paypal for one item, cheque for another. I know this sounds quite crazy, but there is a good business reason for the requirement so I can't just push back.
The best way I can think of implementing it at the moment is to have kind of a hub page, where you can "launch off" into multiple flows for each of the payments by opening new windows. I can't figure out a way of doing this in a linear flow as for example you can't guarantee that a user will come back from paypal, so you'd then lose the user completely.
Is there a neater way of doing this that anyone can think of, or can anyone point me to an example of a site that does somethign similar for inspiration?
Even when opening several windows at once, there is no guarantee that the user will complete all payment methods. So you are most probably going to lose a few users or payments. Be sure to send automated e-mail follow-ups for missing payments to minimize this problem. The e-mails could contain links to your payment providers for easy accesss to their outstanding payment operations.
This is a difficult problem, but how many payment processors do you have to go offsite for? Should only be paypal.
In any case, I'd give the user all their payment options on one page, and let them fill in the amount for each processor or payment type. Then the next page would list those they chose, the amount for each, and a link to "Complete this payment".
The link would open in a new window.
You'll have to have a good back end and javascript, as well as user warnings so that the payment page gets updated as each payment is processed. Consider using popup dialogs to show that a payment has completed, or that the order has sat idle for more than 10-30 minutes without complete payment.
Also, consider sending emails and letting the user complete the payments through links in the emails. Send a new email each time a payment is completed, and a final email if all payments are complete and the order is moving forward.
Send an email one hour, and one day later for uncompleted orders with remaining payments required, that also give them the option of choosing different payment options for the remainder.
Email isn't best (lose more orders that way due to changing minds) but it's good for the type of transactions you're thinking about.
Personally, I'd do it like this:
Let the user fill their basket in the ususal way
Allow them to add payment types and amounts to a list (2nd basket almost)
When the payments balance against the basket, start processing the payments
For external sites, try a frame which has a progress indicator at the top.
In an ideal world it wouldn't be linear. But a lot of users might lose a spawned window, or get confused by the parallelism.
Better to stick to established IxD principals and rely on good feedback instead. Give the user control from the outset and keep it transparent.
Lastly, start the payment process with the most immediate (e.g. paypal) to reduce users giving up. (COD should come last!)
Hope this helps,
Tom
If possible, just separate your order into separate smaller orders based off of the payment selections of the user.
And don't do it linearly. If anything you could open up each payment processor in a separate window so that you maintain presence.
I would take an approach where the whole order is broken down into sub-orders for each of the necessary payment methods. You can load the PayPal portion, the check portion, etc. and process them separately. It's important for the user to know how much is being charged to each of their payment methods, so it makes sense in this case to present the whole order as broken down by payment method (versus displaying as a unified order).
Implementation would be easiest if it's always a certain subset of items that is forced to any payment method. If this differs by user, or if it's when the order reaches a certain amount, the situation could become much more complicated. Can you be more specific about your approach?
Processing Multiple Order Payments
Give the user the option to make a payment for a pending order using any of your payment types.
Let the user specify an (Amount <= [Order Total] - [Payments Received]), if that is part of your requirement.
If the order is still pending after you process the payment (see how below), take them back to step 1 to rinse and repeat.
How to store and process each payment made:
Use a Payments table to store all order payments, the PaymentMethod used and its Amount with its CurrencyCode.
When a payment is received for an order, store the payment and sum all received amounts converted into your base currency as [Payments Received].
If [Payments Received] >= [Order Total], mark the order as Paid. Or, if dealing with double-converted foreign exchange rates, check if it is correct to within a small-enough margin, eg 0.5%.
Optionally, convert any overpayment into prepaid credit for the client.