I have a service that receives a request, generates an email, saves the email to message queue (to be sent by other microservice) and returns httpStatus.Ok.
I want to test that for different requests a relevant email will be generated.
According to Contract Tests vs Functional Tests
my tests are functional, not contract tests.
(If my service would return email content as an api response, using Pact for contract tests will certainly be appropriate).
I have an idea to use Pact infrastructure for such functional tests, in particular
1.Save request and expected generated email into Pact Broker
2. In provider Verify tests submit the request and verify generated email with expected email.
Does it make sense to use Pact in such functional tests?
Does anyone know examples of similar usage?
Any alternative technologies ( preferably in .Net Core) to have similar testing?
I am also considering
https://github.com/approvals/ApprovalTests.Net, but Pact infrastructure attracts me more.
Related note: The Pact normally works with http requests/responses, but Pact V3 (not implemented by PackNet yet) Introduces messages for services that communicate via event streams and message queues. An example describes messages pact contract tests is
https://dius.com.au/2017/09/22/contract-testing-serverless-and-asynchronous-applications/ referenced by
Pact for MessageQueue's : Sample Provider test in case of MessageQueues
I have a service that receives a request, generates an email, saves the email to message queue (to be sent by other microservice) and returns httpStatus.Ok.
So as you say, with Pact whilst the intent is not to be a functional testing tool in the traditional sense, it is almost unavoidable to not test functionality. The objective is really about testing the contract between the systems (this creates a small grey area in which you'll need to decide what is most appropriate for your test strategy).
What you don't want to do with Pact, is run the verification tests and then check that the email was actually sent, it was written to the queue, and that the downstream microservice could process it - that would be going beyond the bounds of a "contract test".
As an aside: what you can definitely do, is create a separate contract test between this component that publishes to the queue and the downstream component that receives from it (see this WIP branch for .NET library: https://github.com/pact-foundation/pact-net/pull/175)
I want to test that for different requests a relevant email will be generated.
If for those "different tests", the responses from the API are predictably shaped - then yes, you could definitely do that with Pact.
So rewording the above to "I have a service that receives a request, returns httpStatus.Ok and the email body sent" is an acceptable contract-testing IMO.
You could then expand upon this with various scenarios:
When the user was created successfully ...
When the user already exists...
etc.
Hope that helps!
P.S. Might be worth jumping in to https://slack.pact.io and chatting with the community further there.
Related
I am implementing a REST API that internally places a message on a message queue and receives a message as a response on a different topic.
How could API implementation handle publishing and consuming different messages and responds to the client?
What if it never receives a message?
How does the service handle this time-out scenario?
Example
I am implementing a REST API to process an order. The implementation internally publishes a series of messages to verify the payment, update inventory, and prepare shipping info. Finally, it sends the response back to the client.
Queues are too low-level abstraction to implement your requirements directly. Look at an orchestration solution like temporal.io that makes programming such async systems trivial.
Disclaimer: I'm one of the founders of the Temporal open source project.
How could API implementation handle publishing and consuming different messages and responds to the client?
Even though messaging systems can be used in RPC like fashion:
there is a request topic/queue and a reply topic/queue
with a request identifier in the messages' header/metadata
this type of communication kills the promise of the messaging system: decouple components in time and space.
Back to your example. If ServiceA receives the request then it publishes a message to topicA and returns with an 202 Accepted status code to indicate that the request is received but not yet processed completely. In the response you can indicate an url on which the consumer of ServiceA's API can retrieve the latest status of its previously issued request.
What if it never receives a message?
In that case the request related data remains in the same state as it was at the time of the message publishing.
How does the service handle this time-out scenario?
You can create scheduled jobs to clean-up never finished/got stuck requests. Based on your business requirements you can simple delete them or transfer them to manual processing by the customer service.
Order placement use case
Rather than creating a customer-facing service which waits for all the processing to be done you can define several statuses/stages of the process:
Order requested
Payment verified
Items locked in inventory
...
Order placed
You can inform your customers about these status/stage changes via websocket, push notification, e-mail, etc.. The orchestration of this order placement flow can be achieved for example via the Saga pattern.
When building an application before scaling to multiple micro services. You have a codebase consisting of services that are decoupled. IE a services no longer depends on another service, not even loosely via a interface. It receives input from a service via a message buss. It has a method receivePaymentRequest but its callee is not the Order service. Its invoked via the message bus, perhaps in the future on another server. But imagine theres no need to run multiple servers at this point.
a order services posts to the message bus payment-request event
the payment services picks up on this message
payment is completed
payment service send a payment-complete event message to the message bus
the order service picks up this message
I"m not thinking about the patterns that enable this to be fault tolerant. But instead when to use this approach since it adds a lot of complexity. So please ignore what i've left out with regards to this
Correct? Is it stupid to implement it like such before scaling to microservices. How does this. Is SOA the step before actual microservices?
When should a class receive/publish on the message buss and when should it depend on a service as a class (even injected via a interface) ?
Recently I am trying to use MassTransit in our microservice ecosystem.
According to MassTransit vocabulary and from documents my understanding is :
Publish: Sends a message to 1 or many subscribers (Pub/Sub Pattern) to propagate the message.
Send: Used to send messages in fire and forget fashion like publish, but instead It is just used for one receiver. The main difference with Publish is that in Send if your destination didn't receive a message, it would return an exception.
Requests: uses request/reply pattern to just send a message and get a response in a different channel to be able to get response value from the receiver.
Now, my question is according to the Microservice concept, to follow the event-driven design, we use Publish to propagate messages(Events) to the entire ecosystem. but what is exactly the usage (use case) of Send here? Just to get an exception if the receiver doesn't exist?
My next question is that is it a good approach to use Publish, Send and Requests in a Microservices ecosystem at the same time? like publish for propagation events, Send for command (fire and forget), and Requests for getting responses from the destination.
----- Update
I also found here which Chris Patterson clear lots of things. It also helps me a lot.
Your question is not related to MassTransit. MassTransit implements well-known messaging patterns thoughtfully described on popular resources such as Enterprise Integration Patterns
As Eben wrote in his answer, the decision of what pattern to use is driven by intent. There are also technical differences in the message delivery mechanics for each pattern.
Send is for commands, you tell some other service to do something. You do not wait for a reply (fire and forget), although you might get a confirmation of the action success or failure by other means (an event, for example).
It is an implementation of the point-to-point channel, where you also can implement competing consumers to scale the processing, but those will be instances of the same service.
With MassTransit using RabbitMQ it's done by publishing messages to the endpoint exchange rather than to the message type exchange, so no other endpoints will get the message even though they can consume it.
Publish is for events. It's a broadcast type of delivery or fan-out. You might be publishing events to which no one is listening, so you don't really know who will be consuming them. You also don't expect any response.
It is an implementation of the publish-subscribe channel.
MassTransit with RabbitMQ creates exchanges for each message type published and publishes messages to those exchanges. Consumers create bindings between their endpoint exchanges and message exchanges, so each consumer service (different apps) will get those in their independent queues.
Request-response can be used for both commands that need to be confirmed, or for queries.
It is an implementation of the request-reply message pattern.
MassTransit has nice diagrams in the docs explaining the mechanics for RabbitMQ.
Those messaging patterns are frequently used in a complex distributed system in different combinations and variations.
The difference between Send and Publish has to do with intent.
As you stated, Send is for commands and Publish is for events. I worked on a large enterprise system once running on webMethods as the integration engine/service bus and only events were used. I can tell you that it was less than ideal. If the distinction had been there between commands and events it would've made a lot more sense to more people. Anyway, technically one needs a message enqueued and on that level it doesn't matter, which is why a queueing mechanism typically would not care about such semantics.
To illustrate this with a silly example: Facebook places and Event on my timeline that one of my friends is having a birthday on a particular day. I can respond directly (send a message) or I could publish a message on my timeline and hope my friend sees it. Another silly example: You send an e-mail to PersonA and CC 4 others asking "Please produce report ABC". PersonA would be expected to produce the report or arrange for it to be done. If that same e-mail went to all five people as the recipient (no CC) then who gets to do it? I know, even for Publish one could have a 1-1 recipient/topic but what if another endpoint subscribed? What would that mean?
So the sender is responsible, still configurable as subscriptions are, to determine where to Send the message to. For my own service bus I use an implementation of an IMessageRouteProvider interface. A practical example in a system I once developed was where e-mails received had to have their body converted to an image for a content store (IBM FileNet P8 if memory serves). For reasons I will not go into the systems were stopped each night at 20h00 and restarted at 6h00 in the morning. This led to a backlog of usually around 8000 e-mails that had to be converted. The conversion endpoint would process a conversion in about 2 seconds but that still takes a while to work through. In the meantime the web front-end folks could request PDF files for conversion to paged TIFF files. Now, these ended up at the end of the queue and they would have to wait hours for that to come back. The solution was to implement another conversion endpoint, with its own queue, and have the web front-end configured to send the same message type, e.g. ConvertDocumentCommand to that "priority" queue for processing. Pretty easy to do. Now, if that had been a publish how would I do that split? The same event going to 2 different endpoints under different circumstances? Well, you could have another subscription store for your system but now you'd need to maintain both. There could be another answer such as coding this logic into the send bit but that is a design choice and would require coding changes.
In my own Shuttle.Esb service bus I only have Send and Publish. For request/response both the sender and receiver have an inbox and a request would be sent (Send) to the receiver and it in turn could reply (also a Send but uses the sender's URI).
I'm new to microservices architecture and want to create a centralised notification microservice to send emails/sms to users.
My first option was to create a notification Kafka queue where all other microservices can send notifications to. The notification microservice would then listen to this queue and send messages accordingly. If the notification service was restarted or taken down, we would not lose any messages as the messages will be stored on the queue.
My second option was to add a notification message API on the notifications microservice. This would make it easier for all other microservices as they just have to call an API as opposed to integrate with the queue. The API would then internally send the message to the notification Kafka queue and send the message. The only issue here is if the API is not available or there is an error, we will lose messages.
Any recommendations on the best way to handle this?
Either works. Some concepts that might help you decide:
A service that fronts "Kafka" would be helpful to:
Hide the implementation. This gives you the flexibility to change Kafka out later for something else. Your wrapper API would only respond with a 200 once it has put the notification request on the queue. I also see giving services direct access to "your" queue similar to allowing services to directly interact with a database they don't own. If you allow direct-access to Kafka and Kafka proves to be inadequate, a change to Kafka will require all of your clients to change their code.
Enforce the notification request contract (ensure the body of the request is well-formed). If you want to make sure that all of the items put on the queue are well-formed according to contract, an API can help enforce that. That will help prevent issues later when the "notifier" service picks notifications off the queue to send.
Adding a wrapper API would be less desirable if:
You don't want to/can't spend the time. Maybe deadlines are driving you to hurry and the days it would take to stand up a wrapper is just too much.
You are a small team and you don't have the resources/tools/time for service-explosion.
Your first design is simple and will work. If you're looking for the advantages I outlined, then consider your second design. And, to make sure I understand it, I would see it unfold like:
Client 1 needs to put out a notification and calls Service A POST /notifications
Service A that accepts POST /notifications
Service A checks the request, puts it on Kafka, responds to client with 200
Service B picks up notification request from Kafka queue.
Service A should be run as multiple instances for reliability.
The Problem
We are currently architecting our new Notification Microservice but having trouble with how to handle aggregated emails. What we need to do is instead of sending one email every action performed (could be 20+ in a few minutes), we would send an email after an hour summarising all the actions that were completed.
What We Have So Far
We so far propose that we have this type of messaging pattern, where Client Service is any service in our cluster and Messagebot is our Notification Microservice.
Client Service sends a notification to Messagebot that it will need to send something in the future
Messagebot stores the details in its database
Messagebot periodically checks its database for what needs to be sent
Messagebot gets the required data from another service (could be Client Service) via API
Messagebot sends email using the data from #3 and an HTML template
The Debate
For the data that needs to be sent, we are less sure and it is what we need help with. So far we think this should be the structure of the JSON from Client Service to Notification Service (step #1):
{
template_id: SOME_TEMPLATE_ID,
user_id: SOME_USER_ID,
objectid: SOME_OBJECT_ID
}
or
{
template_id: SOME_TEMPLATE_ID,
user_id: SOME_USER_ID,
required_objects: { task_id: SOME_TASK_ID, document_id: SOME_DOCUMENT_ID }
}
Where task_id and document_id are just examples and it would change based on the template. It could just as easily be {product_id: SOME_PRODUCT_ID} for a different template.
Why The Debate
Our thoughts so far are that:
We only need template_id because the source of the data would be implied in the objects (like an ENV var). For example, the Task object would be at http://taskservice/:id. Otherwise, we can have problems with failing APIs or switching URLs in the future.
We should use userid instead of email and name because we prevent the issue of email/ name pairs not matching up over multiple messages
For the objects, we're still sceptical because it means that the client app service would need knowledge of the inner workings in Messagebot but a single objectid might not be very extensible. We could easily imagine many of our messages needing more than one object.
In Conclusion
Thank you for reading. The design of this service is important because it will be central to our entire organisation.
Which debated JSON structure is most appropriate in our situation? Also, knowing our requirements, what would be the proper setup for this type of service? (aka. Are we correct in our other assumptions?)
So your messagebot will
store notifications
get data from other services
compile emails from the data and
send the compiled emails
In my opinion, your messagebot were given too many tasks. If I were designing the system, I would like to keep the messagebot simpler. The servces should encapsulate the knowledge to compile the email, e.g. manage it's own template and so on. The services will push the compiled emails to a queue so the messagebot can pick up and send. The only logic in the messagebot is to pick up the emails from the queue and send. In this way, it doesn't matter how many more services you are going to have in the future, the messagebot will stay nice and simple.