I have set up a HttpTriggered Azure Function that handles events from EventGrid. We have downstream APIs that we need to call, sometimes these are down or something unexpected happens. I want to store the events that fail. I have setup dead lettering. However, reading the docs, it looks like EventGrid will immediately dead letter events with these responses:
400 Bad Request, 413 Request Entity Too Large, 403 Forbidden, 404 Not Found, 401 Unauthorized
Is there a way to configure event grid to skip dead lettering of these, and only store all other codes? I don't want to clutter my storage account with these types of events, as I suspect I will see lots of 400 and 403 from some of our event sources.
I guess I can make another function that is blob that is blob triggered and deletes the events with the status codes I don't want, if that is possible. Just wondering if this is something that is possible in the setup of the dead lettering configuration in the subscription.
The best option here might be to add some structured exception handling within your function to manage those downstream APIs. If you don't want to dead-letter the event delivery, then catch the exception and return a different status code so that another delivery attempt will occur.
A second option could be to choose a different handler, such as a service bus queue. This will give you more control over the dead-letter process as well as accommodate for those long-running processes that may exceed the 30 second response/acknowledgement window that Event Grid expects.
Related
Just curious to know how others handle this from a security perspective. In a workaround related to Microsoft Power Automate, I'm using events and bolt-python to listen for a key word that will trigger an action to my bot endpoint. The issue is, as I'm building and testing this, every message posted is triggering an unhandled request and invoking my bot.of course the logic I want will execute fine when the key word is triggered but the amount of unnecessary triggers with the messages is concerning.
If I make a request to the gmail API for Users.messages.list it will return 100 of the newest messages.
If I make another request, 1 minute later and there are no new emails it will return the same 100 most recent messages.
If I make a third request, 2 minutes after the original and there is a reply to one of the 100 most recent messages it will still return the 100 most recent messages.
The problem with this is the it only returns the message id and thread id, not if there is a new reply or not. That would mean that I would have to check every message that I have locally stored, or every one of the 100 returned messages just to know if there was a reply to it or not.
The way that works, you couldn't "check" your email via the api because if you had stored 10000 messages and you were checking replies on all of them you would use up your entire API "number of requests" allocation in a single day!
What's wrong with you Google?
Sure I could use pop3 or imap but why when I could just use something like /list_recent?
You should definitely take a look at this part of the Gmail API documentation: https://developers.google.com/gmail/api/guides/push
Basically, it explains you how you can setup a "watch" over a Gmail account and receive notifications (new e-mail, deleted e-mail, labels added...) through Pub/Sub, the messaging queue protocol from Google.
The notifications will contain an historyId, which is a kind of milestone in the Gmail account. Using the /history endpoint, you'll then get the e-mails that were added and/or deleted since this history point. You have to store the latest historyId you handled somewhere in your app, so that you can query the right changes (and not miss anything) on the next notification.
If you don't need to react to changes in real-time, maybe you can call the /history endpoint periodically, but it will definitely be less efficient at scale.
It takes a bit of work to get this working but at the end of the day you get a very efficient system able to react about changes in real-time.
Let suppose the following simple UC based on a CQRS architecture:
We have a backend managing a Business Object, let says a Movie.
This backend is composed of 2 Microservices: a CommandManager (Create/Update/Delete Movie) and a QueryManager (Query Movie)
We have a frontend that offer a web page for creating a new Movie and this action lead automatically to another web page describing the Movie.
A simple way to do that is:
A web page collect movie information using a form and send them to the frontend.
The frontend make a POST request to the CommandManager
The CommandManager write the new movies to the datastore and return the movie key
The frontend make a GET using this key to the QueryManager
The QueryManager looks for the Movie in the Datastore using the key and return it.
The frontend deliver the page with the Movie Information.
Ok, now I want to transform this UC in a more Event Driven way. Here is the new flow:
A web page collect movie information using a form and send them to the frontend.
The frontend write a Message in the BUS with the new movie information
The CommandManager listen the BUS and create the new movies in the datastore. Eventually, it publish a new message in the BUS specifying that a new Movie has been created.
At this point, the frontend is no more waiting for a response due to the fact that this kind of flow is asynchronous. How could we complete this flow in order to forward the user to the Movie Information Web page? We should wait that the creation process is done before querying the QueryManager.
In a more general term, in a asynchronous architecture based on bus/event, how to execute Query used to provide information in a web page?
In addition to #VoiceOfUnreason's answer,
If the two microservices are RESTFul, the CommandManager could return a 202 Accepted with a link pointing to the resource that will be created in the future. The client could then poll that resource until the server responds with a 200 OK.
Another solution would be that the CommandManager would return a 202 Accepted with a link pointing to a command/status endpoint. The client would poll that endpoint until the status is command-processed (including the URL to the the actual resource) or command-failed (including a descriptive message for the failure).
These solutions could be augmented by sending the status of all processed commands using Server Sent Events. In this way, the client gets notified without polling.
If the client is not aware that the architecture is asynchronous, a solution is to use an API gateway that blocks the client's request until the upstream microservice processes the command and then to respond with the complete resource's data.
At this point, the frontend is no more waiting for a response due to the fact that this kind of flow is asynchronous. How could we complete this flow in order to forward the user to the Movie Information Web page? We should wait that the creation process is done before querying the QueryManager.
Short answer: make the protocol explicit.
Longer answer: a good place to look for inspiration here is HTTP.
The front end makes a POST to the origin server; as a result the origin server places a message on the queue and sends a response back.
The representation sent with this response ought to describe the request's current status and point to (or embed) a status monitor that can provide the user with an estimate of when the request will be fulfilled.
The client can then poll the endpoint to find out what progress has been made.
For instance, the endpoint might be a query into the data store, that looks for evidence that the command manager has processed the original command; or it might be an endpoint that is watching the bus for the MovieCreated message, and changes its answer based on whether or not it has seen that.
It may help clarify things to look into idempotent request handling; when the Command Manager pulls a message off of its queue, how does it know if it has previously processed a copy of that message? Your polling endpoint should be able to use the same information to let the consumer know that the message has been successfully processed.
In addition to #Constantin Galbenu's answer, I would like to put in my two cents.
I would strongly advise you to look at a microservices pattern called "BFF" (Backend-For-Frontend) pattern. Instead of having a thick API gateway doing all the work, you can have an API per use-case. For Example: In your case, you can an API called "CreateMovieBFFHandler" which would receive the POST request from front-end and then this guy would coordinate with other things in the system like message queues, events etc. to track the status of the submitted request. UI might have a protocol with this BFFhandler that if the response doesn't come back in X seconds, then the front-end would consider it as failure and if this handler is able to get a successfully processed messaged from message queue or "MovieCreated" event for this key, then it could send a 200 OK back and then you can redirect the page to call write side and then populate the UI.
Useful Link: https://samnewman.io/patterns/architectural/bff/
I'm working on a real-time chat. I need to change statuses of the room owner and connected users, together with the UI. Since NodeJS/SocketJS/etc don't guarantee message delivery, I switched to pure Ajax for that.
The system works like that:
- User presses a button to change his status
- An Ajax request is being sent to the server, and a status change request is being saved in the queue in DB
- Users send Ajax heartbeats every second. On the server this heartbeat function also processes the queue (when sent by the room owner). Besides it sends the current statuses of users in the room in response every time.
The issue is: there might be temporary internet problems on both sides, which causes all kinds of problems. This happens due to the fact, that the heartbeat Ajax requests are being processed in an arbitrary order on the server, or the responses are being received in a wrong order on the client side. As a result users have wrong data about current statuses and the UI changes are also wrong.
What is the best approach when making a system like that? What am I doing wrong or how can I fix the issues above?
Thank you!
Have a look at Max's blog a Django-Realtime-Chat and how he does it.
When working with pull subscriptions to Office365 calendar folders, I've been getting a lot of ErrorReadEventsFailed messages in the SendNotification request. This error essentially means that the subscription can no longer be found, and the server should no longer expect new notifications.
Checking Microsoft's recommended error handling, the solution is to use Autodiscover to rediscover the ExternalEwsUrl or EwsPartnerUrl, and create a new subscription.
With Office365, the AutoDiscovery service seems near impossible with a combination of OAuth2 service accounts so I've been using https://outlook.office365.com/EWS/Exchange.asmx as the main EWS endpoint.
However, when I try to create a new subscription for the specific calendar folder, I keep getting a generic 500 ErrorNoRespondingCASInDestinationSite error:
Exchange Web Services are not currently available for this request because none of the Client Access Servers in the destination site could process the request.
The strange part is this only happens directly after receiving the initial ErrorReadEventsFailed error. If I try again in, say, 30 seconds, the request goes through without a problem.
After doing some research, it seemed that most users found it helpful to ensure that the X-AnchorMailbox header was set properly for the user that the service account wishes to impersonate. I double-checked this header, and it is indeed being sent along the request to resubscribe.
This problem may be solvable by an exponential back-off solution, or by just retrying X amount of times until the request goes through. It seems to me that when the subscription gets "lost", the O365 service needs time to change the DNS of the Exchange server (it's the only thing I can think of).
Any help would be greatly appreciated!
Given the documentation at: https://msdn.microsoft.com/en-us/library/office/dn458788(v=exchg.150).aspx
When a subscription is lost, or is no longer accessible, it is best to create a new subscription and not include the old watermark in the new subscription. Resubscribing with the old watermark causes a linear scan for events, which is costly.
Instead, create a new subscription and compare folder properties to look for content changes that occurred between the lost subscription and the new subscription. The extended folder properties that we recommend that you check are PR_LOCAL_COMMIT_TIME_MAX (0x670a0040) and PR_DELETED_COUNT_TOTAL (0x670b0003).
You can do this by creating an extended property definition.
I think this may help you!!