A little background.
I am trying to use reactive programming to be able to download file from the other service. The trick is that in case of connection failure or failed Flux element (anything) I would like to retry on the Flux a number of times but once being able to grasp on it I would like to resume without processing the elements from the very start.
What I mean is that, something goes wrong and I got only 56 elements from my Flux out of 100 possible (let's say it's an image in .jpg) because of the connection failure. Once I successfully retry I would like to resume on 57th element so I do not have to process it and perform GET from the start once again.
Here is how the normal retry looks like:
but what I would like to achieve is that on retry I would only have to get the red colored element (as I already have yellow and purple).
Just a sidenote, I would like to achieve the functionality as with HTTP range request headers where I can get bytes in specific range only and in case of failure I would be able to resume from the byte I want.
Is that even possible what I am trying to achieve? If so, what could be the possible course of action?
You need to keep some state (the beginning of the range to request, at least) on a per-subscriber basis. That has to be done upstream of the retry, so that each retry re-evaluates the range. At the same time, the state should be atomically updatable AND visible downstream of the retry (for updating purposes). I'm assuming you're using WebClient:
a flatMap can be used to create a scope in which the range state is visible
in the lambda, an AtomicLong can be used as the state
again in the flatmap lambda, wrap the webclient call in a Flux.defer to ensure lazy creation of the request with re-evaluation of the state for generating the appropriate header (reading from the AtomicLong)
append retry after the defer
update the AtomicLong as needed once each piece is received and processed (eg. in a doOnNext)
Related
Is there any option to wait for user response message in discordgo? I'm looking for something similar to discord.js awaitMessages.
No, but you can make a collection that holds message and event information and checking news messages.
Simply
Make a collection/array
Add message information
Check if the incoming message in the message event handler is in the collection
Handle event
Remove from collection
Don't forget set a timeout and clear expired data from collection.
according the docs: awaitMessages
time: Amount of time in milliseconds the collector should run for
max: Number of messages to successfully pass the filter
In Go, you can easily use a routine with just one keyword go, so implementing asynchronous (async) is very simple.
solving ideas:
Create a message storage center: It has the following features:
store all sent messages
Have a garbage collection mechanism: Since we are constantly collecting messages, we need a mechanism to eliminate old messages.
Need a mutex: Considering that the messages may generate race conditions, this lock is added to ensure security
It can generate filters: to communicate with each filter, we need to provide a chan for each filter.
Whenever a new message is created, we add the message to the message center, and it also notifies each filter.
Define your filter function: The message will be collected whenever this criterion is true. for example
Define the callback function: This is the result of the filter function. You can do something based on this result. for example
Full code
I put the full code on the replit and gist.
If you want to run it, you can copy the code from replit and set your token and channel ID (or user ID) in the environment variables to test it.
The AWS SQS -> Lambda integration allows you to process incoming messages in a batch, where you configure the maximum number you can receive in a single batch. If you throw an exception during processing, to indicate failure, all the messages are not deleted from the incoming queue and can be picked up by another lambda for processing once the visibility timeout has passed.
Is there any way to keep the batch processing, for performance reasons, but allow some messages from the batch to succeed (and be deleted from the inbound queue) and only leave some of the batch un-deleted?
The problem with manually re-enqueueing the failed messages to the queue is that you can get into an infinite loop where those items perpetually fail and get re-enqueued and fail again. Since they are being resent to the queue their retry count gets reset every time which means they'll never fail out into a dead letter queue. You also lose the benefits of the visibility timeout. This is also bad for monitoring purposes since you'll never be able to know if you're in a bad state unless you go manually check your logs.
A better approach would be to manually delete the successful items and then throw an exception to fail the rest of the batch. The successful items will be removed from the queue, all the items that actually failed will hit their normal visibility timeout periods and retain their receive count values, and you'll be able to actually use and monitor a dead letter queue. This is also overall less work than the other approach.
Considerations
Only override the default behavior if there has been a partial batch failure. If all the items succeeded, let the default behavior take its course
Since you're tracking the failures of each queue item, you'll need to catch and log each exception as they come in so that you can see what's going on later
I recently encountered this problem and the best way to handle this without writing any code from our side is to use the FunctionResponseTypes property of EventSourceMapping. Using this we just have to pass the list of failed message Id and the event source will handle to delete the successful message.
Please checkout Using SQS and Lambda
Cloudformation template to configure Eventsource for lambda
"FunctionEventSourceMapping": {
"Type": "AWS::Lambda::EventSourceMapping",
"Properties": {
"BatchSize": "100",
"Enabled": "True",
"EventSourceArn": {"Fn::GetAtt": ["SQSQueue", "Arn"]},
"FunctionName": "FunctionName",
"MaximumBatchingWindowInSeconds": "100",
"FunctionResponseTypes": ["ReportBatchItemFailures"] # This is important
}
}
After you configure your Event source with above configuration it should look something like below
Then we just have to return the response in the below-mentioned format from our lambda
{"batchItemFailures": [{"itemIdentifier": "85f26da9-fceb-4252-9560-243376081199"}]}
Provide the list of failed message Ids in batchIntemFailures list
If your lambda runtime environment is in python than please return dict in the above mentioned format for java based runtime you can use aws-lambda-java-event
Sample Python code
Advantages of this approach are
You don't have to add any code to manually delete the message from SQS queue
You don't have to include any third party library or boto just for deleting the message from the queue it will help you to reduce your final artifact size.
Keep it simple an stupid
On a side note make sure your lambda have the required permission on sqs to get and delete the message.
Thanks
One option is to manually send back the failed messages to the queue, and then replying with a success to the SQS so that there are no duplicates.
You could do something like setting up a fail count, so that if all messages failed you can simply return a failed status for all messages, otherwise if the fail count is < 10 (10 being the max batch size you can get from SQS -> Lambda event) then you can individually send back the failed messages to the queue, and then reply with a success message.
Additionally, to avoid any possible infinite retry loop, add a property to the event such as a "retry" count before sending it back to the queue, and drop the event when "retry" is greater than X.
The idea was that there are 2 different http requests to 2 different end points. The first one is a long expensive calculation and it returns. The second request goes and does the exact same expensive calculation but before it returns does some extra processing with more data reads and calculations. Instead of doing the exact same calculation twice, it would be nice for the first call to write it's results to a channel or queue and the second http endpoint could join that message with the other data reads and processing before returning.
MessageEndpoints and Service activators can subscribe to a channel but how it would be in the same thread as the second http call on the second endpoint I cannot figure out. To me the mystery is how does the second thread on the second end point block until it receives a message that the first end point creates and sends.
Maybe setting up a polling channel would be the better route to go like on the second end point, it could immediately start polling while doing it's other reads and calculations.
Thanks in advance.
Sounds like a task for an Aggregator EI pattern:
http://www.enterpriseintegrationpatterns.com/patterns/messaging/Aggregator.html
https://docs.spring.io/spring-integration/docs/5.0.5.RELEASE/reference/html/messaging-routing-chapter.html#aggregator
Both requests should correlate to the same group.
I somehow believe that it doesn't matter for your who will return first: only the concern is to perform some post-processing when all the data is gathered.
However I even think that Scatter-Gather is a good choice for you as well:
https://docs.spring.io/spring-integration/docs/5.0.5.RELEASE/reference/html/messaging-routing-chapter.html#scatter-gather
There is a Thread Barrier implementation also for your consideration:
https://docs.spring.io/spring-integration/docs/5.0.5.RELEASE/reference/html/messaging-routing-chapter.html#barrier
I have a NetIQ (Novell) IDM 4.0.1 driver. In a policy I have a <do-status> rule with level retry.
Does this retry block any other event from being processed?
From the logic of the application the event for (A) can not be processed until the object (B) is associated by the very same driver. Therefore I have added the retry rule on (A). However, it seems that the event for (B) is blocked when the event for (A) is waiting for being retried. If I use veto instead of retry for (A) then the event for (B) is processed regulary.
Is the behaviour specified somewhere?
This takes the top event in the queue, and retries it every 'interval' (which is defined in an Engine Control Value, defaults to 30 seconds).
So yes, it blocks all following events until it completes and stops being a retry.
What you could do is much simpler. In the Input Transform policy set, look for the operation add-association since that is when the object is successfully added to the connected system.
Then do your rule B stuff.
Unless you mean two different objects A and B, that are otherwise unrelated. If so, would let object A logic go through, and when you see object B come through then do the work on object A that is needed.
I am using a standard LRU queue as defined by the ZeroMQ guide figure 41, and I am wondering how to add in protection so that I don't send messages to end points that have disappeared (server crash, OOM killer, anything along those lines).
From the documentation I read that XREP will just drop the message if it is going to a non-existant end-point, and there is no way I get notified about that. Is there a way to get such a notification? Should I just send out a "ping" first and if I don't get a response then that "worker" is dead meat to me? How will I know that it is the same client that I just sent the ping to that I am getting the message back from?
Or is my use case not a good one for ZeroMQ? I just want to make sure that a message has been received, I don't want it being dropped on the floor without my knowledge...
Pinging a worker to know if it is alive will cause a race condition: the worker might well answer the ping just before it dies.
However, if you assume that a worker will not die during a request processing (you can do little in this case), you can reverse the flow of communication between the workers and the central queue. Let the worker fetch a request from the queue (using a REQ/REP connection) and have it send the answer along with the original envelope when the processing is done (using the same socket as above, or even better through a separate PUSH/PULL connection).
With this scenario, you know that a dead worker will not be sent requests, as it will be unable to fetch them (being dead…). Moreover, your central queue can even ensure that it receives an answer to every request in a given time. If it does not, it can put the request back in the queue so that a new worker will fetch it shortly after. This way, even if a worker dies while processing a request, the request will eventually be served.
(as a side note: be careful if the worker crashes because of a particular request - you do not want to kill your workers one by one, and might want to put a maximum number of tries for a request)
Edit: I wrote some code implementing the other direction to explain what I mean.