Multiple Async tasks with timeout and results - async-await

I'm a new in the async world. Regarding your static method (from https://stackoverflow.com/a/25733275/1596974):
static async Task<TResult[]> WhenAll<TResult>(IEnumerable<Task<TResult>> tasks, TimeSpan timeout)
{
var timeoutTask = Task.Delay(timeout).ContinueWith(_ => default(TResult));
var completedTasks =
(await Task.WhenAll(tasks.Select(task => Task.WhenAny(task, timeoutTask)))).
Where(task => task != timeoutTask);
return await Task.WhenAll(completedTasks);
}
How should I use it in order to retrieve the results of those tasks?
Just to be clear, what I need to achieve here is basically this:
For each task I'm calling to several shipping providers in order to get the different shipping rates from them.
Aggregate the response from all the shipping providers into a big list of shipping rates.
Sometimes one (or more) of the shipping providers can be down. So, I need to retrieve the shipping rates from the tasks that completed successfully and just skip the ones that failed.
I hope I was clear enough.

Ok, I ended up with this code that worked pretty well:
var providers = GetShippingProviders().ToList();
var tasks = new Task<Task>[providers.Count];
var timeout = TimeSpan.FromMilliseconds(10000);
try
{
var shippingRates = new List<IShippingRate>();
for (var i = 0; i < tasks.Length; i++)
{
var provider = providers[i];
tasks[i] = Task.WhenAny(Task.Run(() => provider.GetShippingRates(origin, destination, weight)), Task.Delay(timeout));
}
Task.WaitAll(tasks);
foreach (var tasksResult in tasks.Select(x => x.Result).Where(x => x.Status == TaskStatus.RanToCompletion))
{
var shippingRatesResult = tasksResult as Task<List<IShippingRate>>;
if (shippingRatesResult != null)
shippingRates.AddRange(shippingRatesResult.Result.ToList());
}
}
catch (AggregateException ae)
{
Log.Error("An exception occurred when retrieving the shipping Rates.", ae.Flatten());
}
All the tasks that completed successfully are processed. The ones that failed can be just skipped. There is a way to add some code using the "Task.ContinueWith(...)" so the faulted tasks can be catched for logging the exceptions as well.

For the ones that fail, do they throw an exception? Or do they just hang for a long time?
I might do something like this:
var shippingProviderRateTasks = ...;
var results = ...;
foreach (var task in shippingProviderRateTasks) {
try {
results.Add(await task);
} catch (Exception e) {
// log the error here, if you want, and skip this provider
}
}
One of the primary reasons to use Task.WhenAll is to catch an exception when it occurs, and not later on. If you want to swallow all the exceptions and essentially ignore those errors, you might as well just await each of them one at a time - it shouldn't be any slower.

Related

Successful request/response from saga leaves Canceled message in saga skipped queue

After much toil and trial and error I managed to issue a "request" from my saga and see it handle the response. My jubilation was cut short however by the appearance of a message in my states' skipped queue. (i'm using azure service bus)
It is of type "urn:message:MassTransit.Scheduling:CancelScheduledMessage".
I am a complete newbie at with mass transit and I'm just trying to get a contrived example going.
My saga calls TaxiToRunway/TaxiingComplete. My bit of saga code
Request(()=>TaxiToRunway, config =>
{
config.Timeout = TimeSpan.FromSeconds(30);
});
...
public Request<PlaneState, TaxiToRunway, TaxiingComplete> TaxiToRunway { get; private set; }
...
Initially(
When(ReadyToDepart)
.Then(context =>
{
context.Saga.Altitude = 0;
context.Saga.Speed = 0;
context.Saga.FlightNo = context.Message.FlightNo;
context.Saga.CorrelationId = context.Message.CorrelationId;
Console.WriteLine($"Flight {context.Message.FlightNo} is ready to depart.");
})
.TransitionTo(Taxiing)
.Request(TaxiToRunway,
(context) => context.Init<TaxiToRunway>(new {CorrelationId = context.Saga.CorrelationId}))
...
During(Taxiing,
Ignore(ReadyToDepart),
When(TaxiToRunway.Completed)
.Then(x =>
{
x.ToString();
})
.TransitionTo(TakingOff),
With a debugger attached I hit the x.ToString() line.
The consumer (in a different host):
public class TaxiToRunwayConsumer: IConsumer<TaxiToRunway>
{
public async Task Consume(ConsumeContext<TaxiToRunway> context)
{
await context.RespondAsync<TaxiingComplete>(new
{
context.Message.CorrelationId
});
}
}
Saga startup config:
cfg.AddSagaStateMachine<PlaneStateMachine, PlaneState>()
.MessageSessionRepository();
cfg.AddServiceBusMessageScheduler();
cfg.UsingAzureServiceBus((context, sbCfg) =>
{
var connectionString = appConfig.ServiceBus.ConnectionString;
sbCfg.Host(connectionString);
EndpointConvention.Map<TaxiToRunway>(new Uri("sb://xxx.servicebus.windows.net/taxi-to-runway"));
sbCfg.UseServiceBusMessageScheduler();
sbCfg.ReceiveEndpoint("plane-state", e =>
{
e.UseInMemoryOutbox();
e.RequiresSession = true;
e.PrefetchCount = 50;
e.MaxConcurrentCalls = 50;
e.ConfigureSaga<PlaneState>(context);
});
sbCfg.ConfigureEndpoints(context);
});
I can see this in the log output:
dbug: MassTransit.Messages[0]
SEND sb://dbpdf-us-dev-sam.servicebus.windows.net/plane-state 80d90000-5d7b-2cf0-7a6b-08da0fd3e7b7 MassTransit.Scheduling.CancelScheduledMessage
Am I supposed to be handling this as an event??
Learning curve on this sure is steep! My question is what do I need to do to not have these messages go to skipped?
So, the reason this doesn't work:
The message session saga repository can only correlate by the SessionId, since it's session-stored data.
The requestId, therefore, MUST equal the saga instance correlationId (aka, the SessionId)
The timeout message, sent by the request, gets a tokenId based upon the sequence number of the scheduled message
Which isn't saved anywhere
So the request timeout isn't canceled
The proper approach, in this scenario, is to use a Request/Response that doesn't have a timeout and use a separate Schedule to schedule the timeout yourself.

Using BulkInsert with OracleDB

I have some kind of promblem with BulkInsert on my OracleDB. I need to insert couple of thousand objects so I decided to use EF.BulkInsert.Oracle added by Nuget which is extension of EF6.BulkInsert for Oracle.
private IOracleDbContext _context;//Class property
//method body:
EF6.BulkInsert.ProviderFactory.Register<EF6.BulkInsert.Providers.OracleBulkInsertProvider>("BulkInsertProvider");
using (var context = (OracleDbContext)_context)
{
using (var dbContextTransaction = context.Database.BeginTransaction())
{
try
{
//Preparing list of objects
var opt = new EF6.BulkInsert.BulkInsertOptions();
opt.Connection = context.Database.Connection;
await context.BulkInsertAsync<ObjectType>(ObjectList,opt);
await context.SaveChangesAsync();
dbContextTransaction.Commit();
stopwatch.Stop();
}
catch (Exception ex)
{
dbContextTransaction.Rollback();
throw ex;
}
}
}
Without adding opt (BulkInsertOptions object) as parameter of BulkInsert it is trying to connect with SQLServer (which don't exist so I get connection failure). After add this BulkOptions with connection I get exception that connection is already part of transaction :/
Traditional way (_context.TableName.Add() ) of course works but It takes unacceptable amount of time.
Any idea what I did wrong here?
I found better way (BulkInsert still do not cooperate). I used Array Binding
mentioned here
It reduced insert time from ~6 minutes to ~1-1.5 seconds :D (7770 records)

Async SaveChanges In a transaction to run Validations in Parallel

I'm looking for some guidance on an situation I've been contemplating.
I'm creating an Async Crud Operations, and using the time during communication to validate, and only commit the transaction if this is valid.
public async Task<SkillDTO> Update(int id, SkillDTO skillDTO)
{
using (var transaction = this._context.Database.BeginTransaction())
{
var skill = await this._context.Skills.FindAsync(id);
skillDTO.CopyTo(new SkillAdapter(skill));
//Start SaveChanges Execution
var saveChangesTask = this._context.SaveChangesAsync();
//Use free time to validate
var validator = new SkillValidator();
//Note Validator Will throw if not valid
validator.Validate(skill);
await saveChangesTask;
transaction.Commit();
return new SkillAdapter(skill).ToDTO();
}
}
What the pro's and con's?
Is this a dangerous operation because you are executing code that is unneeded if failed validation?
Or does this seem like a valid way to increase efficiency?

Webmasters API - Quota limits

We're trying to download page data for sites using the Webmasters API .NET Client Library, by calling WebmastersService.SearchAnalytics.Query(). To do this we are using Batching and sending approx. 600 requests in one batch. However most of these fail with the error "Quota Exceeded". The amount that fail varies each time but it is only about 10 of the 600 that work (and it varies where they are within the batch). The only way we can get it to work is to reduce the batch size down to 3, and wait 1 second between each call.
According to the Developer Console our daily quota is set to 1,000,000 (and we have 99% remaining) and our per user limit is set to 10,000 requests / second / user.
The error we get back is:
Quota Exceeded [403] Errors [ Message[Quota Exceeded] Location[ - ]
Reason[quotaExceeded] Domain[usageLimits]]
Is there another quota which is enforced? What does "Domain[usage limits]" mean - is the domain the site we are query the page data for, or is it our user account?
We still get the problem if we run each request separately, unless we wait 1 second between each call. Due to the number of sites and the number of pages we need to download the data for this isn't really an option.
I found this post which points out that just because the max batch size is 1000 doesn't mean to say the Google service you are calling supports batches of those sizes. But I'd really like to find out exactly what the quota limits really are (as they don't relate to the Developer Console figures) and how to avoid the errors.
Update 1
Here's some sample code. Its specially written just to prove the problem so no need to comment on it's quality ;o)
using Google.Apis.Auth.OAuth2;
using Google.Apis.Services;
using Google.Apis.Util.Store;
using Google.Apis.Webmasters.v3;
using Google.Apis.Webmasters.v3.Data;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
namespace ConsoleApplication1
{
class Program
{
static void Main(string[] args)
{
new Program().Run().Wait();
}
private async Task Run()
{
List<string> pageUrls = new List<string>();
// Add your page urls to the list here
await GetPageData("<your app name>", "2015-06-15", "2015-07-05", "web", "DESKTOP", "<your domain name>", pageUrls);
}
public static async Task<WebmastersService> GetService(string appName)
{
//if (_service != null)
// return _service;
//TODO: - look at analytics code to see how to store JSON and refresh token and check runs on another PC
UserCredential credential;
using (var stream = new FileStream("c:\\temp\\WMT.json", FileMode.Open, FileAccess.Read))
{
credential = await GoogleWebAuthorizationBroker.AuthorizeAsync(
GoogleClientSecrets.Load(stream).Secrets,
new[] { Google.Apis.Webmasters.v3.WebmastersService.Scope.Webmasters },
"user", CancellationToken.None, new FileDataStore("WebmastersService"));
}
// Create the service.
WebmastersService service = new WebmastersService(new BaseClientService.Initializer()
{
HttpClientInitializer = credential,
ApplicationName = appName,
});
//_service = service;
return service;
}
private static async Task<bool> GetPageData(string appName, string fromDate, string toDate, string searchType, string device, string siteUrl, List<string> pageUrls)
{
// Get the service from the initial method
bool ret = false;
WebmastersService service = await GetService(appName);
Google.Apis.Requests.BatchRequest b = new Google.Apis.Requests.BatchRequest(service);
try
{
foreach (string pageUrl in pageUrls)
{
SearchAnalyticsQueryRequest qry = new SearchAnalyticsQueryRequest();
qry.StartDate = fromDate;
qry.EndDate = toDate;
qry.SearchType = searchType;
qry.RowLimit = 5000;
qry.Dimensions = new List<string>() { "query" };
qry.DimensionFilterGroups = new List<ApiDimensionFilterGroup>();
ApiDimensionFilterGroup filterGroup = new ApiDimensionFilterGroup();
ApiDimensionFilter filter = new ApiDimensionFilter();
filter.Dimension = "device";
filter.Expression = device;
filter.Operator__ = "equals";
ApiDimensionFilter filter2 = new ApiDimensionFilter();
filter2.Dimension = "page";
filter2.Expression = pageUrl;
filter2.Operator__ = "equals";
filterGroup.Filters = new List<ApiDimensionFilter>();
filterGroup.Filters.Add(filter);
filterGroup.Filters.Add(filter2);
qry.DimensionFilterGroups.Add(filterGroup);
var req = service.Searchanalytics.Query(qry, siteUrl);
b.Queue<SearchAnalyticsQueryResponse>(req, (response, error, i, message) =>
{
if (error == null)
{
// Process the results
ret = true;
}
else
{
Console.WriteLine(error.Message);
}
});
await b.ExecuteAsync();
}
}
catch (Exception ex)
{
Console.WriteLine("Exception occurred getting page stats : " + ex.Message);
ret = false;
}
return ret;
}
}
}
Paste this into program.cs of a new console app and add Google.Apis.Webmasters.v3 via nuget. It looks for the wmt.json file in c:\temp but adjust the authentication code to suit your setup. If I add more than 5 page urls to the pageUrls list then I get the Quota Exceeded exception.
I've found that the stated quotas don't really seem to be the quotas. I had to slow my requests down to avoid this same issue (1/sec), even though I was always at or below the stated rate limit (20/sec). Furthermore, it claims that it gives a rateLimitExceeded error in the docs for going too fast, but really it returns a quotaExceeded error. It might have to do with how Google averages the rate of requests over time (as some of the requests we made were simultaneous, even though the long-run average was designed to be at or below 20/sec), but I cannot be sure.

Service Bus - Retrieve message from session by sequence number

I am currently trying to retrieve a specific message from a session.
To do so I use want to use the .Receive(Int64) on the MessageSession where I pass in the sequence number of the message.
Here is my code -
long msgSequenceNr = 1337;
QueueClient queueClient = QueueClient.CreateFromConnectionString(Constants.ServiceBusConnectionString, Constants.TestQueueEntityName, ReceiveMode.PeekLock);
MessageSession msgSession = queueClient.AcceptMessageSession(Constants.TestSessionId);
var peekedMsg = msgSession.Peek(msgSequenceNr); // <-- Works fine!
var receivedMsg = msgSession.Receive(msgSequenceNr); // <-- MessageNotFoundException
Unfortunately the Receive will result in a MessageNotFoundException while the Peek works fine.
Is this a limitation that I missed or is there another way to achieve this.
Note that it is possible that there are multiple messages in the session
Receive with the SequenceNumber can only be used in combination with the Defer method. This is how you would implement it:
Message received, but it can't be processed right now (maybe it's waiting for a different process to complete).
Persist the SequenceNumber in some persistent storage (Table Storage, SQL Database, ...)
When you know that processing can continue (eg: the dependent process is complete), load all SequenceNumbers from your persistent storage.
Use Receive(int sequenceNumber) or ReceiveBatch(int[] sequenceNumbers) to received and process your deferred messages.
Sample application: https://code.msdn.microsoft.com/windowsazure/Brokered-Messaging-ccc4f879#content
Update:
Form your comment I noticed that "undeferring" a deferred message could be a solution. Here's some sample code to undefer the message which copies the deferred message to a new message, Completes the deferred message and sends the new message back in the queue. This uses a TransactionScope to transactionally Complete and Resend the message to avoid the risk of losing the message:
var messageId = "12434539828282";
// Send.
var msg = new BrokeredMessage {SessionId = "user1", MessageId = messageId };
msg.Properties.Add("Language", "Dutch");
queue.Send(msg);
// Receive.
var session = queue.AcceptMessageSession();
msg = session.Receive();
// Store the sequence number.
var sequenceNumber = msg.SequenceNumber;
// Defer.
msg.Defer();
// Change to true to test if the transaction worked.
var shouldThrow = false;
// Later processing of deferred message.
msg = session.Receive(sequenceNumber);
try
{
using (var ts = new TransactionScope())
{
// Create a new message.
var undeferredMessage = new BrokeredMessage {SessionId = msg.SessionId, MessageId = msg.MessageId};
foreach (var prop in msg.Properties)
undeferredMessage.Properties.Add(prop);
// Complete and send within the same transaction.
msg.Complete();
if (shouldThrow)
throw new InvalidOperationException("Some error");
queue.Send(undeferredMessage);
// Complete the transaction.
ts.Complete();
}
}
catch (Exception ex)
{
msg.Abandon();
}
if (shouldThrow)
{
msg = session.Receive(sequenceNumber);
Console.WriteLine(msg.MessageId + " should match: " + messageId);
}
else
{
try
{
msg = session.Receive(sequenceNumber);
}
catch (Exception ex)
{
Console.WriteLine("Message not found, transaction worked OK.");
}
}
Note: here I'm simply taking a copy of the Properties. Keep into account that you might want to copy the Body and any other additional information.

Resources