Google API Client for .NET: How to implement Exponential Backoff - google-api

Question

I've created a method to add members in a Batch Request to a google group using .NET core and google's .NET client library. The code looks like this:
private void InitializeGSuiteDirectoryService()
{
_directoryServiceCredential = GoogleCredential
.FromJson(GlobalSettings.Instance.GSuiteSettings.Credentials)
.CreateScoped(_scopes)
.CreateWithUser(GlobalSettings.Instance.GSuiteSettings.User);
_directoryService = new DirectoryService(new BaseClientService.Initializer()
{
HttpClientInitializer = _directoryServiceCredential,
ApplicationName = _applicationName
});
}
public OperationResult<int> AddGroupMembers(Group group, IEnumerable<Member> members)
{
var result = new OperationResult<int>();
var memberList = members.ToList();
var batchRequestCount = 0;
if (memberList.Any())
{
var request = new BatchRequest(_directoryService);
foreach (var member in memberList)
{
batchRequestCount++;
request.Queue<Members>(_directoryService.Members.Insert(member, group.Id), (content, error, i, message) =>
{
if (message.IsSuccessStatusCode)
{
//log OK
}
else
{
// Implement Exponential backoff only on the request that failed.
}
});
if (batchRequestCount == 30|| member.Equals(memberList.Last()))
{
request.ExecuteAsync().Wait();
request = new BatchRequest(_directoryService); //Clear queue
}
}
}
return result;
}
The logic works fine if the amount of members is small; however, when the members count is let's say 100( this is the max amount of users in my google's test instance), I get an Error from Google that reads: "quotaExceeded". According to Google's documentation, the limit for a batch request on their Admin SDK is 1000 and I've set my logic to Execute when we reach a limit of 30.
The QUESTION is: How do I implement error handling to retry whenever I get this error? Their documentation suggests implementing 'Exponential Backoff' with a response that contains a 'retry-able error'(I don't see this when I inspect my response).

Answer 1

So here's what I ended up doing to implement Exponential Backoff on my call to add members to a Gsuite group. Since I'm using dotnet core, I was able to use 'Polly', which is a resilience and transient-fault-handling library that offers this functionality out of the box. There may be some need for refactoring, but here's what the code looks like for now:
public OperationResult<int> AddGroupMembers(Group group, IEnumerable<Member> members)
{
var result = new OperationResult<int>();
var memberList = members.ToList();
var batchRequestCount = 0;
if (memberList.Any())
{
var request = new BatchRequest(_directoryService);
foreach (var member in memberList)
{
retryRequest = false; // This variable needs to be declared at the class level to guarantee the value is available to the original thread running the process.
batchRequestCount++;
request.Queue<Members>(_directoryService.Members.Insert(member, group.Id), (content, error, i, message) =>
{
// If error code is 'quotaExceeded' retry the request ( You can add as many error codes as you'd like to retry here)
if (error.Code == 403)
{
retryRequest = true;
}
});
// Execute batch request to add members in batches of 30 member max
if (batchRequestCount == 30|| member.Equals(memberList.Last()))
{
// Below is what the code to retry using polly looks like
var response = Policy
.HandleResult<HttpResponseMessage>(message => message.StatusCode == HttpStatusCode.Conflict)
.WaitAndRetry(new[]
{
TimeSpan.FromSeconds(1),
TimeSpan.FromSeconds(2),
TimeSpan.FromSeconds(4)
}, (results, timeSpan, retryCount, context) =>
{
// Log Warn saying a retry was required.
})
.Execute(() =>
{
var httpResponseMsg = new HttpResponseMessage();
// Execute batch request Synchronously
request.ExecuteAsync().Wait();
if (retryRequest)
{
httpResponseMsg.StatusCode = HttpStatusCode.Conflict;
retryRequest = false;
}
else
{
httpResponseMsg.StatusCode = HttpStatusCode.OK;
}
return httpResponseMsg;
});
if (response.IsSuccessStatusCode)
{
// Log info
}
else
{
// Log warn
}
requestCount = 0;
request = new BatchRequest(_directoryService);
batchCompletedCount++;
}
}
}
return result;
}