Unable to add attendees to google calendar using service account - google-api

Question

I'm having couple of issues with adding attendees to my calendar using google calendar API and service account.
My service account has domain-wide delegation ticket in the cloud console and I am able to create/update a event without adding attendees.
When I try to send request with attendee then I get following response:
Google.Apis.Requests.RequestError
Service accounts cannot invite attendees without Domain-Wide Delegation of Authority. [403]
Errors [Message[Service accounts cannot invite attendees without Domain-Wide Delegation of Authority.] Location[ - ] Reason[forbiddenForServiceAccounts] Domain[calendar]
This is how my code looks like:
var eventDetails = _calendarService.GetEvent(eventDto.EventId).GetAwaiter().GetResult();
eventDetails.Attendees.Add( new EventAttendee
{
DisplayName = "Test Name",
Email = "testemail#example.com" // I tried with few emails but results is same
});
_credential = AuthorizeServiceAccount();
_calendarService = new CalendarService(new BaseClientService.Initializer()
{
HttpClientInitializer = _credential,
ApplicationName = _calendarOptions.AplicationName,
});
var request = _calendarService.Events.Patch(eventRequest,_calendarOptions.CalendarId, eventId);
var result = await request.ExecuteAsync();
public GoogleCredential AuthorizeServiceAccount()
{
var serviceAccountCredential = new ServiceAccountCredential(
new ServiceAccountCredential.Initializer(_calendarOptions.Id) // I'm using service account email here, tried with clientId but same result
{
ProjectId = _calendarOptions.ProjectId,
KeyId = _calendarOptions.KeyId,
User = _calendarOptions.User, // I tried with email from request or null value - for both same result
Scopes = _calendarOptions.Scopes, // ["https://www.googleapis.com/auth/gmail.send", "https://www.googleapis.com/auth/calendar.readonly","https://www.googleapis.com/auth/calendar.events","https://www.googleapis.com/auth/calendar" ],
}
.FromPrivateKey(_calendarOptions.Key));
return GoogleCredential.FromServiceAccountCredential(serviceAccountCredential);
}
Can you please help me so I will be able to add attendees to my events? I don't need to send emails to attendees but emails are required in google API. I don't want also to have G Suite domain account as it's paid.