We've had an application happily running with OAuth 2 security (service account and a p12 key) for a couple of years, updating Google directory information (i.e. telephone numbers, names, etc.) via the directory API. The account (i.e. user#domain.com) was mistakenly disabled, so the updates stopped working. The account has been re-enabled (I have verified this by logging in), but the application is now getting 403 - "Not Authorized to access this resource/api" errors when attempting to perform updates.
I've gone through everything I can think of the in the Google API console, and everything seems okay with the service account. Can anyone think of something obvious that I should check to figure out the problem?
If it helps any, the Java code used to do the build the authentication is:
HttpTransport httpTransport = new NetHttpTransport();
JsonFactory jsonFactory = new JacksonFactory();
GoogleCredential credential = new GoogleCredential.Builder()
.setTransport(httpTransport)
.setJsonFactory(jsonFactory)
.setServiceAccountId(googleServiceAttributes.getServiceAccountId())
.setServiceAccountScopes(serviceAccountScopes)
.setServiceAccountPrivateKey(googleServiceAttributes.getPrivateKey())
.setServiceAccountUser(googleServiceAttributes.getServiceAccountUsername())
.build();
(the account scope being used is "https://www.googleapis.com/auth/admin.directory.user", the client jar being used for the directory API is google-api-services-admin-directory_v1-rev22-1.16.0-rc.jar and the API jar is google-api-client-1.16.0-rc.jar)
Thanks...
Try to use the same API through 'Try It'.
Related
We are experiencing problems with Authentication of Service Accounts for domain-wide delegation.
The main problem is it's hard to investigate and debug the auth configuration so
we would like to ask for some tips how to debug the configuration.
Or maybe we are missing some configuration options and you can point us to them.
Our process is:
Create SA(=Service Account) with enabled domain-wide delegation.
Authenticate SA in GSuite admin console(https://support.google.com/a/answer/162106?hl=en).
use client_id from the credentials file. (now email)
scopes are comma-separated without spaces between.
Ensure the "Security > API Reference > API Reference -> 'Enable API Access'" is checked.
For some GSuite domains this is working configuration, but we got some domains where this configuration results in:
google.auth.exceptions.RefreshError: ('unauthorized_client: Client is unauthorized to retrieve access tokens using this method.', '{\n "error": "unauthorized_client",\n "error_description": "Client is unauthorized to retrieve access tokens using this method."\n}')
In our understanding, this is the error saying the client_id and scopes were not added to the "Manage API client access" page. (=List of authenticated clients)
We really ensured that the GSuite domain we are requesting has the proper client_id and scopes added in the list of authenticated clients + has the 'Enabled API Access'.
We even created Shared Desktop with them and did it by ourselves to be fully sure of it.
But the error still persists.
However, we are not able to replicate this problem on our test GSuite domain.
We tried couple of options using same SA as the client:
The impersonated account hasn't permissions to access the resource.
This result in:
googleapiclient.errors.HttpError: https://www.googleapis.com/admin/directory/v1/users?customer=my_customer&alt=json returned "Not Authorized to access this resource/api">
The scopes are just partial:
google.auth.exceptions.RefreshError: ('access_denied: Requested client not authorized.', '{\n "error": "access_denied",\n "error_description": "Requested client not authorized."\n}')
The 'Enabled API Access' is not checked.
googleapiclient.errors.HttpError: https://www.googleapis.com/admin/directory/v1/users?customer=my_customer&alt=json returned "Domain cannot use apis.">
The error we are receiving from the client("Client is unauthorized to retrieve access tokens using this method."), we are able to replicate only if the client_id is not in the list of authenticated clients at all.
But we are sure, the problematic GSuite domains have the SA authenticated in "Manage API client access" page.
We are using these scopes: https://www.googleapis.com/auth/userinfo.profile,https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/gmail.readonly,https://www.googleapis.com/auth/plus.login,https://www.googleapis.com/auth/calendar.readonly,https://www.googleapis.com/auth/contacts.readonly, https://www.googleapis.com/auth/admin.directory.user.readonly
Do you have any ideas how to debug/solve this issue?
Here is what you need to do. Double check each step. If in doubt, start over.
Enable "Admin SDK API. This is enabled on a per project basis.
Create a service account. Do not add or remove any privileges. Don't change the service account in any way. If you do you will get an error that you are not authorized.
Enable Domain-wide Delegation on the service account.
Follow this document to delegate domain-wide authority to your service account:
Delegate domain-wide authority to your service account
When creating the service account credentials (from the downloaded Json) you will need the following scopes for full G Suite management:
"https://www.googleapis.com/auth/admin.directory.group",
"https://www.googleapis.com/auth/admin.directory.user"
Impersonate a user account which creates new credentials. The user account needs to be a G Suite superadmin. This account must have logged into G Suite at least once and accepted the Terms of Service.
Create your client using the credentials from step #5.
Working Python Example:
from googleapiclient.discovery import build
from google.oauth2 import service_account
# This is the service account credentials file
credentials_file = 'google-directory-api.json'
# In this example I only need to send email
credentials = service_account.Credentials.from_service_account_file(
credentials_file,
scopes=['https://www.googleapis.com/auth/gmail.send'])
# This user is a G Suite superadmin
impersonate = 'username#example.com'
credentials = credentials.with_subject(impersonate)
service = build('gmail', 'v1', credentials=credentials)
I think we are going to need to take this in stages. Lets start with your first error and see if that fixes your issue.
Background info
There are several types of clients that you can create in Google developer console. Here are the top three.
Browser client: Used for web applications
Native client (other): used for installed desktop applications
Service account: used for server to server communication.
The clients are different the client.json file you download is different and the code used to authenticated to the authentication server is also different.
Error 1: code / client missmatch
unauthorized_client: Client is unauthorized to retrieve access tokens using this method.
Can mean one of two things. Either you have created a service account client and you are not using service account code to authenticate or you are are using the code to authenticate with a service account but you have not created a service account client. You haven't posted what language you are using or any code so I cant tell you if the code you are using is intended to be used with a service account or not. Your going to have to look into this one a bit.
Check in developer console make sure your client is like this If it is check your code. If it isnt then create a real service account client and try again.
Where I work we use Google Apps for Work. For the last 9 months we've been using the Gmail API (~2,000 requests per day) to pull in new emails for our support email accounts.
This is how we originally set it up:
Go to https://console.developers.google.com/project/
Click on the project (or create a new one)
Click on API's & Auth
Click on Credentials
Click on Create new Client ID
Click on Service account
Download a JWT (json) for the account.
Follow the node.js quickstart guide with an installed/native type token for the same account, and authorize it through the console. The JWT tokens did not work unless we did this step, once for each account.
We did this for each of our individual support email accounts to avoid having to turn on domain wide delegation for any of them in the admin console. We were then able to authenticate with the tokens using the officially supported npm library googleapis, similar to this:
var google = require('googleapis');
var jwtClient = new google.auth.JWT(
token.client_email,
null,
token.private_key,
['https://www.googleapis.com/auth/gmail.readonly'],
'supportemail#mycompany.com'
);
jwtClient.authorize(function(err, tokens) {
if (err) {
return cb(err);
}
var gmail = google.gmail('v1');
var requestOptions = {
auth: jwtClient,
userId: 'me',
id: messageId,
format: 'raw'
};
gmail.users.messages.get(requestOptions, function(err, response) {
if (err) {
return cb(err);
}
// do stuff with the response
});
});
Like I said, we used this for a long time and never had any issues. Yesterday around 10am MST every one of the accounts stopped being able to authenticate at the same time, with jwtClient.authorize() suddenly returning the error [Error: unauthorized_client].
I tried doing the same thing with a new token on a new service account (the web interface to get the token has changed quite a bit in the last 9 months), and it returns the same error.
The version of googleapis that we were using was 0.9.7, but we can't get JWT authentication to work on the newest version either.
We opened a ticket with the Google APIs support team, but the support person we spoke with had never read the Gmail API specs before and was ultimately unable to help us, so he redirected us here in order to get in touch with the API engineering support team.
We have noticed that authentication works if we enable the scope for domain wide delegation in the admin console, but we would prefer not to do that. We don't need to impersonate the accounts and would prefer to use an individual JWT for each account.
It turns out that the auth flow we were using was never supported, and probably was broken due to a bugfix on Google's part.
In the question comments #Brandon Jewett-Hall and #Steve Bazyl recommended that we use the installed app auth flow instead, as it allows for indefinite refreshing of access tokens and is supported.
More information about the different auth flows can be found in the Google API docs.
One of the features of our Marketplace app makes use of accessing the user's Gmail account via IMAP. We are using the google-api-java-client and google-oauth-java-client libraries and code similar to this example in the java-gmail-imap project as follows:
GoogleCredential credential = new GoogleCredential.Builder().setTransport(HTTP_TRANSPORT)
.setJsonFactory(JSON_FACTORY)
.setServiceAccountId(SERVICE_ACCOUNT_ID)
.setServiceAccountScopes(Arrays.asList(GMAIL_SCOPE))
.setServiceAccountPrivateKey(PRIVATE_KEY)
.setServiceAccountUser(emailAddress)
.build();
credential.refreshToken();
We are then using code based on the examples at https://code.google.com/p/google-mail-oauth2-tools to make the IMAP connection e.g.
IMAPStore imapStore = OAuth2Authenticator.connectToImap("imap.googlemail.com",
993, emailAddress, credential.getAccessToken(), false);
The majority of the time this appears to work correctly, however we are seeing that for a small but significant number of requests the call to Google made by refreshToken() fails with an HTTP 500 error and an HTML response where the JSON would normally be returned e.g.
<p class="large"><b>500.</b> <ins>That's an error.</ins></p>
<p class="large">The server could not process your request.
<ins>That's all we know.</ins></p>
We were advised by a developer advocate at Google that we refresh tokens are not supported for service accounts and we should be using an approach like in this example.
However, it seems like without the call to refreshToken then accessToken is not populated on the credential object and then this results in a NullPointerException when we call OAuth2Authenticator.connectToImap
From the source for GoogleCredential it did seem like executeRefreshToken() is overridden to handle service accounts i.e. instead of performing a refresh it simply requests a new token, and then this bit of code in Credential then handles populating the access token:
TokenResponse tokenResponse = executeRefreshToken();
if (tokenResponse != null) {
setFromTokenResponse(tokenResponse); ....
We were unsure whether we need to enclose our call to refreshToken() in a retry loop to work around the intermittent 500 errors or whether we need to make other changes to our code to follow the recommended approach for this scenario.
Can anyone advise?
I use the java-gmail-imap example code in production (but it is only used to display an inbox in our University portal, there isn't much interaction that would require me to reuse the same refresh token for instance).
Depending on your usage, I wonder if in your case some kind of throttling is coming into play (I've read in places that Gmail can occasionally throttle access).
Elsewhere I've seen Google APIs talk about making retries using an exponential backoff algorithm.
You have to be a little careful when comparing the usage of OAuth 2.0 with the other Google Service APIs and Gmail. Gmail is special in that it uses XOAUTH2. That said I've seen other Google API's that appear to need the refreshToken call. The documentation is a bit unclear and says things like "Refresh the access token, if necessary" (as you say it doesn't seem to work without this step but I haven't done any experimentation with re-using refresh tokens via credential.setRefreshToken(String refreshToken)).
I'd be interested to hear how you get on.
I am trying to get an access token for Google Service Account. Following is my code -
String SERVICE_ACCOUNT_EMAIL = "edited#developer.gserviceaccount.com";
List scope = new ArrayList();
scope.add("https://www.googleapis.com/auth/admin.directory.user");
String keyFile = "C:\\edited-privatekey.p12";
HttpTransport HTTP_TRANSPORT = new NetHttpTransport();
JsonFactory JSON_FACTORY = new JacksonFactory();
GoogleCredential credential = new GoogleCredential.Builder()
.setTransport(HTTP_TRANSPORT)
.setJsonFactory(JSON_FACTORY)
.setServiceAccountId(SERVICE_ACCOUNT_EMAIL)
.setServiceAccountScopes(scope)
.setServiceAccountPrivateKeyFromP12File(new java.io.File(keyFile))
.build();
credential.refreshToken();
String accessTokens = credential.getAccessToken();
Although the code works fine and I do get an access token, when I try to use it to 'GET' a Google Apps User using the Google Directory APIs, I get a 403 - Forbidden response code. Could someone please help?
I know the code for GET user is correct because it works fine with the access token generated by Google Apps Admin.
You need to set an admin account with:
.setServiceAccountUser(some_admin_email)
And make sure your App (with the correct scopes) is granted access in the cpanel.
Proceed to https://admin.google.com . Login and add Security control if not exists from More Controls.
Click on Security->Advance Settings->Manage ThirdParty OAuth Client Access and check that those scopes are added(comma separated) for your xxxxxxxxxxxxxxxxxx.apps.googleusercontent.com service account id/client id.
You have to enable the specific api before using it inside https://console.developers.google.com/ library, to make it work with your api key.
watch the video https://www.youtube.com/watch?v=s_G5CnAu69M.
I'm trying to invoke any of the Google API using "Service account" authorization access. I have downloaded ".pk2" file and activated "URL Shortener API" in Services tab of Google API console. Whenever I try to invoke any API (URL shortener or Adsense). I've got following exception -
com.google.api.client.auth.oauth2.TokenResponseException: 400 Bad Request
{
"error" : "invalid_grant"
}
at com.google.api.client.auth.oauth2.TokenResponseException.from(TokenResponseException.java:105)
at com.google.api.client.auth.oauth2.TokenRequest.executeUnparsed(TokenRequest.java:303)
at com.google.api.client.auth.oauth2.TokenRequest.execute(TokenRequest.java:323)
at com.google.api.client.googleapis.auth.oauth2.GoogleCredential.executeRefreshToken(GoogleCredential.java:345)
at com.google.api.client.auth.oauth2.Credential.refreshToken(Credential.java:526)
at com.google.api.client.auth.oauth2.Credential.intercept(Credential.java:287)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:836)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:412)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:345)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:463)
Below is code snippet -
HttpTransport HTTP_TRANSPORT = new NetHttpTransport();
JsonFactory JSON_FACTORY = new JacksonFactory();
File privateKey = new File(ReportAdsense.class.getResource("mykey.p12").toURI());
GoogleCredential credential = new GoogleCredential.Builder().setTransport(HTTP_TRANSPORT)
.setJsonFactory(JSON_FACTORY)
.setServiceAccountId("my_valid_account_id#developer.gserviceaccount.com")
.setServiceAccountScopes(UrlshortenerScopes.URLSHORTENER)
.setServiceAccountPrivateKeyFromP12File(privateKey)
.build();
Urlshortener service = new Urlshortener.Builder(new NetHttpTransport(), JSON_FACTORY, null).setHttpRequestInitializer(credential).build();
UrlHistory history = service.url().list().execute();
First of all "Service account" will not work for Adsense, since it requires user authorization. Hence for Adsense you should use Oauth 2.0. When you are authorized first time using URL https://accounts.google.com/o/oauth2/token, copy-paste and hardcode your refresh token. Than you can use it to get access token, specify client_id, client_secret and your refresh_token to get new access token. Now access token can be used in your application.
Regarding your error, I have faced with similar issue and spent plenty of time to resolve it. First of all, make sure that you are using valid ServiceAccountId - it should point to email which finishes with "developer.gserviceaccount.com". Make sure, that you specified account scopes and activated services in Google Console API.
I fixed this issue by synchronizing system clock in my machine.
There are a lot of topics with similar error without answers. Even more, some people says, that sometimes it works, sometimes it returns invalid grant. It could work on one machine and fail on another. I don't know if it is system clock issue, but I would avoid using Service Account API, since looks like there are bugs and support would not help you