Summary:
The example for using the Gmail API in Go includes code for reading credentials from a known JSON file inside the filesystem. I would like to take advantage of Application Default Credentials (ADC) since we're deploying to k8s with access to the Gmail API.
Code:
Please find an excerpt from the full demo below:
//error handling omited for brevity
b, _ := ioutil.ReadFile("credentials.json")
config, _ := google.ConfigFromJSON(b, gmail.GmailReadonlyScope) //is it possible to replace this?
client := getClient(config)
srv, _ := gmail.New(client)
How can I replace line 2 in the excerpt to get the same configuration but without using an explicit JSON credentials file?
You can print an access token using gcloud auth application-default print-access-token . Use it to make requests to the API, instead of reading from the file. – Aerials
Related
Hello All,
Wanted to do server-side integration for Gmail API. The basic need is using enabling Gmail API I want to read my Gmail inbox for some analytics purpose.
Golang Package - "google.golang.org/api/gmail/v1"
As per documentation, I have followed the below steps.
New Signed up with Gmail
Added billing details to use GCP services
Created test project
Enabled Gmail API
Created Service Account and key inside it. Got credentials.json file
On the backend side, I am using the Golang package to extract the
Gmail inbox.
After successful integration, I am trying to run my code but getting
the below error
{"level":"error","msg":"Error while creating gmail service : oauth2/google: no credentials found","time":"2021-07-25T15:11:23+05:30"}
Can anyone help me to figure out what is missing?
I currently use OAuth with YouTube and the device flow [1], so maybe this can be helpful. First you need to make a one time request like this:
data := url.Values{
"client_id": {"something.apps.googleusercontent.com"},
"scope": {"https://www.googleapis.com/auth/youtube"},
}
res, err := http.PostForm("https://oauth2.googleapis.com/device/code", data)
You'll get a response like this:
type OAuth struct {
Device_Code string
User_Code string
Verification_URL string
}
which you can do a one time prompt to user like this:
1. Go to
https://www.google.com/device
2. Enter this code
HNDN-ZWBL
3. Sign in to your Google Account
Then, after user log in, you can do an exchange request like this:
data := url.Values{
"client_id": {"something.apps.googleusercontent.com"},
"client_secret": {"super secret"},
"device_code": {"device code from above"},
"grant_type": {"urn:ietf:params:oauth:grant-type:device_code"},
}
res, err := http.PostForm("https://oauth2.googleapis.com/token", data)
This will give you an access_token and refresh_token, that you can save locally for reuse. In addition to "device flow", you also have "native app" flow [2], but I found the former to be a simpler process.
https://developers.google.com/identity/sign-in/devices
https://developers.google.com/identity/protocols/oauth2/native-app
How do I authorize access to the backend(with go google libary) given that ive authenticated the user from the front end? Front end Auth, I have access_token or id_token.
Is there a way to convert id_token to an access token?
Is there a way to use id_token to run calendar.NewService?
Is there a way to use access_token to run calendar.NewService?
my setup
In the extension, I done both:
From GCP creds oauth2 "chrome app" i can get the "access token".
from GCP creds oauth2 "web app", i can get the "id token".
In the backend, using go google api library for calendar
config := &oauth2.Config{...}
// ...
token, err := config.Exchange(ctx, ...)
calendarService, err := calendar.NewService(ctx, option.WithTokenSource(config.TokenSource(ctx, token)))
res, err := calService.Events.List("myemail#gmail.com").Do()
I have no idea how to use my id_token or access_token to use this lib. So far i can do curl requests with the access_token, but that doesnt use this library. is there a way with this google library?
Attempts
Ive read in cross identity, that so long as you point to the same client ID in the same project youre good to go. but i keep getting, token expired or not found
i hear id_token is just jwt. so i tried, but i cant get the types correct, so cant even run it.
jwt, err := google.JWTConfigFromJSON(g.key, gmail.GmailReadonlyScope)
jwt.Subject = "myname#gmail.com" //impersonate user
service, err := calendar.NewService(ctx, option.WithHTTPClient(jwt.Client(ctx)))
tried with oauth2 key.json
serviceAccountKey, err := ioutil.ReadFile("oauth2_webapp.json")
conf, err := google.ConfigFromJSON(serviceAccountKey, calendar.CalendarReadonlyScope)
token, err := conf.Exchange(ctx,"code") // code seems like another method
calendarService, err := calendar.NewService(ctx,
option.WithTokenSource(config.TokenSource(ctx, token)))
res, err := calService.Events.List("myemail#gmail.com").Do()
"code" shouldnt matter, since i do not want to auth the user via browser link. at this point The user should assume already authenticated from front end. but this doesnt work either.
sorry the docs dont have examples. Yeah i tried variations, and finally got it.
id_token is useless.
prior to access token, i had an authCode. I wish in their docs, they said authcode instead of code. i simply passed the auth code from front end to back end. since i am new to this, remove any html encoding. ie(%2f => /). that was also one reason i couldnt get it.
below works:
authCode := "4/3AGEkPVEN9O**70ish char***G0uOPYtQWkUSc"
// authcode was html encoded which the conf.Exchange needed a decoded version.
saKey, err := ioutil.ReadFile("oauth2_webapp.json")
conf, err := google.ConfigFromJSON(saKey, calendar.CalendarReadonlyScope)
token, err := conf.Exchange(ctx,authCode)
Goal:
Generate signed URLs inside GKE pods without manually injecting a service account JSON key. The syntax for generating them requires a service account email and private key.
//import "cloud.google.com/go/storage"
url, err := storage.SignedURL(bucketName, objectName, &storage.SignedURLOptions{
ContentType: contentType,
GoogleAccessID: saEmail,
PrivateKey: saPrivateKey,
})
In other words, I'd like to load saEmail and saPrivateKey from the default credentials automatically available in GKE nodes.
Attempt:
ctx := context.Background()
//errors ignored for brevity
//import "golang.org/x/oauth2/google"
creds, _ := google.FindDefaultCredentials(ctx, storage.ScopeReadWrite)
cfg, _ := google.JWTConfigFromJSON(creds.JSON)
url, _ := storage.SignedURL(bucketName, objectName, &storage.SignedURLOptions{
ContentType: contentType,
GoogleAccessID: cfg.Email,
PrivateKey: cfg.PrivateKey,
})
When I ran google.FindDefaultCredentials() inside a GKE pod, the result JSON is empty.
Environment:
Go 1.13
GKE 1.14.10-gke.36
cloud.google.com/go v0.58.0
cloud.google.com/go/storage v1.8.0
Additional Notes:
I've tested two possible alternatives involving injecting the service account key (JSON) manually into the pod, but I hope to avoid them if possible:
Writing the service account key into a file and setting GOOGLE_APPLICATION_CREDENTIALS to its path. When this is done, google.FindDefaultCredentials() loads the email and private key.
Passing the service account key as a string into the pod and parsing it with google.JWTConfigFromJSON().
For generating a signed URL, you need to have a private key.
When you are on GCP services (here on Compute Instances, the node of your K8S cluster, but it's the same thing with Cloud RUn, Cloud Functions et other GCP services) and you use the default credential (and there is no GOOGLE_APPLICATION_CREDENTIALS env var defined), the library use the metadata server.
The metadata server allows you to generate an access token
curl -H "Metadata-Flavor: Google" \
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token
or an identity token (with the audience in parameter)
curl -H "Metadata-Flavor: Google" \
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/identity?audience=https://www.google.com
Thus, without having any secret or private key on your side, the libraries are able to generate a token (access or identity) for reaching external API.
However, the metadata server doesn't provide the secret (the private key) and you can't use it to generated signed URLs.
You need a service account key file here
You have several way to provide it to the pod in secure manner.
The standard K8S way: use a secret volume
The GCP managed solution: Secret manager. Thanks to the metadata server, you can instantiate the secret manager client and get your secret (your service account key file) and then use it as before. An alternative, I wrote a script that preload the secret into env variables when the container start, and thus, you just have to use the env var without interacting with the secret manager service
I don't recommend you to put your service account key file directly in the container, it's not really secure
Another solution
Ultimately, you can generate on the fly a key and defined it as the service account key (it's name user-defined service account key).
You could generate it when you deploy your service on the cluster. Like this, you don't have to store a secret, it's generated on the fly each time.
You could generate a key at container startup, set it in the service account, and keep it in memory.
Then, use it when you need it in your code and it should work because it's link to your service account.
However, you also need to think how to clean the old and useless keys.
I ran into the same problem and found a solution here:
https://github.com/googleapis/google-cloud-go/issues/1130
TL;DR
Instead of passing in the private key to storage.SignedURL, you can give it a custom signing function. You can use Google's IAM SDK to sign a blob using your Service Account's credentials. That way you don't really need to know the private key at all.
A potential downside is that SignBlob() will perform an HTTP request to do the signing, whereas if you pass in a private key the signature will be computed locally.
Prerequisite: Give your Service Account the Service Account Token Creator role.
Your code would then look like this:
//import (
// "cloud.google.com/go/storage"
// credentialspb "google.golang.org/genproto/googleapis/iam/credentials/v1"
// credentials "cloud.google.com/go/iam/credentials/apiv1"
//)
ctx := context.Background()
saEmail := "your-service-account-email#something-something.iam.gserviceaccount.com"
c, err := credentials.NewIamCredentialsClient(ctx)
if err != nil {
panic(err)
}
url, err := storage.SignedURL(bucketName, objectName, &storage.SignedURLOptions{
ContentType: contentType,
GoogleAccessID: saEmail,
SignBytes: func(b []byte) ([]byte, error) {
req := &credentialspb.SignBlobRequest{
Payload: b,
Name: saEmail,
}
resp, err := c.SignBlob(ctx, req)
if err != nil {
panic(err)
}
return resp.SignedBlob, err
}
})
As of version 1.18.0 you can do it like this:
storageClient, _ := storage.NewClient(ctx)
s, _ := storageClient.Bucket(bucketName).SignedURL(objectName, &storage.SignedURLOptions{
Method: http.MethodGet,
Expires: expires,
})
Quoted from here. Thanks #Fogia for pointing me to the PR that added it.
Because Google AutoML does not have a golang client, I have to use the AutoML http client. To do so, requires an auth token from google which comes from running the following cli command:
gcloud auth application-default print-access-token
I am currently authing my Golang server with a credentials json file that has access to AutoML as well (example usage)
storageClient, err := storage.NewClient(ctx, option.WithCredentialsFile(gcloudCredsJSONPath))
My question is: how would I get an auth token from the Golang Google client if I have a JSON credentials file? Is this even possible?
Thank you for any help!
You can only use API tokens with certain Google Cloud APIs. Using tokens is discourage by Google Cloud as you can read in this article:
https://cloud.google.com/docs/authentication/
If your production environment is also Google Cloud, you might not need to use any JSON file at all. Google Cloud has the concept of "DefaultCredentials" that it injects in your services via the environment. You might be able to simplify your code to:
storageClient, err := storage.NewClient(ctx)
It's also recommended to use a "ServiceAccount" so the credentials that your application use can be scopes to it. You can read more here:
https://cloud.google.com/docs/authentication/getting-started
We are having issues accessing the reseller api using service accounts.
The example with client secrets work well, but we would need to deploy this in k8s (Kubernetes Engine) without the need to refresh the oauth session on a recurring basis (especially doing this once, as it is kinda hard in a docker container).
While there is a lot of documentation on how to do this with python we could not find any way of getting access using a service account.
We tried two accounts, the default compute engine one and one created directly for our use case.
Both got the reseller scope in G Suite.
https://www.googleapis.com/auth/apps.order,
https://www.googleapis.com/auth/admin.directory.user,
https://www.googleapis.com/auth/siteverification,
We keep getting "googleapi: Error 403: Authenticated user is not authorized to perform this action., insufficientPermissions" errors though when using
client, err = google.DefaultClient(ctx, reseller.AppsOrderScope)
if err != nil {
log.Fatal("creating oauth client failed", zap.Error(err))
}
subs, err := client.Subscriptions.List().Do()
if err != nil {
log.Fatal("listing subscriptions failed", zap.Error(err))
}
I read on a post on StackOverflow, that the Reseller API requires user impersonation but searching on this throughout Google and the oauth2 and client lib repos did not result in a way to do this.
Python does this like described in the End-to-End Tutorial
credentials = ServiceAccountCredentials.from_json_keyfile_name(
JSON_PRIVATE_KEY_FILE,
OAUTH2_SCOPES).create_delegated(RESELLER_ADMIN_USER)
but for Go i could not find any documented way of doing this.
So few points here:
Reseller API only requires impersonation / domain-wide delegation when using a service account. In other words, the service account itself has no rights to call the API directly but it does have the ability to impersonate a reseller user (e.g. admin#reseller.example.com or such) who has rights to call the Reseller API.
You may be able to use regular 3-legged OAuth instead of a service account. You just need to make sure you request offline access so that you get a refresh token that is long lived.
Impersonation / domain-wide delegation is not compatible with the default service accounts built-in to AppEngine and ComputeEngine. You must use a service account you created in your API project.
See if the samples Google provides get you where you need to be.
The problem was resolved by using a different config and then setting the jwt.Subject which apparently does impersonation:
const envVar = "GOOGLE_APPLICATION_CREDENTIALS"
if filename := os.Getenv(envVar); filename != "" {
serviceAccount, err := ioutil.ReadFile(filename)
if err != nil {
log.Fatal("creating oauth client failed", zap.Error(err))
}
config, err := google.JWTConfigFromJSON(serviceAccount,
reseller.AppsOrderScope,
)
config.Subject = *impersonationUser // like user#google.com
client = config.Client(ctx)
}