AWS' Golang SDK says that I should use stscreds.AssumeRoleProvider to assume a cross-account role (in this case, for querying another account's DynamoDb table from a web server). This code works:
var sess *session.Session
func init() {
sess = session.Must(session.NewSession(&aws.Config{
Region: aws.String("us-west-2"),
}))
}
func getDynamoDbClient() *dynamodb.DynamoDB {
crossAccountRoleArn := "arn:...:my-cross-account-role-ARN"
creds := stscreds.NewCredentials(sess, crossAccountRoleArn, func(arp *stscreds.AssumeRoleProvider) {
arp.RoleSessionName = "my-role-session-name"
arp.Duration = 60 * time.Minute
arp.ExpiryWindow = 30 * time.Second
})
dynamoDbClient := dynamodb.New(sess, aws.NewConfig().WithCredentials(creds))
return dynamoDbClient
}
According to the documentation, the returned client is thread-safe:
DynamoDB methods are safe to use concurrently.
The question is, since the credential are auto-renewed via stscreds.AssumeRoleProvider, do I
Need to new up a new client on each request (to ensure that I've got unexpired credentials), or
Can I new up a DynamoDb client when the web server starts up, and reuse it for the life of the web server?
Edited To Note:
I dug into the source code for the Golang AWS SDK, and it looks like the credentials returned by stscreds.NewCredentials() are nothing more than a wrapper around a reference to the stscreds.AssumeRoleProvider. So it seems likely to me that the client will magically get auto-renewed credentials.
AWS' documentation leaves something to be desired.
roleArn := "arn:aws:iam::1234567890:role/my-role"
awsSession, _ := session.NewSession(&aws.Config{
Region: aws.String("us-west-2"),
})
stsClient := sts.New(awsSession)
stsRequest := sts.AssumeRoleInput{
RoleArn: aws.String(roleArn),
RoleSessionName: aws.String("my-role-test"),
DurationSeconds: aws.Int64(900), //min allowed
}
stsResponse, err := stsClient.AssumeRole(&stsRequest)
if err != nil {
log.Fatal("an exception occurred when attempting to assume the my role. error=" + err.Error())
}
os.Setenv("AWS_ACCESS_KEY_ID", *stsResponse.Credentials.AccessKeyId)
os.Setenv("AWS_SECRET_ACCESS_KEY", *stsResponse.Credentials.SecretAccessKey)
os.Setenv("AWS_SESSION_TOKEN", *stsResponse.Credentials.SessionToken)
os.Setenv("AWS_SECURITY_TOKEN", *stsResponse.Credentials.SessionToken)
os.Setenv("ASSUMED_ROLE", roleArn)
Related
Im using the following code which works as expected, I use from the cli gcloud auth application-default login and enter my credentials and I was able to run the code successfully from my macbook.
Now I need to run this code in my CI and we need to use different approach , what should be the approach to get the client_secret
and client_id or service account / some ENV variable, what is the way for doing it via GO code?
import "google.golang.org/api/compute/v1"
project := "my-project"
region := "my-region"
ctx := context.Background()
c, err := google.DefaultClient(ctx, compute.CloudPlatformScope)
if err != nil {
log.Fatal(err)
}
computeService, err := compute.New(c)
if err != nil {
log.Fatal(err)
}
req := computeService.Routers.List(project, region)
if err := req.Pages(ctx, func(page *compute.RouterList) error {
for _, router := range page.Items {
// process each `router` resource:
fmt.Printf("%#v\n", router)
// NAT Gateways are found in router.nats
}
return nil
}); err != nil {
log.Fatal(err)
}
Since you're using Jenkins you probably want to start with how to create a service account. It guides you on creating a service account and exporting a key to be set as a var in another CI/CD system.
Then refer to the docs from the client library on how to create a new client with source credential.
e.g.
client, err := storage.NewClient(ctx, option.WithCredentialsFile("path/to/keyfile.json"))
If you provided no source, it would attempt to read the credentials locally and act as the service account running the operation (not applicable in your use case).
Many CIs support the export of specific env vars. Or your script / conf can do it too.
But if you want to run in a CI why you need such configuration? Integration tests?
Some services can be used locally for unit/smoke testing. Like pubsub, there is a way to run a fake/local pubsub to perform some tests.
Or perhaps I did not understand your question, in this case can you provide an example?
I am trying to implement iamcredentials Go API client to generate an Access Token to access some Google APIs via REST API, I am using this code
package main
import (
"context"
"log"
"google.golang.org/api/iamcredentials/v1"
)
func main() {
iamcredentialsService, err := iamcredentials.NewService(context.Background())
if err != nil {
log.Println("error initialize iamcredential Service ", err)
return
}
accessTokenCall := iamcredentialsService.Projects.ServiceAccounts.GenerateAccessToken(
"projects/-/serviceAccounts/some-sa#some-project-id.iam.gserviceaccount.com:generateAccessToken",
&iamcredentials.GenerateAccessTokenRequest{
Scope: []string{
iamcredentials.CloudPlatformScope,
},
},
)
iamResp, err := accessTokenCall.Do()
if err != nil {
log.Println("error generate access token", err)
return
}
log.Println(iamResp)
}
But when I tried to run the above snippet, I got this message
go run main.go
error generate access token googleapi: Error 400: Request contains an invalid argument., badRequest
Is there any way to check which one is causing the above response? I am not sure since there isn't any good example of implementation. Any help would be appreciated, Thanks.
Notes :
I have checked following documentation on this topic https://cloud.google.com/iam/docs/creating-short-lived-service-account-credentials and this https://pkg.go.dev/google.golang.org/api/iamcredentials/v1#pkg-overview
I have already setup the Service account using Service Account Token Creator role on IAM and also enabled the IAM API from the console
Also I have added GOOGLE_APPLICATION_CREDENTIALS to the environment variables as suggested
#DanielFarrell is right, you need to remove the :generateAccessToken at the end. Here the documentation in the code. Don't hesitate to explore it, it's open source ;)
// GenerateAccessToken: Generates an OAuth 2.0 access token for a
// service account.
//
// - name: The resource name of the service account for which the
// credentials are requested, in the following format:
// `projects/-/serviceAccounts/{ACCOUNT_EMAIL_OR_UNIQUEID}`. The `-`
// wildcard character is required; replacing it with a project ID is
// invalid.
func (r *ProjectsServiceAccountsService) GenerateAccessToken(name string, generateaccesstokenrequest *GenerateAccessTokenRequest) *ProjectsServiceAccountsGenerateAccessTokenCall {
c := &ProjectsServiceAccountsGenerateAccessTokenCall{s: r.s, urlParams_: make(gensupport.URLParams)}
c.name = name
c.generateaccesstokenrequest = generateaccesstokenrequest
return c
}
I'd like to create Signed URLs to Google Cloud Storage resources from an app deployed using CloudRun.
I set up CloudRun with a custom Service Account with the GCS role following this guide.
My intent was to use V4 Signing to create Signed URLs from CloudRun. There is a guide for this use-case where a file service_account.json is used to generate JWT config. This works for me on localhost when I download the file from google's IAM. I'd like to avoid having this file committed in the repository use the one that I provided in CloudRun UI.
I was hoping that CloudRun injects this service account file to the app container and makes it accessible in GOOGLE_APPLICATION_CREDENTIALS variable but that's not the case.
Do you have a recommendation on how to do this? Thank you.
As you say, Golang Storage Client Libraries require a service account json file to sign urls.
There is currently a feature request open in GitHub for this but you should be able to work this around with this sample that I found here:
import (
"context"
"fmt"
"time"
"cloud.google.com/go/storage"
"cloud.google.com/go/iam/credentials/apiv1"
credentialspb "google.golang.org/genproto/googleapis/iam/credentials/v1"
)
const (
bucketName = "bucket-name"
objectName = "object"
serviceAccount = "[PROJECTNUMBER]-compute#developer.gserviceaccount.com"
)
func main() {
ctx := context.Background()
c, err := credentials.NewIamCredentialsClient(ctx)
if err != nil {
panic(err)
}
opts := &storage.SignedURLOptions{
Method: "GET",
GoogleAccessID: serviceAccount,
SignBytes: func(b []byte) ([]byte, error) {
req := &credentialspb.SignBlobRequest{
Payload: b,
Name: serviceAccount,
}
resp, err := c.SignBlob(ctx, req)
if err != nil {
panic(err)
}
return resp.SignedBlob, err
},
Expires: time.Now().Add(15*time.Minute),
}
u, err := storage.SignedURL(bucketName, objectName, opts)
if err != nil {
panic(err)
}
fmt.Printf("\"%v\"", u)
}
Cloud Run (and other compute platforms) does not inject a service account key file. Instead, they make access_tokens available on the instance metadata service. You can then exchange this access token with a JWT.
However, often times, Google’s client libraries and gcloud works out of the box on GCP’s compute platforms without explicitly needing to authenticate. So if you use the instructions on the page you linked (gcloud or code samples) it should be working out-of-the-box.
I'm following the Creating HTTP Target tasks guide.
When I run the code posted below I get this error:
cloudtasks.CreateTask: rpc error: code = PermissionDenied
desc = The principal (user or service account)
lacks IAM permission "cloudtasks.tasks.create" for the resource
"projects/my_project/locations/europe-west1/queues/my_queue"
(or the resource may not exist).
I have signed in with gcloud auth login my#email.com.
my#email.com has the following permissions set by my custom cloud task role:
cloudtasks.locations.get
cloudtasks.locations.list
cloudtasks.queues.get
cloudtasks.queues.list
cloudtasks.tasks.create
cloudtasks.tasks.delete
cloudtasks.tasks.fullView
cloudtasks.tasks.get
cloudtasks.tasks.list
cloudtasks.tasks.run
I don't get it. What more should I check?
main.go
// Run `PROJECT_ID=my_project QUEUE_ID=my_queue go run main.go`
package main
import (
"context"
"fmt"
"os"
cloudtasks "cloud.google.com/go/cloudtasks/apiv2"
taskspb "google.golang.org/genproto/googleapis/cloud/tasks/v2"
)
var (
locationID = "europe-west1"
url = "example.com/callback"
message = "testing"
)
func main() {
projectID := os.Getenv("PROJECT_ID")
queueID := os.Getenv("QUEUE_ID")
task, err := createHTTPTask(projectID, locationID, queueID, url, message)
if err != nil {
fmt.Println(err)
}
fmt.Println(task)
}
// createHTTPTask creates a new task with a HTTP target then adds it to a Queue.
func createHTTPTask(projectID, locationID, queueID, url, message string) (*taskspb.Task, error) {
// Create a new Cloud Tasks client instance.
// See https://godoc.org/cloud.google.com/go/cloudtasks/apiv2
ctx := context.Background()
client, err := cloudtasks.NewClient(ctx)
if err != nil {
return nil, fmt.Errorf("NewClient: %v", err)
}
// Build the Task queue path.
queuePath := fmt.Sprintf("projects/%s/locations/%s/queues/%s", projectID, locationID, queueID)
// Build the Task payload.
// https://godoc.org/google.golang.org/genproto/googleapis/cloud/tasks/v2#CreateTaskRequest
req := &taskspb.CreateTaskRequest{
Parent: queuePath,
Task: &taskspb.Task{
// https://godoc.org/google.golang.org/genproto/googleapis/cloud/tasks/v2#HttpRequest
MessageType: &taskspb.Task_HttpRequest{
HttpRequest: &taskspb.HttpRequest{
HttpMethod: taskspb.HttpMethod_POST,
Url: url,
},
},
},
}
// Add a payload message if one is present.
req.Task.GetHttpRequest().Body = []byte(message)
createdTask, err := client.CreateTask(ctx, req)
if err != nil {
return nil, fmt.Errorf("cloudtasks.CreateTask: %v", err)
}
return createdTask, nil
}
The Cloud Tasks API is enabled.
I've been having the same issue for the past couple of days and figured it out. The library I was using to create the API client and create a task was using different credentials than I expected.
For those that are using "application default credentials", or at least letting the client find credentials automatically, take a look at this page: https://cloud.google.com/docs/authentication/production#finding_credentials_automatically
I had created a service account with all the right roles and was assuming the API client was using the service account. Turns out I wasn't passing in the key file and thus it was using the "application default credentials". For my use case, "application default credentials" referred to the App Engine default service account. When I supplied the API client with a key file for my custom service account, it worked.
Application Default Credentials (ADC) provide a method to get credentials used in calling Google APIs. The gcloud auth application-default command group allows you to manage active credentials on your machine that are used for local application development.
Acquire new user credentials to use for ADC with the following command:
gcloud auth application-default login
I am trying to upload a CustomSchema to all Users of a company in GSuite. This Custom Schema contains their Github Usernames, which I extracted with the github API.
The problem is, after running the code, the account in Gsuite is not added.
Relevant code (A connection to GSuite with admin Authentication is established, the map has all user entries. If you still want more code, I can provide you with it - just trying to keep it simple):
for _, u := range allUsers.Users {
if u.CustomSchemas != nil {
log.Printf("%v", string(u.CustomSchemas["User_Names"]))
}else{
u.CustomSchemas = map[string]googleapi.RawMessage{}
}
nameFromGsuite := u.Name.FullName
if githubLogin, ok := gitHubAccs[nameFromGsuite]; ok {
userSchemaForGithub := GithubAcc{GitHub: githubLogin}
jsonRaw, err := json.Marshal(userSchemaForGithub)
if err != nil {
log.Fatalf("Something went wrong logging: %v", err)
}
u.CustomSchemas["User_Names"] = jsonRaw
adminService.Users.Update(u.Id, u)
} else {
log.Printf("User not found for %v\n", nameFromGsuite)
}
}
This is the struct for the json encoding:
type GithubAcc struct {
GitHub string `json:"GitHub"`
}
For anyone stumbling upon this.
Everything in the code snippet is correct. By the way the method is written, I expected that adminService.Users.Update() actually updates the user. Instead, it returns an UserUpdatesCall.
You need to execute that update by calling .Do()
From the API:
Do executes the "directory.users.update" call.
So the solution is to change adminService.Users.Update(u.Id, u)
into adminService.Users.Update(u.Id, u).Do()