I'm trying to create a signed URL for a GET object request in S3. I have this code working flawlessly for putting objects in S3 but I can't seem to get it to work for GET. I sign the URL with this code
//Create the signed url using the company id
func (user *User) signURLForUser(sess *session.Session) (*URLSign, error) {
svc := s3.New(sess)
svc.Config.Region = aws.String(os.Getenv("REGION"))
req, _ := svc.GetObjectRequest(&s3.GetObjectInput{
Bucket: aws.String("bucket"),
Key: aws.String(user.CompanyID + "/" + user.FileRequest),
})
var urlSign URLSign
//urlSign.Size = *out.ContentLength
str, err := req.Presign(time.Minute * 60 * 24 * 5) //Expire in 5 days
if err != nil {
log.Println("Error signing URL Request")
return nil, err
}
urlSign.URL = str
return &urlSign, nil
}
But when I try to use the URL it returns I get this error:
<Error>
<Code>AuthorizationQueryParametersError</Code>
<Message>X-Amz-Algorithm only supports "AWS4-HMAC-SHA256"</Message>
<RequestId>9D7CFB14B195A260</RequestId>
<HostId>
Dgh+SqrHbrdKcbkCYrAj3nObLMAwS7k5+VR1zwC/8ZMS3S4++IAAEXXh3zMZ3CpOAyxX1Kc7Opg=
</HostId>
</Error>
I've check the IAM permissions, they're set for GetObject. I can't think of what else I'm doing wrong.
EDIT: Here's an example of the URL
For sure:
https://rsmachiner-user-code.s3.amazonaws.com//CFDMP_ServoGear.gcode?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=ASIAVEENPDKJRUDZKEVM%2F20180812%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180812T005443Z&X-Amz-Expires=432000&X-Amz-Security-Token=FQoGZXIvYXdzEBoaDIVdv9t408gWWi9vvSLjAaa0pZNA%2BXu83%2FFSyng4XvFdv5%2B7nRB%2FQydMLyi%2BBS84yXqwP6VYn7VlInw4ip1M0lkjHRXQf8OAvQLPrIl%2FQZoTe%2Fy3N6bqhLDOnFVJ3UZzYDQ4%2FbX%2Brc6mvVbkhRsQPiarKBuLYDiOD%2FNoSaItMwI9FsMDknw1qX0Pf%2BZ5La0GmanHrTt9YUI01cIUKJ40No5mKJIwcXw3%2F5QOpUc59rZ2zEzlWP9OXeEwWKp%2Bog5P0v7ABX1lRPsCx4HGEstKhw3ZWmJfQhAcAvhrjmXIMqGNKkaCI5L0ap23jf4GvPMGd4%2BcKIKKvtsF&X-Amz-SignedHeaders=host&X-Amz-Signature=82dfb9b392b5e1ef44c7140259ad884e696b48f8094bdd2d223b8650ebdf59f7
Such issues occur when you are not correctly specifying the algorithm. For me, I was using this url: https://s3_endpoint?X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026
which returned to me the same error. I replaced the \u0026 with & and it worked fine.
Related
While trying to run the msgraph-sdk-go training code from here: https://github.com/microsoftgraph/msgraph-training-go, I'm getting InvalidAuthenticationTokenmsg: Access token is empty while executing the Graph API calls.
I configured a Microsoft developer account with instant sandbox for trial purpose.
I created an app registration as mentioned in the tutorial here and granted required permissions for the app.
The code is able to get the AppToken, but for calls to get Users, it fails with the above error. Am I missing something here?
I tried below code from the example for msgraph-training
func (g *GraphHelper) InitializeGraphForAppAuth() error {
clientId := os.Getenv("CLIENT_ID")
tenantId := os.Getenv("TENANT_ID")
clientSecret := os.Getenv("CLIENT_SECRET")
credential, err := azidentity.NewClientSecretCredential(tenantId, clientId, clientSecret, nil)
if err != nil {
return err
}
g.clientSecretCredential = credential
// Create an auth provider using the credential
authProvider, err := auth.NewAzureIdentityAuthenticationProviderWithScopes(g.clientSecretCredential, []string{
"https://graph.microsoft.com/.default",
})
if err != nil {
return err
}
// Create a request adapter using the auth provider
adapter, err := msgraphsdk.NewGraphRequestAdapter(authProvider)
if err != nil {
return err
}
// Create a Graph client using request adapter
client := msgraphsdk.NewGraphServiceClient(adapter)
g.appClient = client
return nil
}
// This part works, and I get the AppToken with required scope, once decoded.
func (g *GraphHelper) GetAppToken() (*string, error) {
token, err := g.clientSecretCredential.GetToken(context.Background(), policy.TokenRequestOptions{
Scopes: []string{
"https://graph.microsoft.com/.default",
},
})
if err != nil {
return nil, err
}
fmt.Println("expires on : ", token.ExpiresOn)
return &token.Token, nil
}
// The GetUsers function errors out
func (g *GraphHelper) GetUsers() (models.UserCollectionResponseable, error) {
var topValue int32 = 25
query := users.UsersRequestBuilderGetQueryParameters{
// Only request specific properties
Select: []string{"displayName", "id", "mail"},
// Get at most 25 results
Top: &topValue,
// Sort by display name
Orderby: []string{"displayName"},
}
resp, err := g.appClient.Users().
Get(context.Background(),
&users.UsersRequestBuilderGetRequestConfiguration{
QueryParameters: &query,
})
if err != nil {
fmt.Println("Users.Get got Error", err.Error(), resp)
printOdataError(err)
}
resp, err = g.appClient.Users().
Get(context.Background(),
nil)
if err != nil {
fmt.Println("Users.Get got Error with nil", err.Error(), resp)
}
return resp, err
}
I have added the User.Read.All permission in the app as mentioned in the tutorial.
Instead of getting the list of users, I'm getting below error:
Users.Get got Error error status code received from the API <nil>
error: error status code received from the API
code: InvalidAuthenticationTokenmsg: Access token is empty.Users.Get got Error with nil error status code received from the API <nil>
As you are using client Credential Flow ,you can verify your permission in azure portal , if you have User.Read.All delegated permission , Removes the delegated ones and add the corresponding application ones and don't forget to click on grant administrator consent after that. It should then work.
Hope this helps
Thanks.
Okay, so the fix that did work for me after trial and error was a version mismatch in the example and the actual application I was trying out.
The version of the beta msgraph application I was using was v0.49, whereas the msgraphsdk tutorial was using v0.48. The go mod command picked up the latest v0.49 initially I guess, I updated the go.mod file to use v0.48 after looking at the go.mod file from msgraph-training repository and things started working.
Hope this helps someone else later on.
I am attempting to get a signed url from Google cloud where we can upload a document.
func GetSignedURL(bucketName string, objectName string) string {
ctx := context.Background()
// Get a connection to gcloud
client, err := storage.NewClient(ctx, option.WithCredentialsFile(config.GetSettings().GoogleCloudKey))
if err != nil {
msg := err.Error() + " # " + whereami.WhereAmI() + " on " + helpers.GetTimeInTimezone()
panic(msg)
}
defer client.Close()
opts := &storage.SignedURLOptions{
Scheme: storage.SigningSchemeV4,
Method: "PUT",
PrivateKey: []byte(getPrivateKey()), //<- a Google service account private key, obtainable from the Google Developers Console
Expires: time.Now().Add(15 * time.Minute),
Insecure: false,
}
//Use connection to get url
url, err := client.Bucket(bucketName).SignedURL(objectName, opts)
if err != nil {
msg := err.Error() + " # " + whereami.WhereAmI() + " on " + helpers.GetTimeInTimezone()
panic(msg)
}
return url
}
Per https://cloud.google.com/storage/docs/access-control/signing-urls-with-helpers#storage-signed-url-object-go
I do sign the request with my Private key, I get a url similar to this:
https://storage.googleapis.com/yyy-vv-upload-xxx/aa41dcaed3a24f65b8d5a9ac94b4c0a6?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=xxx-assets-gcs-yyy-devops%40yyy-devops.iam.gserviceaccount.com%2F20226667%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20221227T183246Z&X-Goog-Expires=899&X-Goog-Signature=260b2e695999ccdea5d5d5e46f0f83d204609dbfc0efcdf1fa8381b9ec748eacad36b525e695c6fc2fb02277a56e3c10674a75de2ba83c668c2052a89192345ddbdd0fe841ea28dc7df01648f2ca420b43d6fa162540e1c34865847d34c832ded157f2ec58bd8d57c47eef32d632ff02dc1927dc94f7509795e78c357dc556e2c96ea8e86dfb5015c796627f327caf04f45560bad4d5cdb714c5bc97d442cbc87639f44b2a102ce9d9a3fc83e81bcaf9fde38ce56efd02decd3fe8a1cf5fc823e46c85b39d7ada41618993e80b5c2e61a0380d403d0d6763310350ab1cf22d539c80fdaafc2320cbdeb3f1e7c21b4c40e3dc8889b348f67573852d5e7986167e&X-Goog-SignedHeaders=host
When I load it in a browser I get the following:
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message>
<StringToSign>GOOG4-RSA-SHA256 20221227T183246Z 20221227/auto/storage/goog4_request 78f5677e7572233dc56657f7b055601eee26e7913bb6426194c888367c521990</StringToSign>
<CanonicalRequest>GET /yyy-vv-upload-xxx/aa41dcaed3a24f65b8d5a9ac94b4c0a6 X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=xxx-assets-gcs-yyy-devops%40yyy-devops.iam.gserviceaccount.com%2F20221227%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20221227T183246Z&X-Goog-Expires=899&X-Goog-SignedHeaders=host host:storage.googleapis.com host UNSIGNED-PAYLOAD</CanonicalRequest>
</Error>
Any advice?
The error is completely misleading, it should have complained about the http method. Once I put the generated url in Postman and made a PUT request it worked. I could upload an image of a goldfish. The error above is generated for a GET request.
According to Coinbase pro API docs:
The CB-ACCESS-SIGN header is generated by creating a sha256 HMAC using the base64-decoded secret key on the prehash string timestamp +
method + requestPath + body (where + represents string concatenation)
and base64-encode the output. The timestamp value is the same as the
CB-ACCESS-TIMESTAMP header.
The body is the request body string or omitted if there is no request
body (typically for GET requests).
The method should be UPPER CASE.
I borrowed a signing function from a better programmer and feed it something like this:
1619383731POST/reports{{"end_date":"2021-01-02T11:59:59Z","start_date":"2020-01-01T00:00:00Z","type":"account"}}
But keep getting invalid signature from Coinbase.
Signing function for reference:
// sign
func (e *exchange) sign(msg string) string {
key, err := base64.StdEncoding.DecodeString(e.http.secret)
if e.checkErr(err) {
return "bad_sig"
}
signature := hmac.New(sha256.New, key)
_, err = signature.Write([]byte(msg))
if e.checkErr(err) {
return "bad_sig"
}
return base64.StdEncoding.EncodeToString(signature.Sum(nil))
}
Where am I screwing up?
Have you checked out the go-coinbase github repo implementing this as:
https://github.com/preichenberger/go-coinbasepro/blob/master/client.go
h := make(map[string]string)
h["CB-ACCESS-KEY"] = c.Key
h["CB-ACCESS-PASSPHRASE"] = c.Passphrase
h["CB-ACCESS-TIMESTAMP"] = timestamp
message := fmt.Sprintf(
"%s%s%s%s",
timestamp,
method,
url,
data,
)
sig, err := generateSig(message, c.Secret)
if err != nil {
return nil, err
}
h["CB-ACCESS-SIGN"] = sig
return h, nil
It turns out using req.Body directly is the culprit. Why I don't know and something I need to find out, but reading into a []byte and then casting to a string solves part of it.
I am trying to use the aws-sdk-go-v2 to retrieve some data from an S3 bucket. In order to do so I need to be able to set the Request Payer option, however, since I am new to using the SDK, I have no idea how to do so.
I've tried setting this as an env variable AWS_REQUEST_PAYER=requester, but scanning the source code for this golang SDK quickly, I couldn't find that it would be picked up by the SDK as an option.
Using the SDK as directed also fails with an Unauthorized response:
import (
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/feature/s3/manager"
"github.com/aws/aws-sdk-go-v2/service/s3"
)
type awsArchive struct {
bucket string
config config.Config
client *s3.Client
}
func (s *awsArchive) download(uri string, dest string) error {
downloader := manager.NewDownloader(s.client)
paginator := s3.NewListObjectsV2Paginator(s.client, &s3.ListObjectsV2Input{
Bucket: &s.bucket,
Prefix: &uri,
})
for paginator.HasMorePages() {
page, err := paginator.NextPage(context.TODO())
if err != nil {
return err
}
for _, obj := range page.Contents {
if err := downloadToFile(downloader, dest, s.bucket, aws.ToString(obj.Key)); err != nil {
return err
}
}
}
return nil
}
func downloadToFile(downloader *manager.Downloader, targetDirectory, bucket, key string) error {
// Create the directories in the path
file := filepath.Join(targetDirectory, key)
if err := os.MkdirAll(filepath.Dir(file), 0775); err != nil {
return err
}
// Set up the local file
fd, err := os.Create(file)
if err != nil {
return err
}
defer fd.Close()
// Download the file using the AWS SDK for Go
fmt.Printf("Downloading s3://%s/%s to %s...\n", bucket, key, file)
_, err = downloader.Download(context.TODO(), fd, &s3.GetObjectInput{Bucket: &bucket, Key: &key})
return err
}
Error: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: ..., HostID: ..., api error AccessDenied: Access Denied
Would anyone be able to provide me with an example of using the Golang SDK to get S3 files from a Requestor Pays enabled bucket please, i.e. the equivalent of:
aws s3 sync --request-payer requester source_bucket destination_folder
It seems you can use field named 'RequestPayer' in the GetObjectInput struct. Found it from pkg document.
From the link:
type GetObjectInput struct {
// The bucket name containing the object. When using this action with an access
...
Bucket *string
// Key of the object to get.
// This member is required.
Key *string
...
...
// Confirms that the requester knows that they will be charged for the request.
// Bucket owners need not specify this parameter in their requests. For information
// about downloading objects from requester pays buckets, see Downloading Objects
// in Requestor Pays Buckets
// (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html)
// in the Amazon S3 Developer Guide.
RequestPayer types.RequestPayer
You can refer to 'RequestPayer' definition here.
I'm starting to build a regular web app with golang and Angular2, and most importantly I'm trying to secure my login with the help of auth0.com. I download the quickstart code from here and try to run the code, it worked for a while and then next time I run it, the /tmp/session file cannot be found any more.
Here are some basic idea of the code auth0.com provides.
1. Initialize the gorilla sessions filesystemstore
2. Then start the authentification process
code is provided below
app.go
var (
Store *sessions.FilesystemStore
)
func Init() error {
Store = sessions.NewFilesystemStore("", []byte("something-very-secret"))
gob.Register(map[string]interface{}{})
return nil
}
login.go
func LoginHandler(w http.ResponseWriter, r *http.Request) {
domain := os.Getenv("AUTH0_DOMAIN")
aud := os.Getenv("AUTH0_AUDIENCE")
conf := &oauth2.Config{
ClientID: os.Getenv("AUTH0_CLIENT_ID"),
ClientSecret: os.Getenv("AUTH0_CLIENT_SECRET"),
RedirectURL: os.Getenv("AUTH0_CALLBACK_URL"),
Scopes: []string{"openid", "profile"},
Endpoint: oauth2.Endpoint{
AuthURL: "https://" + domain + "/authorize",
TokenURL: "https://" + domain + "/oauth/token",
},
}
if aud == "" {
aud = "https://" + domain + "/userinfo"
}
// Generate random state
b := make([]byte, 32)
rand.Read(b)
state := base64.StdEncoding.EncodeToString(b)
session, err := app.Store.Get(r, "state")
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
session.Values["state"] = state
err = session.Save(r, w)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
audience := oauth2.SetAuthURLParam("audience", aud)
url := conf.AuthCodeURL(state, audience)
http.Redirect(w, r, url, http.StatusTemporaryRedirect)
}
The Error log is
open /tmp/session_46CNLLHBKD... : no such file or directory
I try to understand the code and findout that error log comes from login.go line 39(session, err := app.Store.Get(r, "state")). And I started to track down the code and find out.
login.go:39 -->store.go: 180-->session.go:132-->store.go:186-->store.go:272
you can find store.go and session.go here.
The error log comes from this line: fdata, err := ioutil.ReadFile(filename)
Through the whole process I have not found any function call to save the session file.
I don't understand this error and I don't know why I can run the code at the very beginning, please help me with this problem.
Your any suggestion will be greatly appreciated.Thanks a lot.
I figure out the answer myself
It turns out that I changed my secret key while initializing the gorilla session filesystemstore, but I have not deleted my cookie file in chrome, so it cannot find the tmp sesiion file needed.
I change the key, then delete the coorsponding cookie file and everything is ok now.