I need to upload to an S3 bucket using SignedURLs. I also need to upload multipart files. I have a golang API that vends:
MultiPart Upload
Signed URL for each part
Complete Multipart Upload
Abort Multipart upload
I am attempting to complete a multipart upload. I have a web endpoint that takes an input from my swift 5 client:
type CompletedPart struct {
Etag string
PartNumber int
}
type CompleteMultipartUploadRequest struct {
BucketName string
Key string
UploadID string
MultipartUpload []CompletedPart
}
func putObjectMultipartComplete(w http.ResponseWriter, r *http.Request) {
var completeMultipartUploadRequest CompleteMultipartUploadRequest
err := json.NewDecoder(r.Body).Decode(&completeMultipartUploadRequest)
if err != nil {
http.Error(w, fmt.Sprintf("unable to decode request: %s", err.Error()), http.StatusBadRequest)
return
}
var completedParts []types.CompletedPart
for _, part := range completeMultipartUploadRequest.MultipartUpload {
temp := types.CompletedPart{
ETag: aws.String(part.Etag),
PartNumber: int32(part.PartNumber),
}
completedParts = append(completedParts, temp)
}
input := &s3.CompleteMultipartUploadInput{
Bucket: aws.String(completeMultipartUploadRequest.BucketName),
Key: aws.String(completeMultipartUploadRequest.Key),
UploadId: aws.String(completeMultipartUploadRequest.UploadID),
MultipartUpload: &types.CompletedMultipartUpload {
Parts: completedParts,
},
}
client := s3Client()
response, err := client.CompleteMultipartUpload(context.TODO(), input)
if err != nil {
log.Println(err.Error())
http.Error(w, fmt.Sprintf("could not complete multipart upload: %s", err.Error()), http.StatusFailedDependency)
return
}
object := CompleteMultipartUploadResponse{
BucketName: aws.ToString(response.Bucket),
Key: aws.ToString(response.Key),
Etag: aws.ToString(response.ETag),
}
json, err := json.Marshal(object)
w.Header().Set("Content-Type", "application/json")
w.Write(json)
}
I get a 424 back. When I check the error in the server logs I get:
2022/02/01 16:39:29 operation error S3: CompleteMultipartUpload, https response error StatusCode: 400, RequestID: QREDACTE9, HostID: OdE2MREDACThk=, api error MalformedXML: The XML you provided was not well-formed or did not validate against our published schema
What am I missing here?
This code ended up being almost correct. The AWS S3 API mentioned that a part number ID must be a whole number. I had to start counting at 1 and not 0 as sending a part with the number 0 was not valid.
Related
Questions about attaching a piece of metadata (an "initial" request) when initiating a client-side streaming gRPC have already been asked before (here, here), but some answers are suggesting that it's not possible and suggest using oneof where first request towards a server contains the metadata in question, and subsequent requests contain the actual data to be processed by the server. I'm wondering if it's safe to encode metadata with a binary encoding of choice and send it to the server where it can be extracted from the Context object and deserialized back into meaningful data. I'm fairly certain that it's perfectly fine when it comes to text-based encodings such as JSON. But what about protobuf? Assuming we define our service like so:
service MyService {
rpc ChitChat (stream ChatMessage) returns (stream ChatMessage);
}
message ChatMessage {
// ...
}
message Meta {
// ...
}
We can include a Meta object in the request:
meta := &pb.Meta{
// ...
}
metab, err := proto.Marshal(meta)
if err != nil {
log.Fatalf("marshaling error: %v", err)
}
newCtx := metadata.NewOutgoingContext(context.Background(), metadata.Pairs("meta-bin", string(metab)))
// ...ChitChat(newCtx)
And access it on the server side:
func (s *server) ChitChat(stream pb.MyService_ChitChatServer) error {
md, ok := metadata.FromIncomingContext(stream.Context())
if !ok {
return fmt.Errorf("no metadata received")
}
metaStr := md.Get("meta-bin")
if len(metaStr) != 1 {
return fmt.Errorf("expected 1 md; got: %v", len(metaStr))
}
meta := new(pb.Meta)
if err := proto.Unmarshal([]byte(metaStr[0]), meta); err != nil {
return fmt.Errorf("error during deserialization: %v", err)
}
// ...
return nil
}
It appears to be working quite well - am I missing something? How easy is it to shoot yourself in the foot with this approach?
Yes, gRPC supports binary headers, so this approach isn't invalid; it is a little less clear that it is expected, but then: that's true for the oneof approach too, so ... not much difference there.
We have a Github repository that stores image files.
https://github.com/rollthecloudinc/ipe-objects/tree/dev/media
We would like to serve those image files via golang. The golang api runs on aws api gateway as a lambda function. The function in its current state which goes to a blank screen is below.
func GetMediaFile(req *events.APIGatewayProxyRequest, ac *ActionContext) (events.APIGatewayProxyResponse, error) {
res := events.APIGatewayProxyResponse{StatusCode: 500}
pathPieces := strings.Split(req.Path, "/")
siteName := pathPieces[1]
file, _ := url.QueryUnescape(pathPieces[3]) // pathPieces[2]
log.Printf("requested media site: " + siteName)
log.Printf("requested media file: " + file)
// buf := aws.NewWriteAtBuffer([]byte{})
// downloader := s3manager.NewDownloader(ac.Session)
/*_, err := downloader.Download(buf, &s3.GetObjectInput{
Bucket: aws.String(ac.BucketName),
Key: aws.String("media/" + file),
})
if err != nil {
return res, err
}*/
ext := strings.Split(pathPieces[len(pathPieces)-1], ".")
contentType := mime.TypeByExtension(ext[len(ext)-1])
if ext[len(ext)-1] == "md" {
contentType = "text/markdown"
}
suffix := ""
if os.Getenv("GITHUB_BRANCH") == "master" {
suffix = "-prod"
}
var q struct {
Repository struct {
Object struct {
ObjectFragment struct {
Text string
IsBinary bool
ByteSize int
} `graphql:"... on Blob"`
} `graphql:"object(expression: $exp)"`
} `graphql:"repository(owner: $owner, name: $name)"`
}
qVars := map[string]interface{}{
"exp": githubv4.String(os.Getenv("GITHUB_BRANCH") + ":media/" + file),
"owner": githubv4.String("rollthecloudinc"),
"name": githubv4.String(siteName + suffix),
}
err := ac.GithubV4Client.Query(context.Background(), &q, qVars)
if err != nil {
log.Print("Github latest file failure.")
log.Panic(err)
}
// log.Printf(q.Repository.Object.ObjectFragment.Text)
// json.Unmarshal([]byte(q.Repository.Object.ObjectFragment.Text), &obj)
// log.Printf("END GithubFileUploadAdaptor::LOAD %s", id)
log.Print("content")
log.Print(q.Repository.Object.ObjectFragment.Text)
res.StatusCode = 200
res.Headers = map[string]string{
"Content-Type": contentType,
}
res.Body = q.Repository.Object.ObjectFragment.Text //base64.StdEncoding.EncodeToString([]byte(q.Repository.Object.ObjectFragment.Text))
res.IsBase64Encoded = true
return res, nil
}
The full api file can viewed below but excludes the changes above for migration to Github. This api has been running fine using s3. However, we are now trying to migrate to Github for object storage instead. Have successfully implemented write but are having difficulties described above for reading the file using our lambda.
https://github.com/rollthecloudinc/verti-go/blob/master/api/media/main.go
Help requested to figure out how to serve image files from our Github repo using the golang lambda on aws which can be accessed here as a blank screen.
https://81j44yaaab.execute-api.us-east-1.amazonaws.com/ipe/media/Screen%20Shot%202022-02-02%20at%202.00.29%20PM.png
However, this repo is also a pages site which serves the image just fine.
https://rollthecloudinc.github.io/ipe-objects/media/Screen%20Shot%202022-02-02%20at%202.00.29%20PM.png
Thanks
Further debugging the Text property appears to be empty inside the log.
The IsBinary property value being false lead use to the discovery of a typo. The name input for the graph QL invocation was missing -objects. Once the typo was corrected IsBinary started showing up true. However, the Text property value is still empty.
Having managed to find some similar issues but for uploading many have suggested that graph QL isn't the right tool for uploading binary data to begin with. Therefore, rather than chase tail we have decided to try the Github REST v3 api. Specifically, the go-github package for golang instead.
https://github.com/google/go-github
Perhaps using the REST api instead will lead to successful results.
An additional step was necessary to fetch the blob contents of the object queried via the graph QL api. Once this was achieved the media file was served with success. This required using the go-github blob api to fetch the blob base64 contents from github.
https://81j44yaaab.execute-api.us-east-1.amazonaws.com/ipe/media/Screen%20Shot%202022-02-02%20at%202.00.29%20PM.png
GetMediaFile lambda
func GetMediaFile(req *events.APIGatewayProxyRequest, ac *ActionContext) (events.APIGatewayProxyResponse, error) {
res := events.APIGatewayProxyResponse{StatusCode: 500}
pathPieces := strings.Split(req.Path, "/")
siteName := pathPieces[1]
file, _ := url.QueryUnescape(pathPieces[3]) // pathPieces[2]
log.Print("requested media site: " + siteName)
log.Print("requested media file: " + file)
// buf := aws.NewWriteAtBuffer([]byte{})
// downloader := s3manager.NewDownloader(ac.Session)
/*_, err := downloader.Download(buf, &s3.GetObjectInput{
Bucket: aws.String(ac.BucketName),
Key: aws.String("media/" + file),
})
if err != nil {
return res, err
}*/
ext := strings.Split(pathPieces[len(pathPieces)-1], ".")
contentType := mime.TypeByExtension(ext[len(ext)-1])
if ext[len(ext)-1] == "md" {
contentType = "text/markdown"
}
suffix := ""
if os.Getenv("GITHUB_BRANCH") == "master" {
suffix = "-prod"
}
owner := "rollthecloudinc"
repo := siteName + "-objects" + suffix
var q struct {
Repository struct {
Object struct {
ObjectFragment struct {
Oid githubv4.GitObjectID
} `graphql:"... on Blob"`
} `graphql:"object(expression: $exp)"`
} `graphql:"repository(owner: $owner, name: $name)"`
}
qVars := map[string]interface{}{
"exp": githubv4.String(os.Getenv("GITHUB_BRANCH") + ":media/" + file),
"owner": githubv4.String(owner),
"name": githubv4.String(repo),
}
err := ac.GithubV4Client.Query(context.Background(), &q, qVars)
if err != nil {
log.Print("Github latest file failure.")
log.Panic(err)
}
oid := q.Repository.Object.ObjectFragment.Oid
log.Print("Github file object id " + oid)
blob, _, err := ac.GithubRestClient.Git.GetBlob(context.Background(), owner, repo, string(oid))
if err != nil {
log.Print("Github get blob failure.")
log.Panic(err)
}
res.StatusCode = 200
res.Headers = map[string]string{
"Content-Type": contentType,
}
res.Body = blob.GetContent()
res.IsBase64Encoded = true
return res, nil
}
Full Source: https://github.com/rollthecloudinc/verti-go/blob/master/api/media/main.go
I am trying to test the following line of code:
httpReq.Header.Set("Content-Type", "application/json")
I am mocking the request to an external api in this way:
httpmock.RegisterResponder(http.MethodPost, "do-not-exist.com",
httpmock.NewStringResponder(http.StatusOK, `{
"data":{"Random": "Stuff"}}`),
)
And want to test if the request to the api has the header that I assigned. Is there a way I could achieve this?
With the help of the comment by #Kelsnare I was able to solve this issue in the following way:
httpmock.RegisterResponder(http.MethodPost, "do-not-exist.com",
func(req *http.Request) (*http.Response, error) {
require.Equal(t, req.Header.Get("Content-Type"), "application/json")
resp, _ := httpmock.NewStringResponder(http.StatusOK, `{
"data":{"Random": "Stuff"}}`)(req)
return resp, nil},
)
I wrote my own func of http.Responder type and used httpmock.NewStringResponder inside that func.
response_test.go illustrates how the header is tested:
response, err := NewJsonResponse(200, test.body)
if err != nil {
t.Errorf("#%d NewJsonResponse failed: %s", i, err)
continue
}
if response.StatusCode != 200 {
t.Errorf("#%d response status mismatch: %d ≠ 200", i, response.StatusCode)
continue
}
if response.Header.Get("Content-Type") != "application/json" {
t.Errorf("#%d response Content-Type mismatch: %s ≠ application/json",
i, response.Header.Get("Content-Type"))
continue
You can see an example of table-driven test with httpmock.RegisterResponder here.
I am trying validate JWT returned from a login from AWS Cognito (hosted UI). I noticed that once the login is done in cognito, it tries to access my app with some params like "id_token" and "access_token". Checked with jwt.io and looks like "id_token" is the jwt.
As a test, I wrote a post function in GO expecting a body with the jwt token and the access token (and implemented from this answer)
func auth(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
keyset, err := jwk.Fetch(context.Background(), "https://cognito-idp.{Region}.amazonaws.com/{poolID}/.well-known/jwks.json")
if err != nil {
w.WriteHeader(http.StatusInternalServerError)
json.NewEncoder(w).Encode(&model.ErrorResponse{
Response: model.Response{
Result: false,
},
StatusCd: "500",
StatusDesc: "Failed to fetch jwks. Authorization failed.",
Error: "errRes",
})
}
authRequest := &model.AuthRequest{}
json.NewDecoder(r.Body).Decode(&authRequest)
parsedToken, err := jwt.Parse(
[]byte(authRequest.Token), //This is the JWT
jwt.WithKeySet(keyset),
jwt.WithValidate(true),
jwt.WithIssuer("https://cognito-idp.{Region}.amazonaws.com/{poolID}"),
jwt.WithAudience("{XX APP CLIENT ID XX}"),
jwt.WithClaimValue("key", authRequest.Access), //This is the Access Token
)
if err != nil {
w.WriteHeader(http.StatusInternalServerError)
json.NewEncoder(w).Encode(&model.ErrorResponse{
Response: model.Response{
Result: false,
},
StatusCd: "500",
StatusDesc: "Failed token parse. Authorization failed.",
Error: "errRes",
})
}
result := parsedToken
json.NewEncoder(w).Encode(result)
}
Packages I am using are
"github.com/lestrrat-go/jwx/jwk"
"github.com/lestrrat-go/jwx/jwt"
Obviously, it failed at the token parse. What am I doing wrong and also what should I do with the parsedToken ?
I am new to this so, I have no clue if this is the correct approach and would really like some guidance.
If you're using the github.com/golang-jwt/jwt package (formally known as github.com/dgrijalva/jwt-go,) then you'd probably benefit from this example:
You can check out more JWKs Go examples here: github.com/MicahParks/keyfunc/tree/master/examples.
package main
import (
"fmt"
"log"
"time"
"github.com/golang-jwt/jwt"
"github.com/MicahParks/keyfunc"
)
func main() {
// Get the JWKs URL from your AWS region and userPoolId.
//
// See the AWS docs here:
// https://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-user-pools-using-tokens-verifying-a-jwt.html
regionID := "" // TODO Get the region ID for your AWS Cognito instance.
userPoolID := "" // TODO Get the user pool ID of your AWS Cognito instance.
jwksURL := fmt.Sprintf("https://cognito-idp.%s.amazonaws.com/%s/.well-known/jwks.json", regionID, userPoolID)
// Create the keyfunc options. Use an error handler that logs. Refresh the JWKs when a JWT signed by an unknown KID
// is found or at the specified interval. Rate limit these refreshes. Timeout the initial JWKs refresh request after
// 10 seconds. This timeout is also used to create the initial context.Context for keyfunc.Get.
refreshInterval := time.Hour
refreshRateLimit := time.Minute * 5
refreshTimeout := time.Second * 10
refreshUnknownKID := true
options := keyfunc.Options{
RefreshErrorHandler: func(err error) {
log.Printf("There was an error with the jwt.KeyFunc\nError:%s\n", err.Error())
},
RefreshInterval: &refreshInterval,
RefreshRateLimit: &refreshRateLimit,
RefreshTimeout: &refreshTimeout,
RefreshUnknownKID: &refreshUnknownKID,
}
// Create the JWKs from the resource at the given URL.
jwks, err := keyfunc.Get(jwksURL, options)
if err != nil {
log.Fatalf("Failed to create JWKs from resource at the given URL.\nError:%s\n", err.Error())
}
// Get a JWT to parse.
jwtB64 := "eyJraWQiOiJmNTVkOWE0ZSIsInR5cCI6IkpXVCIsImFsZyI6IlJTMjU2In0.eyJzdWIiOiJLZXNoYSIsImF1ZCI6IlRhc2h1YW4iLCJpc3MiOiJqd2tzLXNlcnZpY2UuYXBwc3BvdC5jb20iLCJleHAiOjE2MTkwMjUyMTEsImlhdCI6MTYxOTAyNTE3NywianRpIjoiMWY3MTgwNzAtZTBiOC00OGNmLTlmMDItMGE1M2ZiZWNhYWQwIn0.vetsI8W0c4Z-bs2YCVcPb9HsBm1BrMhxTBSQto1koG_lV-2nHwksz8vMuk7J7Q1sMa7WUkXxgthqu9RGVgtGO2xor6Ub0WBhZfIlFeaRGd6ZZKiapb-ASNK7EyRIeX20htRf9MzFGwpWjtrS5NIGvn1a7_x9WcXU9hlnkXaAWBTUJ2H73UbjDdVtlKFZGWM5VGANY4VG7gSMaJqCIKMxRPn2jnYbvPIYz81sjjbd-sc2-ePRjso7Rk6s382YdOm-lDUDl2APE-gqkLWdOJcj68fc6EBIociradX_ADytj-JYEI6v0-zI-8jSckYIGTUF5wjamcDfF5qyKpjsmdrZJA"
// Parse the JWT.
token, err := jwt.Parse(jwtB64, jwks.KeyFunc)
if err != nil {
log.Fatalf("Failed to parse the JWT.\nError:%s\n", err.Error())
}
// Check if the token is valid.
if !token.Valid {
log.Fatalf("The token is not valid.")
}
log.Println("The token is valid.")
}
I would suggest to start out by doing the minimal checks -- i.e., first try just parsing without validation, then add validations one by one:
jwt.Parse([]byte(token)) // probably fails because of JWS
jwt.Parse([]byte(token), jwt.WithKeySet(...)) // should be OK?
jwt.Parse(..., jwt.WithValidation(true), ...) // add conditions one by one
Please note that I have no idea what's in id_token, as I have never used Cognito If it's a raw JWT, you shouldn't need a key set, and (1) should work.
I am trying to use the aws-sdk-go-v2 to retrieve some data from an S3 bucket. In order to do so I need to be able to set the Request Payer option, however, since I am new to using the SDK, I have no idea how to do so.
I've tried setting this as an env variable AWS_REQUEST_PAYER=requester, but scanning the source code for this golang SDK quickly, I couldn't find that it would be picked up by the SDK as an option.
Using the SDK as directed also fails with an Unauthorized response:
import (
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/feature/s3/manager"
"github.com/aws/aws-sdk-go-v2/service/s3"
)
type awsArchive struct {
bucket string
config config.Config
client *s3.Client
}
func (s *awsArchive) download(uri string, dest string) error {
downloader := manager.NewDownloader(s.client)
paginator := s3.NewListObjectsV2Paginator(s.client, &s3.ListObjectsV2Input{
Bucket: &s.bucket,
Prefix: &uri,
})
for paginator.HasMorePages() {
page, err := paginator.NextPage(context.TODO())
if err != nil {
return err
}
for _, obj := range page.Contents {
if err := downloadToFile(downloader, dest, s.bucket, aws.ToString(obj.Key)); err != nil {
return err
}
}
}
return nil
}
func downloadToFile(downloader *manager.Downloader, targetDirectory, bucket, key string) error {
// Create the directories in the path
file := filepath.Join(targetDirectory, key)
if err := os.MkdirAll(filepath.Dir(file), 0775); err != nil {
return err
}
// Set up the local file
fd, err := os.Create(file)
if err != nil {
return err
}
defer fd.Close()
// Download the file using the AWS SDK for Go
fmt.Printf("Downloading s3://%s/%s to %s...\n", bucket, key, file)
_, err = downloader.Download(context.TODO(), fd, &s3.GetObjectInput{Bucket: &bucket, Key: &key})
return err
}
Error: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: ..., HostID: ..., api error AccessDenied: Access Denied
Would anyone be able to provide me with an example of using the Golang SDK to get S3 files from a Requestor Pays enabled bucket please, i.e. the equivalent of:
aws s3 sync --request-payer requester source_bucket destination_folder
It seems you can use field named 'RequestPayer' in the GetObjectInput struct. Found it from pkg document.
From the link:
type GetObjectInput struct {
// The bucket name containing the object. When using this action with an access
...
Bucket *string
// Key of the object to get.
// This member is required.
Key *string
...
...
// Confirms that the requester knows that they will be charged for the request.
// Bucket owners need not specify this parameter in their requests. For information
// about downloading objects from requester pays buckets, see Downloading Objects
// in Requestor Pays Buckets
// (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html)
// in the Amazon S3 Developer Guide.
RequestPayer types.RequestPayer
You can refer to 'RequestPayer' definition here.