How to handle large file uploads with low memory footprint? - go

Question

I'm writing an application in Go where user can upload a file and it would be eventually uploaded to Amazon S3 bucket. I've written an endpoint using which user can upload a file as multipart form data. Once the file is uploaded, I'm uploading it to S3 bucket.
func UploadRoutes(route *gin.Engine) {
route.POST("/upload", uploadHandler)
}
func uploadHandler(context *gin.Context) {
fileHeader, err := context.FormFile("file")
// check err
file, err := fileHeader.Open()
// check err
// uploads to S3 bucket
err = utils.Upload(file, fileHeader.Filename)
}
But I am not sure where the uploaded file data is being store between two uploads. It seems like file would be in memory after upload from user is completed and before it is uploaded to S3 bucket (Reference: https://pkg.go.dev/mime/multipart#File)
If that's the case, large file uploads would consume too much server memory. As a workaround, I can think of writing the file on disk and then initiating multipart upload to S3 bucket. Are there better alternatives?
Here is a related question but the I think I'm already using multipart upload as suggested in the answer: AWS S3 uploading/downloading huge files with low memory footprint

Answer 1

you can use Upload, you can pass the multipart reader directly to s3 instead of storing files in server before uploading
func uploadHandler(context *gin.Context) {
fileHeader, err := context.FormFile("file")
// check err
config := &aws.Config{
Region: aws.String("us-west-1"), // your region here
}
AwsSession := session.Must(session.NewSession(config))
uploader := s3manager.NewUploader(AwsSession)
result, err := uploader.Upload(&s3manager.UploadInput{
Bucket: aws.String(bucketname),
Key: aws.String(filePath),
Body: fileHeader, // you can pass the reader here
})
}