I am trying to upload a file to Amazon S3 Using gin framework of Go. Since aws-sdc requires to read the file i need to open file using os.open('filename').But since i am getting the file from "formFile" I don't have the path of the file to open, so os.Open() is giving error
The system cannot find the file specified.
My approach is as follows
package controllers
import (
"bytes"
"log"
"net/http"
"os"
"github.com/gin-gonic/gin"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
)
const (
S3_REGION = "MY REGION"
S3_BUCKET = "<MY BUCKET>"
)
func UploadDocument(c *gin.Context) {
var Buf bytes.Buffer
file, _ := c.FormFile("file")
creds := credentials.NewSharedCredentials("", "default")
s, err := session.NewSession(&aws.Config{
Region: aws.String(S3_REGION),
Credentials: creds,
})
if err != nil {
log.Fatal(err)
}
err = AddFilesToS3(s, file.fileName)
if err != nil {
log.Fatal(err)
}
}
func AddFilesToS3(s *session.Session, fileDir string) error {
file, err := os.Open(fileDir)
if err != nil {
return err
}
defer file.Close()
fileInfo, _ := file.Stat()
var size int64 = fileInfo.Size()
buffer := make([]byte, size)
file.Read(buffer)
_, err = s3.New(s).PutObject(&s3.PutObjectInput{
Bucket: aws.String(S3_BUCKET),
Key: aws.String("myfolder" + "/" + fileDir),
ACL: aws.String("private"),
Body: bytes.NewReader(buffer),
ContentLength: aws.Int64(size),
ContentType: aws.String(http.DetectContentType(buffer)),
ContentDisposition: aws.String("attachment"),
ServerSideEncryption: aws.String("AES256"),
})
return err
}
I am sending my file through POSTMAN like this
what I need to pass to my 'AddFilesToS3' function, since I am sending just the file name, os.Open(fileDir) is failing to look to the actual path of the file.
Where am I doing wrong or is there any better method available to do this?
You're not even reading the file from the form.
You need to call FileHeader.Open. This returns a multipart.File which implements the standard io.Reader interface.
You should change AddFilesToS3 to take a filename and io.Reader. This can then be called for files from gin as well as regular files.
fileHeader, err := c.FormFile("file")
// check err
f, err := fileHeader.Open()
// check err
err = AddFilesToS3(s, fileHeader.fileName, f)
And your AddFilesToS3 is now:
func AddFilesToS3(s *session.Session, fileDir string, r io.Reader) error {
// left as an exercise for the OP
You may need to pass fileHeader.Size() as well.
Related
I am using Pinata to upload files in Go to a private gateway using its submarine feature; while it works well, I am wondering how to pin a directory instead. The documentation says it is "identically to pinning a file, with the main difference being an array of files and need to provide a relative file path for each file in the directory". How can I tweak my current Go code to work for a directory.
package ipfs
import (
"bytes"
"fmt"
"io"
"io/ioutil"
"mime/multipart"
"net/http"
"os"
"path/filepath"
)
func PinFile(filePath string, fileName string) {
//set pinata url for file pinning
url := "https://managed.mypinata.cloud/api/v1/content"
//init method and payload
method := "POST"
payload := &bytes.Buffer{}
writer := multipart.NewWriter(payload)
file, errFile1 := os.Open(filePath)
defer file.Close()
part1, errFile1 := writer.CreateFormFile("files", filepath.Base(filePath))
_, errFile1 = io.Copy(part1, file)
if errFile1 != nil {
fmt.Println(errFile1)
return
}
//set additional pinata option
_ = writer.WriteField("name", "fileName")
_ = writer.WriteField("metadata", "{\"keyvalues\": { \"app\": \"...\" }}")
_ = writer.WriteField("wrapWithDirectory", "false")
_ = writer.WriteField("pinToIPFS", "false")
//close writer if error
err := writer.Close()
if err != nil {
fmt.Println(err)
return
}
client := &http.Client{}
//set new request
req, err := http.NewRequest(method, url, payload)
if err != nil {
fmt.Println(err)
return
}
// set Submarine Key
req.Header.Add("x-api-key", env.GetEnv["SUBMARINE_KEY"])
//add content type to request header
req.Header.Set("Content-Type", writer.FormDataContentType())
res, err := client.Do(req)
if err != nil {
fmt.Println(err)
return
}
defer res.Body.Close()
body, err := ioutil.ReadAll(res.Body)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(string(body))
}
I am trying to get the content of a publicly available file using ioutil.ReadFile() but it doesn't find the file: panic: open http://www.pdf995.com/samples/pdf.pdf: No such file or directory
Here's my code:
// Reading and writing files are basic tasks needed for
// many Go programs. First we'll look at some examples of
// reading files.
package main
import (
"fmt"
"io/ioutil"
)
// Reading files requires checking most calls for errors.
// This helper will streamline our error checks below.
func check(e error) {
if e != nil {
panic(e)
}
}
func main() {
fileInUrl, err := ioutil.ReadFile("http://www.pdf995.com/samples/pdf.pdf")
if err != nil {
panic(err)
}
fmt.Printf("HERE --- fileInUrl: %+v", fileInUrl)
}
Here's a go playground example
ioutil.ReadFile() does not support http.
If you look at the source code(https://golang.org/src/io/ioutil/ioutil.go?s=1503:1549#L42), open the file using os.Open.
I think I can do this coding.
package main
import (
"io"
"net/http"
"os"
)
func main() {
fileUrl := "http://www.pdf995.com/samples/pdf.pdf"
if err := DownloadFile("example.pdf", fileUrl); err != nil {
panic(err)
}
}
func DownloadFile(filepath string, url string) error {
// Get the data
resp, err := http.Get(url)
if err != nil {
return err
}
defer resp.Body.Close()
// Create the file
out, err := os.Create(filepath)
if err != nil {
return err
}
defer out.Close()
// Write the body to file
_, err = io.Copy(out, resp.Body)
return err
}
but, go playgound not protocol(go error dial tcp: Protocol not available).
so, You have to do it PC.
I am trying to deploy a golang code on Heroku. My code needs a text file as input and I need to fetch this text file from S3 bucket. My go-code takes the filename as input, Can someone provide a code snippet for reading a file from S3 and storing its contents into a file?
My GOlang code-
func getDomains(path string) (lines []string, Error error) {
file, err := os.Open(path)
if err != nil {
log.Fatalln(err)
}
defer file.Close()
scanner := bufio.NewScanner(file)
for scanner.Scan() {
lines = append(lines, scanner.Text())
}
return lines, scanner.Err()
}
func Process(w http.ResponseWriter, r *http.Request) {
urls := make(chan *Http, Threads*10)
list, err := getDomains("**NEED A TEXT FILE FROM S3 HERE as an argument**")
if err != nil {
log.Fatalln(err)
}
var wg sync.WaitGroup
for i := 0; i < Threads; i++ {
wg.Add(1)
go func() {
for url := range urls {
url.DNS()
}
wg.Done()
}()
}
for i := 0; i < len(list); i++ {
Progress := fmt.Sprintln(w, len(list))
urls <- &Http{Url: list[i], Num: Progress}
}
close(urls)
wg.Wait()
fmt.Printf("\r%s", strings.Repeat(" ", 100))
fmt.Fprintln(w, "\rTask completed.\n")
}
Can someone suggest a good library for reading the file from S3 into a text file? I cannot download the file from S3 because I have to deploy it on Heroku.
A code snippet for example will be highly appreciated!
The code snippet below should work (given that you have installed the proper dependencies):
package main
import (
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
"github.com/aws/aws-sdk-go/service/s3/s3manager"
"fmt"
"log"
"os"
)
func main() {
// NOTE: you need to store your AWS credentials in ~/.aws/credentials
// 1) Define your bucket and item names
bucket := "<YOUR_BUCKET_NAME>"
item := "<YOUR_ITEM_NAME>"
// 2) Create an AWS session
sess, _ := session.NewSession(&aws.Config{
Region: aws.String("us-west-2")},
)
// 3) Create a new AWS S3 downloader
downloader := s3manager.NewDownloader(sess)
// 4) Download the item from the bucket. If an error occurs, log it and exit. Otherwise, notify the user that the download succeeded.
file, err := os.Create(item)
numBytes, err := downloader.Download(file,
&s3.GetObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(item),
})
if err != nil {
log.Fatalf("Unable to download item %q, %v", item, err)
}
fmt.Println("Downloaded", file.Name(), numBytes, "bytes")
}
For more details you can check the AWS Go SDK and the Github Example
Using current stable AWS lib for go:
sess := session.Must(session.NewSession(&aws.Config{
....
}))
svc := s3.New(sess)
rawObject, err := svc.GetObject(
&s3.GetObjectInput{
Bucket: aws.String("toto"),
Key: aws.String("toto.txt"),
})
buf := new(bytes.Buffer)
buf.ReadFrom(rawObject.Body)
myFileContentAsString := buf.String()
Here is a function for getting an object using V2 of the SDK (adapted from examples in https://github.com/aws/aws-sdk-go-v2):
Note: No Error handling - demo code only.
package s3demo
import (
"os"
"context"
"fmt"
"io/ioutil"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/aws/awserr"
"github.com/aws/aws-sdk-go-v2/aws/external"
"github.com/aws/aws-sdk-go-v2/service/s3"
)
func GetObjectWithV2SDKDemo() {
bucket := "YOUR_BUCKET"
key := "YOUR_OBJECT_KEY"
fileName := "YOUR_FILE_PATH"
// may need AWS_PROFILE and AWS_REGION populated as environment variables
cfg, err := external.LoadDefaultAWSConfig()
if err != nil {
panic("failed to load config, " + err.Error())
}
svc := s3.New(cfg)
ctx := context.Background()
req := svc.GetObjectRequest(&s3.GetObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
})
resp, err := req.Send(ctx)
if err != nil {
panic(err)
}
s3objectBytes, err := ioutil.ReadAll(resp.Body)
if err != nil {
panic(err)
}
// create file
f, err := os.Create(fileName)
defer f.Close()
if err != nil {
panic(err)
}
bytesWritten, err := f.Write(s3objectBytes)
if err != nil {
panic(err)
}
fmt.Printf("Fetched %d bytes for S3Object\n", bytesWritten)
fmt.Printf("successfully downloaded data from %s/%s\n to file %s", bucket, key, fileName)
}
I'm using the Amazon s3 SDK to download files like below:
file, err := os.Create("/tmp/download_file")
downloader := s3manager.NewDownloader(session.New(&aws.Config{
Region: aws.String("us-west-2")}))
numBytes, err := downloader.Download(file,
&s3.GetObjectInput{
Bucket: aws.String(bucketName),
Key: aws.String(fileName),
})
It downloads to a file.
How do I get the download content into a []byte slice (buffer) directly.
I tried something like:
var tBuf bytes.Buffer
tBufIo := bufio.NewWriter(&tBuf)
instead of "file". But I get an error for io.WriterAt interface
cannot use *tBufIo (type bufio.Writer) as type io.WriterAt in argument to downloader.Download
Found it from the link
https://groups.google.com/forum/#!topic/Golang-Nuts/4z8rcWEZ8Os
buff := &aws.WriteAtBuffer{}
downloader := s3manager.NewDownloader(session.New(&aws.Config{
Region: aws.String(S3_Region)}))
numBytes, err := downloader.Download(buff,....
data := buff.Bytes() // now data is my []byte array
Works and fits the need.
See also: https://docs.aws.amazon.com/sdk-for-go/api/aws/#WriteAtBuffer
If you are running AWS SDK v2 then you can create a WriterAt like this
import (
"context"
"errors"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/credentials"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/aws/aws-sdk-go-v2/feature/s3/manager"
)
func DownloadS3File(objectKey string, bucket string, s3Client *s3.Client) ([]byte, error) {
buffer := manager.NewWriteAtBuffer([]byte{})
downloader := manager.NewDownloader(s3Client)
numBytes, err := downloader.Download(context.TODO(), buffer, &s3.GetObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(objectKey),
})
if err != nil {
return nil, err
}
if numBytes < 1 {
return nil, errors.New("zero bytes written to memory")
}
return buffer.Bytes(), nil
}
I'm attempting to upload an image, resize it and then upload it to Amazon S3 in go, however I'm struggling to figure out how to convert the image from multipart.File to image.Image
package controllers
import (
"github.com/gin-gonic/gin"
"github.com/mitchellh/goamz/aws"
"github.com/mitchellh/goamz/s3"
"github.com/nfnt/resize"
_ "image/jpeg"
"log"
"os"
"strconv"
)
type ResizeController struct {
}
func NewResizeController() *ResizeController {
return &ResizeController{}
}
func (rc ResizeController) Resize(c *gin.Context) {
auth, err := aws.EnvAuth()
if err != nil {
log.Fatal(err)
}
client := s3.New(auth, aws.EUWest)
bucket := client.Bucket(os.Getenv("AWS_BUCKET_NAME"))
file, header, err := c.Request.FormFile("file")
filename := header.Filename
height := c.Query("height")
width := c.Query("width")
h64, err := strconv.ParseUint(height, 10, 32)
w64, err := strconv.ParseUint(width, 10, 32)
h := uint(h64)
w := uint(w64)
m := resize.Resize(w, h, file, resize.Lanczos3)
err = bucket.Put("/content/"+filename, m, "content-type", s3.Private)
c.JSON(200, gin.H{"filename": header.Filename})
}
I'm getting the error controllers/resize_controller.go:43: cannot use file (type multipart.File) as type image.Image in argument to resize.Resize:
Figured it out, I just needed to use
image, err := jpeg.Decode(file)