I'm attempting to access an array of files and values posted to an API written in Gin (golang). I've got a function which takes a file, height and width. It then calls functions to resize the file, and then upload it to S3. However, I'm attempting to also upload multiple files.
func (rc *ResizeController) Resize(c *gin.Context) {
file, header, err := c.Request.FormFile("file")
filename := header.Filename
if err != nil {
log.Fatal(err)
}
height := c.PostForm("height")
width := c.PostForm("width")
finalFile := rc.Crop(height, width, file)
go rc.Upload(filename, finalFile, "image/jpeg", s3.BucketOwnerFull)
c.JSON(200, gin.H{"filename": filename})
}
I couldn't see anywhere in the docs how to access data in the following format:
item[0]file
item[0]width
item[0]height
item[1]file
item[1]width
item[1]height
etc.
I figured something along the lines of:
for index, element := range c.Request.PostForm("item") {
fmt.Println(element.Height)
}
But that threw "c.Request.Values undefined (type *http.Request has no field or method Values)"
You can access the File slice directly instead of using the FormFile method on Request. Assuming you have a form array for width and height that correspond to the order that the files were uploaded.
if err := ctx.Request.ParseMultipartForm(32 << 20); err != nil {
// handle error
}
for i, fh := range ctx.Request.MultipartForm.File["item"] {
// access file header using fh
w := ctx.Request.MultipartForm.Value["width"][i]
h := ctx.Request.MultipartForm.Value["height"][i]
}
The FormFile method on Request is just a wrapper around MultipartForm.File that returns the first file at that key.
Related
I have the following bit of code:
imageFile, _ := os.Open("image.png") // returns *os.File, error
decodedImage, _, _ := image.Decode(imageFile) // returns image.Image, string, error
// imageFile has been modified!
When I try to work with imageFile after calling image.Decode it no longer behaves the same, leading me to believe image.Decode modified imageFile in some way.
Why is image.Decode modifying the original value while at the same time returning a new value for decodedImage - isn't this misleading?
How do I retain the original value? Is there a way to make a "real" copy of the file, pointing to a new part of allocated memory?
I just started out with Go, so apologies if I am missing something obvious here.
The file on disk is not modified by the code in the question.
The current position of imageFile is somewhere past the beginning of the file after the image is decoded. To read the fie again, Seek back the beginning of the file:
imageFile, err := os.Open("image.png")
if err != nil { log.Fatal(err) }
decodedImage, _, err := image.Decode(imageFile)
if err != nil { log.Fatal(err) }
// Rewind back to the start of the file.
_, err := imageFile.Seek(0, io.SeekStart)
if err != nil { log.Fatal(err) }
// Do something with imageFile here.
Replace the log.Fatal error handling with whatever is appropriate for your application.
I am a newbie to Go. Was starting to write my first code in which I have to download a bunch of CSV's from AWS. I don't understand why it is giving me the below error with O_APPEND mode. If I remove os.O_APPEND, I only get the last file data which is not the objective.
The objective is to download all CSV files into one file locally. I'd like to understand what I'm doing incorrectly.
package main
import (
"fmt"
"os"
"path/filepath"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
"github.com/aws/aws-sdk-go/service/s3/s3manager"
)
const (
AccessKeyId = "xxxxxxxxx"
SecretAccessKey = "xxxxxxxxxxxxxxxxxxxx"
Region = "eu-central-1"
Bucket = "dexter-reports"
bucketKey = "Jenkins/pluginVersions/"
)
func main() {
// Load the Shared AWS Configuration
os.Setenv("AWS_ACCESS_KEY_ID", AccessKeyId)
os.Setenv("AWS_SECRET_ACCESS_KEY", SecretAccessKey)
filename := "JenkinsPluginDetais.txt"
cred := credentials.NewStaticCredentials(AccessKeyId, SecretAccessKey, "")
config := aws.Config{Credentials: cred, Region: aws.String(Region), Endpoint: aws.String("s3.amazonaws.com")}
file, err := os.OpenFile(filename, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0666)
if err != nil {
panic(err)
}
defer file.Close()
sess, err := session.NewSession(&config)
if err != nil {
fmt.Println(err)
}
//list Buckets
ObjectList := listBucketObjects(sess)
//loop over the obectlist. First initialize the s3 downloader via s3manager
downloader := s3manager.NewDownloader(sess)
for _, item := range ObjectList.Contents {
csvFile := filepath.Base(*item.Key)
if csvFile != "pluginVersions" {
downloadBucketObjects(downloader, file, csvFile)
}
}
}
func listBucketObjects(sess *session.Session) *s3.ListObjectsV2Output {
//create a new s3 client
svc := s3.New(sess)
resp, err := svc.ListObjectsV2(&s3.ListObjectsV2Input{
Bucket: aws.String(Bucket),
Prefix: aws.String(bucketKey),
})
if err != nil {
panic(err)
}
return resp
}
func downloadBucketObjects(downloader *s3manager.Downloader, file *os.File, keyobj string) {
fileToDownload := bucketKey + keyobj
numBytes, err := downloader.Download(file,
&s3.GetObjectInput{
Bucket: aws.String(Bucket),
Key: aws.String(fileToDownload),
})
if err != nil {
panic(err)
}
fmt.Println("Downloaded", file.Name(), numBytes, "bytes")
}
Firstly, I don't get it why do you even need os.O_APPEND flag in the first place. As per my understanding, you can omit os.O_APPEND.
Now, let's come to the actual problem of why it's happening:
Doc for O_APPEND (Ref: https://man7.org/linux/man-pages/man2/open.2.html):
O_APPEND
The file is opened in append mode. Before each write(2),
the file offset is positioned at the end of the file, as
if with lseek(2). The modification of the file offset and
the write operation are performed as a single atomic step.
So for every call to write the file offset is positioned at the end of the file.
But (*s3Manager.Download).Download supposedly be using WriteAt method, i.e.,
Doc for WriteAt:
$ go doc os WriteAt
package os // import "os"
func (f *File) WriteAt(b []byte, off int64) (n int, err error)
WriteAt writes len(b) bytes to the File starting at byte offset off. It
returns the number of bytes written and an error, if any. WriteAt returns a
non-nil error when n != len(b).
If file was opened with the O_APPEND flag, WriteAt returns an error.
Notice the last line, that if the file's opened with O_APPEND flag it will result in an error and it's even right because WriteAt's second argument is an offset but mixing O_APPEND's behaviour and WriteAt offset seeking might create problem resulting in unexpected results and it errors out.
Consider the definition of s3manager.Downloader:
func (d Downloader) Download(w io.WriterAt, input *s3.GetObjectInput, options ...func(*Downloader)) (n int64, err error)
The first argument is an io.WriterAt; this interface is:
type WriterAt interface {
WriteAt(p []byte, off int64) (n int, err error)
}
This means that the Download function is going to call the WriteAt method in the File you are passing it. As per the documentation for File.WriteAt
If file was opened with the O_APPEND flag, WriteAt returns an error.
So this explains why you are getting the error but raises the question "why is Download using WriteAt and not accepting an io.Writer (and calling Write)?"; the answer can be found in the documentation:
The w io.WriterAt can be satisfied by an os.File to do multipart concurrent downloads, or in memory []byte wrapper using aws.WriteAtBuffer
So, to increase performance, Downloader might make multiple simultaneous requests for parts of the file and then write these out as they are received (meaning it may not write the data in order). This also explains why calling the function multiple times with the same File results in overwritten data (when Downloader retrieves the each chunk of the file it writes it out at the appropriate position in the output file; this overwrites any data already there).
The above quote from the documentation also points to a possible solution; use an aws.WriteAtBuffer and, once the download is finished, write the data to your file (which could then be opened with O_APPEND) - something like this:
buf := aws.NewWriteAtBuffer([]byte{})
numBytes, err := downloader.Download(buf,
&s3.GetObjectInput{
Bucket: aws.String(Bucket),
Key: aws.String(fileToDownload),
})
if err != nil {
panic(err)
}
_, err = file.Write(buf.Bytes())
if err != nil {
panic(err)
}
An alternative would be to download into a temporary file and then append that to your output file (you may need to do this if the files are large).
I'm trying to use os.open(fileDir) to read a file, then upload that file to an s3 bucket. Here's what I have so far.
func addFileToS3(s *session.Session, fileDir string) error {
file, err := os.Open(fileDir)
if err != nil {
return err
}
defer file.Close()
// Get file size and read the file content into a buffer
fileInfo, _ := file.Stat()
var size int64 = fileInfo.Size()
buffer := make([]byte, size)
file.Read(buffer)
// code to upload to s3
return nil
My directory structure is like
|--project-root
|--test
|--functional
|--assets
|-- good
|--fileINeed
But my code is running inside
|--project-root
|--services
|--service
|--test
|--myGoCode
How do a I pass in the correct fileDir? I need a solution that works locally and when the code gets deployed. I looked at the package path/filepath but I wasn't sure whether to get the absolute path first, then go down the hierarchy or something else.
You can add the following small function to get the expected file path.
var (
_, file, _, _ = runtime.Caller(0)
baseDir = filepath.Dir(file)
projectDir = filepath.Join(baseDir, "../../../")
)
func getFileINeedDirectory() string {
fileINeedDir := path.Join(projectDir, "test/functional/assets/good/fileINeed")
return fileINeedDir // project-dir/test/functional/assets/good/fileINeed
}
I'm studying Go and am a real newbie in this field.
I am facing a problem when I try to copy some value.
What I am doing is:
I want to get some response in [response] using httpRequest.
httpClient := &http.Client{}
response, err := httpClient.Do(req)
if err != nil {
panic(err)
}
After that, I want to save the stored value in response at 'origin.txt'
origin_ ,_:= ioutil.ReadAll(response.Body)
f_, err := os.Create("origin.txt")
f_.Write(origin_);
And I want to get a specific value by using goquery package.
doc, err := goquery.NewDocumentFromReader(response.Body)
if err != nil {
log.Fatal(err)
}
doc.Find(".className").Each(func(i int, s *goquery.Selection) {
w.WriteString("============" + strconv.Itoa(i) + "============")
s.Find("tr").Each(func(i int, s_ *goquery.Selection) {
fmt.Println(s_.Text())
w.WriteString(s_.Text())
})
}
)
But in this case, I can get a value exactly what I want from 2) but cannot get anything from 3).
At first, I think the problem is, the response object at 3) is affected by 2) action. Because it is a reference object.
So I tried to copy it to another object and then do it again.
origin := *response
but, I got the same result as first.
What should I do?
How can I assign a reference value to another one by its value?
Should I request it twice for each attempt?
I actually don't see where you use shared resources between 2 and 3.
However that being said origin := *response won't buy you much. The data (response.Body) is a io.ReadCloser. The ioutil.ReadAll() will consume and store all the data that the stream has. You only get to do this once.
However you have the data stored in origin. If you need another io.Reader for that data (say for case 3), then you can make that byte slice look like an io.Reader again: bytes.NewReader(origin).
func SimpleUploader(r *http.Request, w http.ResponseWriter) {
// temp folder path
chunkDirPath := "./creatives/.uploads/" + userUUID
// create folder
err = os.MkdirAll(chunkDirPath, 02750)
// Get file handle from multipart request
var file io.Reader
mr, err := r.MultipartReader()
var fileName string
// Read multipart body until the "file" part
for {
part, err := mr.NextPart()
if err == io.EOF {
break
}
if part.FormName() == "file" {
file = part
fileName = part.FileName()
fmt.Println(fileName)
break
}
}
// Create files
tempFile := chunkDirPath + "/" + fileName
dst, err := os.Create(tempFile)
defer dst.Close()
buf := make([]byte, 1024*1024)
file.Read(buf)
// write/save buffer to disk
ioutil.WriteFile(tempFile, buf, os.ModeAppend)
if http.DetectContentType(buf) != "video/mp4" {
response, _ := json.Marshal(&Response{"File upload cancelled"})
settings.WriteResponse(w, http.StatusInternalServerError, response)
return
}
// joinedFile := io.MultiReader(bytes.NewReader(buf), file)
_, err = io.Copy(dst, file)
if err != nil {
settings.LogError(err, methodName, "Error copying file")
}
response, _ := json.Marshal(&Response{"File uploaded successfully"})
settings.WriteResponse(w, http.StatusInternalServerError, response)
}
I am uploading a Video file.
Before uploading the entire file I want to do some checks so I save the first 1mb to a file :
buf := make([]byte, 1024*1024)
file.Read(buf)
// write/save buffer to disk
ioutil.WriteFile(tempFile, buf, os.ModeAppend)
Then if the checks pass I want to upload the rest of the file dst is the same file used to save the 1st 1 mb so basically i am trying to append to the file :
_, err = io.Copy(dst, file)
The uploaded file size is correct but the file is corrupted(can't play the video).
What else have I tried? : Joining both the readers and saving to a new file. But with this approach the file size increases by 1 mb and is corrupted.
joinedFile := io.MultiReader(bytes.NewReader(buf), file)
_, err = io.Copy(newDst, joinedFile)
Kindly help.
You've basically opened the file twice by doing os.Create and ioutil.WriteFile
the issue being is that os.Create's return value (dst) is like a pointer to the beginning of that file. WriteFile doesn't move where dst points to.
You are basically doing WriteFile, then io.Copy on top of the first set of bytes WriteFile wrote.
Try doing WriteFile first (with Create flag), and then os.OpenFile (instead of os.Create) that same file with Append flag to append the remaining bytes to the end.
Also, it's extremely risky to allow a client to give you the filename as it could be ../../.bashrc (for example), to which you'd overwrite your shell init with whatever the user decided to upload.
It would be much safer if you computed a filename yourself, and if you need to remember the user's selected filename, store that in your database or even a metadata.json type file that you load later.