I'm attempting to compose an image in memory and send it out through http.ResponseWriter without ever touching the file system.
I use the following to create a new file:
file := os.NewFile(0, "temp_destination.png")
However, I don't seem to be able to do anything at all with this file. Here is the function I'm using (which is being called within an http.HandleFunc, which just sends the file's bytes to the browser), which is intended to draw a blue rectangle on a temporary file and encode it as a PNG:
func ComposeImage() ([]byte) {
img := image.NewRGBA(image.Rect(0, 0, 640, 480))
blue := color.RGBA{0, 0, 255, 255}
draw.Draw(img, img.Bounds(), &image.Uniform{blue}, image.ZP, draw.Src)
// in memory destination file, instead of going to the file sys
file := os.NewFile(0, "temp_destination.png")
// write the image to the destination io.Writer
png.Encode(file, img)
bytes, err := ioutil.ReadAll(file)
if err != nil {
log.Fatal("Couldn't read temporary file as bytes.")
}
return bytes
}
If I remove the png.Encode call, and just return the file bytes, the server just hangs and does nothing forever.
Leaving the png.Encode call in results in the file bytes (encoded, includes some of the PNG chunks I'd expect to see) being vomited out to stderr/stdout (I can't tell which) and server hanging indefinitely.
I assume I'm just not using os.NewFile correctly. Can anyone point me in the right direction? Alternative suggestions on how to properly perform in-memory file manipulations are welcome.
os.NewFile is a low level function that most people will never use directly. It takes an already existing file descriptor (system representation of a file) and converts it to an *os.File (Go's representation).
If you never want the picture to touch your filesystem, stay out of the os package entirely. Just treat your ResponseWriter as an io.Writer and pass it to png.Encode.
png.Encode(yourResponseWriter, img)
If you insist on writing to an "in memory file", I suggest using bytes.Buffer:
buf := new(bytes.Buffer)
png.Encode(buf, img)
return buf.Bytes()
Please have a detailed read of the NewFile documentation. NewFile does not create a new file, not at all! It sets up a Go os.File which wraps around an existing file with the given file descriptor (0 in your case which is stdin I think).
Serving images without files is much easier: Just Encode your image to your ResponseWriter. That's what interfaces are there for. No need to write to ome magic "in memory file", no need to read it back with ReadAll, plain and simple: Write to your response.
Related
After a lot of tests, we cannot seem to match the speed of gsutil when using the GS Go client libraries.
Even a skeleton file with simplest io.Copy() take A LOT longer the the simplest gsutil.
ctx := context.Background()
client, err := storage.NewClient(ctx, option.WithCredentialsFile(*flags.credsFile))
bucket := client.Bucket("my_bucket")
File, _ := os.Open("path_to_file")
wc := bucket.Object("remoteFile").NewWriter(ctx)
_, _ = io.Copy(wc, File)
err = wc.Close()
Also tried with io.CopyBuffer() when buffer is 128x1024, better, but still slow.
Any way to speed up the upload while using go? we dont want to call any external utilities...
It sounds like the io.Copy implementation is not GCS aware, and instead is doing actual byte copies (reading from the source file and writing to the destination file). In contrast, gsutil is calling the GCS Rewrite API, which for the cases where the source and destination are in the same location and storage class, will do metadata-only copies (avoiding byte copying). Doing it that way is far faster, which would match what you're observing, performance-wise.
Can you use a GCS aware Go implementation -- i.e., one that will call Rewrite rather than reading/writing the underlying object bytes?
My program normally uses the controlling terminal to read input from the user.
// GetCtty gets the file descriptor of the controlling terminal.
func GetCtty() (*os.File, error) {
return os.OpenFile("/dev/tty", os.O_RDONLY, 0)
}
I am currently constructing several times a s := bufio.NewScanner(GetCtty()) during the programm and read the input from s.Scan() with s.Text(). Which works nice.
However, for testing I am simulating the following input on stdin to my CLI Go-Program
echo -e "yes\nno\nyes\n" | app
This will not work correctly because the first construction of s and s.Scan() will already have buffered other test inputs which will not be available to a new construction by bufio.NewScanner and subsequent scan.
I am wondering how I can make sure that only one line is read from the stdin stream by s *bufio.Scanner or how I can mock my input to the controlling terminal.
I had several guesses but I am not sure if they work:
using only one bufio.Scanner in the whole program is a solution but I did not want to go this way...
write back the buffered data to GetCtty() with s.WriteTo(GetCtty()) (?) want work as the stuff gets appended instead of prepended on stdin?
Somehow only read a single line and do not consume more bytes, does that untimately mean to read not in chunks but byte by bytes (?)...
Use iotest.OneByteReader to disable buffering in the scanner:
s := bufio.NewScanner(iotest.OneByteReader(GetCtty()))
I am building a simple testing API in golang for uploading and download image files(PNG, JPEG, JPG):
/pic [POST] for uploading the image and save it to a folder; /pic [GET] for downloading the image to the client.
I have successfully built the /pic [POST] and the image is successfully uploaded to the server's file. And I can open the file in the storage folder. (In both windows localhost server and ubuntu server)
However, when I built the /pic [GET] for downloading the picture, I am able to download the file to the client (my computer), but the downloaded file is somehow corrupted since when I try to open it with different image viewer such as gallery or Photoshop, it says "It looks like we don't support this file format". So it seems like the download is not successful.
Postman result:
File opening in gallery:
Any ideas on why this happens and how should I fix it?
The golang code for downloading the pic is as follows(omitting the error handling):
func PicDownload(w http.ResponseWriter, r *http.Request){
request := make(map[string]string)
reqBody, _ := ioutil.ReadAll(r.Body)
err = json.Unmarshal(reqBody, &request)
// Error handling
file, err := os.OpenFile("./resources/pic/" + request["filename"], os.O_RDONLY, 0666)
// Error handling
buffer := make([]byte, 512)
_, err = file.Read(buffer)
// Error handling
contentType := http.DetectContentType(buffer)
fileStat, _ := file.Stat()
// Set header
w.Header().Set("Content-Disposition", "attachment; filename=" + request["filename"])
w.Header().Set("Content-Type", contentType)
w.Header().Set("Content-Length", strconv.FormatInt(fileStat.Size(), 10))
// Copying the file content to response body
io.Copy(w, file)
return
}
When you are reading the first 512 bytes from the file in order to determine the content type, the underlying file stream pointer moves forward by 512 bytes. When you later call io.Copy, reading continues from that position.
There are two ways to correct this.
The first is to call file.Seek(0, io.SeekStart) before the call to io.Copy(). This will place the pointer back to the start of the file. This solution requires the least amount of code, but means reading the same 512 bytes from the file twice which causes some overhead.
The second solution is to create a buffer that contains the entire file using buffer := make([]byte, fileStat.Size() and using that buffer for both the http.DetectContentType() call and to write the output (write it with w.Write(buffer) instead of using io.Copy(). This approach has the possible downside of loading the entire file into memory at once, which isn't ideal for very large files (io.Copy uses 32KB chunks instead of loading the whole file).
Note: As Peter mentioned in a comment, you must ensure users cannot traverse your filesystem by posting ../../ or something as a filename.
As part of a large file upload feature.
We're using the following to write a 'chunk' of bytes to a file at 'path'. This works fine on local filesystems. Each chunk is correctly written at 'offset'.
f, err := os.OpenFile(path, os.O_APPEND|os.O_WRONLY, os.ModeAppend)
n, err := f.WriteAt(bytes, offset)
On NFS attached storage however the bytes are written at the beginning of the file and not at the requested 'offset'.
Even though it doesn't appear that the process will be able to obtain a lock on the file over NFS. Is there a technique of workaround we could follow to append to a file at 'offset'?
buff := bytes.NewBuffer(somebytes)
How to write on top of buff? Currently I'm creating a new buffer. Is this the right way?
newBuff := bytes.NewBuffer(otherbytes)
newBuff.ReadFrom(buff)
bytes.NewBuffer() returns a *Buffer. *Buffer implements io.Writer (and io.Reader) so you can simply write to it by calling its Write() or WriteString() methods.
Example:
somebytes := []byte("abc")
buff := bytes.NewBuffer(somebytes)
buff.Write([]byte("def"))
fmt.Println(buff)
Output as expected (try it on the Go Playground):
abcdef
If you want to start with an empty buffer, you can simply create an empty Buffer struct (and take its address):
buff := &bytes.Buffer{}
If you want to "overwrite" the current content of the buffer, you can use the Buffer.Reset() method or the equivalent Buffer.Truncate(0) call.
Note that resetting or truncating the buffer will throw away the content (or only a part of it in case of Truncate(). But the allocated buffer (byte slice) in the background is kept and reused.
Note:
What you really want to do is not possible directly: just imagine if you want to insert some data in front of an existing content, the existing content would have to be shifted every time you write / insert something in front of it. This is not really efficient.
Instead create your body in a Buffer. Once it's done, you will know what your header will be. Create the header in another Buffer, and when it's done, copy (write) the body (from the first Buffer) into the second already containing the header.
Or if you don't need to store the whole data, you don't need to create a 2nd Buffer for the header. Once the body is ready, write the header to your output, and then write the body from the Buffer.