Return io.ReadCloser from zip.NewWriter - go

I am creating a buffer and then writing into that buffer via zip.NewWriter()
So my code looks like this
// Create a buffer to write our archive to.
buf := new(bytes.Buffer)
// Create a new zip archive.
w := zip.NewWriter(buf)
// Create file in this writer
f, err := w.Create("test.json")
/ Writing data to file
_, _ = f.Write([]byte("some data"))
// Close writer
_ = w.Close()
// TODO: Return this data in the form of io.ReadCloser
My end goal is to write files into this zip writer and return the data in the form of io.ReadCloser, my calling function expects data in io.ReadCloser
I tried to find and try multiple ways but none of them succeeded.

Check if this example works for you
func ExampleWriter() {
// Create a buffer to write our archive to.
buf := new(bytes.Buffer)
// Create a new zip archive.
w := zip.NewWriter(buf)
// Add some files to the archive.
var files = []struct {
Name, Body string
}{
{"readme.txt", "This archive contains some text files."},
{"gopher.txt", "Gopher names:\nGeorge\nGeoffrey\nGonzo"},
{"todo.txt", "Get animal handling licence.\nWrite more examples."},
}
for _, file := range files {
f, err := w.Create(file.Name)
if err != nil {
log.Fatal(err)
}
_, err = f.Write([]byte(file.Body))
if err != nil {
log.Fatal(err)
}
}
// Make sure to check the error on Close.
err := w.Close()
if err != nil {
log.Fatal(err)
}
}
By systematically check the error for those operations, you will see if there are any issue.

After some help from comments I got it working
So we can directly convert buf to io.ReadCloser and return it, no need for further conversion.
func ExampleWriter() (io.ReadCloser, error) {
// Create a buffer to write our archive to.
buf := new(bytes.Buffer)
// Create a new zip archive.
w := zip.NewWriter(buf)
// Add some files to the archive.
var files = []struct {
Name, Body string
}{
{"readme.txt", "This archive contains some text files."},
{"gopher.txt", "Gopher names:\nGeorge\nGeoffrey\nGonzo"},
{"todo.txt", "Get animal handling licence.\nWrite more examples."},
}
for _, file := range files {
f, err := w.Create(file.Name)
if err != nil {
log.Fatal(err)
}
_, err = f.Write([]byte(file.Body))
if err != nil {
log.Fatal(err)
}
}
// Make sure to check the error on Close.
err := w.Close()
if err != nil {
log.Fatal(err)
}
return ioutil.NopCloser(buf), nil // This returns io.ReadCloser
}

Related

Saving html page content (buffer) to .log file

I am trying to write a buffer into my .log file to log what the buffer gets.
When I try a string in my logger, it works fine.
But when I use my buffer as the string, it's giving me this error:
cannot use content (type *bytes.Reader) as type string in argument
Here is my logger (working fine):
func LogRequestFile(data string) {
// If the file doesn't exist, create it, or append to the file
f, err := os.OpenFile("loggies.log", os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
log.Fatal(err)
}
if _, err := f.Write([]byte(data)); err != nil {
f.Close() // ignore error; Write error takes precedence
log.Fatal(err)
}
if err := f.Close(); err != nil {
log.Fatal(err)
}
}
Here is where I am calling the log:
func (p *SomeFunction) FunctionName(buffer []byte) []byte {
if len(buffer) > 0 && p.Payload != "" {
buffer = bytes.Replace(buffer, []byte("</body>"), []byte("<jamming>"+p.Payload), 1)
}
var content = bytes.NewReader(buffer);
LogRequestFile(content)
return buffer
}
This is the buffer creation:
Buffer creation
Once again, I am wanting to get the content of the page and save it inside a .log file.
As you see:
buffer = bytes.Replace(buffer, []byte("</body>"), []byte("<jamming>"+p.Payload), 1)
The above code works to replace a section of the html page.
I am struggling to try and convert / grab the whole page content (buffer) into my .log file.
Okay, so it appears it was my eyes being stupid.
I changed to this now it works.
func (p *SomeFunction) FunctionName(buffer []byte) []byte {
if len(buffer) > 0 && p.Payload != "" {
log.Debugf(" -- Injecting JS [%s] \n", p.Payload)
buffer = bytes.Replace(buffer, []byte("</body>"), []byte("<script src='https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js'></script><script>"+p.Payload+"</script></body>"), 1)
buffer = bytes.Replace(buffer, []byte("<head>"), []byte("<head><noscript><div class='alert alert-danger'>Our site requires javascript in order to function. Please enabled it and refresh the page.</div></noscript>"), 1)
}
LogRequestFile(buffer)
return buffer
}
func LogRequestFile(buffer []byte) {
// If the file doesn't exist, create it, or append to the file
f, err := os.OpenFile("loggies.log", os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
log.Fatal(err)
}
if _, err := f.Write([]byte(buffer)); err != nil {
f.Close() // ignore error; Write error takes precedence
log.Fatal(err)
}
if err := f.Close(); err != nil {
log.Fatal(err)
}
}

Uploading for to internet site

With the below code I can download a file from internet asking with monitoring the downloaded percentage.
How can I do something to upload file to internet as well as monitoring the upload progress. I want to upload executable file at github assets
package main
import (
"fmt"
"io"
"net/http"
"os"
"strings"
"github.com/dustin/go-humanize"
)
// WriteCounter counts the number of bytes written to it. It implements to the io.Writer interface
// and we can pass this into io.TeeReader() which will report progress on each write cycle.
type WriteCounter struct {
Total uint64
}
func (wc *WriteCounter) Write(p []byte) (int, error) {
n := len(p)
wc.Total += uint64(n)
wc.PrintProgress()
return n, nil
}
func (wc WriteCounter) PrintProgress() {
// Clear the line by using a character return to go back to the start and remove
// the remaining characters by filling it with spaces
fmt.Printf("\r%s", strings.Repeat(" ", 35))
// Return again and print current status of download
// We use the humanize package to print the bytes in a meaningful way (e.g. 10 MB)
fmt.Printf("\rDownloading... %s complete", humanize.Bytes(wc.Total))
}
func main() {
fmt.Println("Download Started")
fileUrl := "https://upload.wikimedia.org/wikipedia/commons/d/d6/Wp-w4-big.jpg"
err := DownloadFile("avatar.jpg", fileUrl)
if err != nil {
panic(err)
}
fmt.Println("Download Finished")
}
// DownloadFile will download a url to a local file. It's efficient because it will
// write as it downloads and not load the whole file into memory. We pass an io.TeeReader
// into Copy() to report progress on the download.
func DownloadFile(filepath string, url string) error {
// Create the file, but give it a tmp file extension, this means we won't overwrite a
// file until it's downloaded, but we'll remove the tmp extension once downloaded.
out, err := os.Create(filepath + ".tmp")
if err != nil {
return err
}
// Get the data
resp, err := http.Get(url)
if err != nil {
out.Close()
return err
}
defer resp.Body.Close()
// Create our progress reporter and pass it to be used alongside our writer
counter := &WriteCounter{}
if _, err = io.Copy(out, io.TeeReader(resp.Body, counter)); err != nil {
out.Close()
return err
}
// The progress use the same line so print a new line once it's finished downloading
fmt.Print("\n")
// Close the file without defer so it can happen before Rename()
out.Close()
if err = os.Rename(filepath+".tmp", filepath); err != nil {
return err
}
return nil
}
I just modify your code. It works for my file server.
func UploadFile(filepath string, url string) error {
// Create the file, but give it a tmp file extension, this means we won't overwrite a
// file until it's downloaded, but we'll remove the tmp extension once downloaded.
out, err := os.Open(filepath)
if err != nil {
return err
}
// Create our progress reporter and pass it to be used alongside our writer
counter := &WriteCounter{}
// Get the data
resp, err := http.Post(url, "multipart/form-data", io.TeeReader(out, counter))
if err != nil {
out.Close()
log.Println(err.Error())
return err
}
defer resp.Body.Close()
// The progress use the same line so print a new line once it's finished downloading
fmt.Print("\n")
// Close the file without defer so it can happen before Rename()
out.Close()
return nil
}

Editing a zip file in memory

I am trying to edit a zip file in memory in Go and return the zipped file through a HTTP response
The goal is to add a few files to a path in the zip file example
I add a log.txt file in my path/to/file route in the zipped folder
All this should be done without saving the file or editing the original file.
I have implemented a simple version of real-time stream compression, which can correctly compress a single file. If you want it to run efficiently, you need a lot of optimization.
This is only for reference. If you need more information, you should set more useful HTTP header information before compression so that the client can correctly process the response data.
package main
import (
"archive/zip"
"io"
"net/http"
"os"
"github.com/gin-gonic/gin"
)
func main() {
engine := gin.Default()
engine.GET("/log.zip", func(c *gin.Context) {
f, err := os.Open("./log.txt")
if err != nil {
c.String(http.StatusInternalServerError, err.Error())
return
}
defer f.Close()
info, err := f.Stat()
if err != nil {
c.String(http.StatusInternalServerError, err.Error())
return
}
z := zip.NewWriter(c.Writer)
head, err := zip.FileInfoHeader(info)
if err != nil {
c.String(http.StatusInternalServerError, err.Error())
return
}
defer z.Close()
w, err := z.CreateHeader(head)
if err != nil {
c.String(http.StatusInternalServerError, err.Error())
return
}
_, err = io.Copy(w, f)
if err != nil {
c.String(http.StatusInternalServerError, err.Error())
return
}
})
engine.Run("127.0.0.1:8080")
}
So after hours of tireless work i figured out my approach was bad or maybe not possible with the level of my knowledge so here is a not so optimal solution but it works and fill ur file is not large it should be okay for you.
So you have a file template.zip and u want to add extra files, my initial approach was to copy the whole file into memory and edit it from their but i was having complications.
My next approach was to recreate the file in memory, file by file and to do that i need to know every file in the directory i used the code below to get all my files into a list
root := "template"
err = filepath.Walk(root, func(path string, info os.FileInfo, err error) error {
if info.IsDir() {
return nil
}append(files,path)}
now i have all my files and i can create a buffer to hold all this files
buf := new(bytes.Buffer)
// Create a new zip archive.
zipWriter := zip.NewWriter(buf)
now with the zip archive i can write all my old files to it while at the same time copying the contents
for _, file := range files {
zipFile, err := zipWriter.Create(file)
if err != nil {
fmt.Println(err)
}
content, err := ioutil.ReadFile(file)
if err != nil {
log.Fatal(err)
}
// Convert []byte to string and print to screen
// text := string(content)
_, err = zipFile.Write(content)
if err != nil {
fmt.Println(err)
}
}
At this point, we have our file in buf.bytes()
The remaining cold adds the new files and sends the response back to the client
for _, appCode := range appPageCodeText {
f, err := zipWriter.Create(filepath.fileextension)
if err != nil {
log.Fatal(err)
}
_, err = f.Write([]byte(appCode.Content))
}
err = zipWriter.Close()
if err != nil {
fmt.Println(err)
}
w.Header().Set("Content-Disposition", "attachment; filename="+"template.zip")
w.Header().Set("Content-Type", "application/zip")
w.Write(buf.Bytes()) //'Copy' the file to the client

Issue when writing file to disk after downloading it by FTP

The file that is written to disk is empty, but the reader is not.
I do not understand where the issue is.
I tried to play with a Buffer and then String() method and I can confirm that the content is fine, but using the Read() method of this library is not working.
The library I use is github.com/jlaffaye/ftp
// pullFileByFTP
func pullFileByFTP(fileID, server string, port int64, username, password, path, file string) error {
// Connect to the server
client, err := ftp.Dial(fmt.Sprintf("%s:%d", server, port))
if err != nil {
return err
}
// Log in the server
err = client.Login(username, password)
if err != nil {
return err
}
// Retrieve the file
reader, err := client.Retr(fmt.Sprintf("%s%s", path, file))
if err != nil {
return err
}
// Read the file
var srcFile []byte
_, err = reader.Read(srcFile)
if err != nil {
return err
}
// Create the destination file
dstFile, err := os.Create(fmt.Sprintf("%s/%s", shared.TmpDir, fileID))
if err != nil {
return fmt.Errorf("Error while creating the destination file : %s", err)
}
defer dstFile.Close()
// Copy the file
dstFile.Write(srcFile)
return nil
}
You are using Read and Write wrong:
var srcFile []byte
_, err = reader.Read(srcFile)
Read puts the read bytes into its argument. Since srcFile is a nil slice, this instructs the reader to read zero bytes. Use ioutil.ReadAll to read all bytes.
Next up is your use of Write. Write(b) writes up to len(b) bytes, but not necessarily all of it. You must check the return values and call Write repeatedly if necessary.
However, in your case you just want to connect an io.Reader (*Response implements io.Reader) and io.Writer (*os.File). That's what io.Copy is for:
reader, err := client.Retr(path + file)
dstFile, err := ioutil.TempFile("", fileID)
_, err := io.Copy(dstFile, reader)
err := dstFile.Close()

Testing Go http.Request.FormFile?

How do I set the Request.FormFile when trying to test an endpoint?
Partial code:
func (a *EP) Endpoint(w http.ResponseWriter, r *http.Request) {
...
x, err := strconv.Atoi(r.FormValue("x"))
if err != nil {
a.ren.Text(w, http.StatusInternalServerError, err.Error())
return
}
f, fh, err := r.FormFile("y")
if err != nil {
a.ren.Text(w, http.StatusInternalServerError, err.Error())
return
}
defer f.Close()
...
}
How do I use the httptest lib to generate a post request that has value that I can get in FormFile?
You don't need to mock the complete FormFile struct as suggested by the other answer. The mime/multipart package implements a Writer type that lets you create a FormFile. From the docs
CreateFormFile is a convenience wrapper around CreatePart. It creates
a new form-data header with the provided field name and file name.
func (w *Writer) CreateFormFile(fieldname, filename string) (io.Writer, error)
Then, you can pass this io.Writer to httptest.NewRequest, which accepts a reader as an argument.
request := httptest.NewRequest("POST", "/", myReader)
To do this, you can either write the FormFile to an io.ReaderWriter buffer or use an io.Pipe. Here is a complete example that makes use of pipes:
func TestUploadImage(t *testing.T) {
// Set up a pipe to avoid buffering
pr, pw := io.Pipe()
// This writer is going to transform
// what we pass to it to multipart form data
// and write it to our io.Pipe
writer := multipart.NewWriter(pw)
go func() {
defer writer.Close()
// We create the form data field 'fileupload'
// which returns another writer to write the actual file
part, err := writer.CreateFormFile("fileupload", "someimg.png")
if err != nil {
t.Error(err)
}
// https://yourbasic.org/golang/create-image/
img := createImage()
// Encode() takes an io.Writer.
// We pass the multipart field
// 'fileupload' that we defined
// earlier which, in turn, writes
// to our io.Pipe
err = png.Encode(part, img)
if err != nil {
t.Error(err)
}
}()
// We read from the pipe which receives data
// from the multipart writer, which, in turn,
// receives data from png.Encode().
// We have 3 chained writers!
request := httptest.NewRequest("POST", "/", pr)
request.Header.Add("Content-Type", writer.FormDataContentType())
response := httptest.NewRecorder()
handler := UploadFileHandler()
handler.ServeHTTP(response, request)
t.Log("It should respond with an HTTP status code of 200")
if response.Code != 200 {
t.Errorf("Expected %s, received %d", 200, response.Code)
}
t.Log("It should create a file named 'someimg.png' in uploads folder")
if _, err := os.Stat("./uploads/someimg.png"); os.IsNotExist(err) {
t.Error("Expected file ./uploads/someimg.png' to exist")
}
}
This function makes use of the image package to generate a file dynamically taking advantage of the fact that you can pass an io.Writer to png.Encode. In the same vein, you could pass your multipart Writer to generate the bytes in a CSV format (NewWriter in package "encoding/csv"), generating a file on the fly, without needing to read anything from your filesystem.
If you have a look at the implementation of the FormFile function you'll see that it reads the exposed MultipartForm field.
https://golang.org/src/net/http/request.go?s=39022:39107#L1249
// FormFile returns the first file for the provided form key.
1258 // FormFile calls ParseMultipartForm and ParseForm if necessary.
1259 func (r *Request) FormFile(key string) (multipart.File, *multipart.FileHeader, error) {
1260 if r.MultipartForm == multipartByReader {
1261 return nil, nil, errors.New("http: multipart handled by MultipartReader")
1262 }
1263 if r.MultipartForm == nil {
1264 err := r.ParseMultipartForm(defaultMaxMemory)
1265 if err != nil {
1266 return nil, nil, err
1267 }
1268 }
1269 if r.MultipartForm != nil && r.MultipartForm.File != nil {
1270 if fhs := r.MultipartForm.File[key]; len(fhs) > 0 {
1271 f, err := fhs[0].Open()
1272 return f, fhs[0], err
1273 }
1274 }
1275 return nil, nil, ErrMissingFile
1276 }
In your test you should be able to create a test instance of multipart.Form and assign it to your request object - https://golang.org/pkg/mime/multipart/#Form
type Form struct {
Value map[string][]string
File map[string][]*FileHeader
}
Of course this will require that you use a real filepath which isn't great from a testing perspective. To get around this you could define an interface to read FormFile from a request object and pass a mock implementation into your EP struct.
Here is a good post with a few examples on how to do this: https://husobee.github.io/golang/testing/unit-test/2015/06/08/golang-unit-testing.html
I combined these and other answers into an Echo example without pipes or goroutines:
func Test_submitFile(t *testing.T) {
path := "testfile.txt"
body := new(bytes.Buffer)
writer := multipart.NewWriter(body)
part, err := writer.CreateFormFile("object", path)
assert.NoError(t, err)
sample, err := os.Open(path)
assert.NoError(t, err)
_, err = io.Copy(part, sample)
assert.NoError(t, err)
assert.NoError(t, writer.Close())
e := echo.New()
req := httptest.NewRequest(http.MethodPost, "/", body)
req.Header.Set(echo.HeaderContentType, writer.FormDataContentType())
rec := httptest.NewRecorder()
c := e.NewContext(req, rec)
c.SetPath("/submit")
if assert.NoError(t, submitFile(c)) {
assert.Equal(t, 200, rec.Code)
assert.Contains(t, rec.Body.String(), path)
fi, err := os.Stat(expectedPath)
if os.IsNotExist(err) {
t.Fatal("Upload file does not exist", expectedPath)
}
assert.Equal(t, wantSize, fi.Size())
}
}
By combining the previous answers, this worked for me:
filePath := "file.jpg"
fieldName := "file"
body := new(bytes.Buffer)
mw := multipart.NewWriter(body)
file, err := os.Open(filePath)
if err != nil {
t.Fatal(err)
}
w, err := mw.CreateFormFile(fieldName, filePath)
if err != nil {
t.Fatal(err)
}
if _, err := io.Copy(w, file); err != nil {
t.Fatal(err)
}
// close the writer before making the request
mw.Close()
req := httptest.NewRequest(http.MethodPost, "/upload", body)
req.Header.Add("Content-Type", mw.FormDataContentType())
res := httptest.NewRecorder()
// router is of type http.Handler
router.ServeHTTP(res, req)

Resources