Handle Http Upload Zip file in Golang - go

I'm using golang net/http package to retrieve the uploaded zip file via postman.
The attachment file link. It is not dangerous file. Feel free to check out.
Development env
local machine m1 macbook pro golang 1.17.2 - no issue
server docker image golang:1.17.5-stretch - got issue.
Code to capture the post form transSourceFile file.
func HandleFileReqTest(w http.ResponseWriter, req *http.Request, params map[string]string) err {
if err := req.ParseMultipartForm(32 << 20); err != nil {
return err
}
file, header, err := req.FormFile("transSourceFile")
if err != nil {
return err
}
defer file.Close()
fmt.Println("header.Size:", header.Size)
return nil
}
I tried below code also no use
func HandleFileReqTest(w http.ResponseWriter, req *http.Request, params map[string]string) err {
if err := req.ParseForm(); err != nil {
return err
}
req.ParseMultipartForm(32 << 20)
file, header, err := req.FormFile("transSourceFile")
if err != nil {
return err
}
defer file.Close()
fmt.Println("header.Size:", header.Size)
return nil
}
Result:
Local machine got the same file size as the origin file.
Server with golang:1.17.5-stretch got the different file size compare to origin file.
As the result on this, i'm unable to unzip the file in the server. Anyone can help?

You need to copy form file to the actual file:
f, err := os.Create("some.zip")
defer f.Close()
n, err := io.Copy(f, file)

Data isn't being flushed to the file completely. You should close the file first to ensure that the data is fully flushed.
// create a local file filename
dst, err := os.Create("filename.zip")
// save it
fl, err = io.Copy(dst, src)
// Close the file
dst.Close()
stat, _ := dst.Stat()
//Now check the size stat.Size() or header.Size after flushing the file.

Related

ClamAv not detecting eicar signature in a zip file

I have a zip file (considerably large for ClamAV) that has EICAR file in it and for whatever reason, clam av is unable to detect it. When I unzip the file and pass the folder path, it is able to detect the EICAR signature. It is also able to detect eicar signatures on small zip files consistently but not so consistent with large files. I have also observed that ClamAV is not able to detect EICAR signatures on some golang and java lib compressed files but is able to detect them when compressed using the zip command line util.
Max file size and scan size are set to 0 to disable any limit.
Steps to reproduce: Please clone the repo here and compress using golang's archive/zip. Pass this on to ClamAV to find that the EICAR signature is not detected.
Here is what I have used to compress the file in golang.
package main
import (
"archive/zip"
"io"
"log"
"os"
"path/filepath"
)
func zipSource(source, target string) error {
// 1. Create a ZIP file and zip.Writer
f, err := os.Create(target)
if err != nil {
return err
}
defer f.Close()
writer := zip.NewWriter(f)
defer writer.Close()
// 2. Go through all the files of the source
return filepath.Walk(source, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
// 3. Create a local file header
header, err := zip.FileInfoHeader(info)
if err != nil {
return err
}
// set compression
header.Method = zip.Deflate
// 4. Set relative path of a file as the header name
header.Name, err = filepath.Rel(filepath.Dir(source), path)
if err != nil {
return err
}
if info.IsDir() {
header.Name += "/"
}
// 5. Create writer for the file header and save content of the file
headerWriter, err := writer.CreateHeader(header)
if err != nil {
return err
}
if info.IsDir() {
return nil
}
f, err := os.Open(path)
if err != nil {
return err
}
defer f.Close()
_, err = io.Copy(headerWriter, f)
return err
})
}
func main() {
if err := zipSource({sourcefolderLocation}, {targetZipFileName}); err != nil {
log.Fatal(err)
}
}
Any help in understanding this unpredictable behavior is highly appreciated.

File has no content after downloading xlsx file using http/net

I tried downloading an Excel file from a URL using http/net by calling the GET method. I don't know if this is releveant, but as for my case, I use dropbox to store the file on the cloud (it's open for public, not restricted, it can be accessed on incognito).
But when I open the file that's saved on the local, it has no content at all. It has just an empty sheet. The file is supposed to have filled with lots of data in cell.
What's happening here? Anyone knows how to solve it? There's no error at all when I print it.
func main() {
filePath := "./file/filename.xlsx"
url := "http://www.dropbox.com/somethingsomething.xlsx"
out, err := os.Create(filePath)
if err != nil {
fmt.Println(err)
}
defer out.Close()
resp, err := http.Get(url)
if err != nil {
fmt.Println(err)
}
defer resp.Body.Close()
_, err = io.Copy(out, resp.Body)
if err != nil {
fmt.Println(err)
}
return
}
Does the dropbox URL have dl=0 query param?
If so, try changing it to dl=1 to force download the file.
I tried the same with one of my files and it worked.
Thanks!

How to add images into specific folder in Go? Getting error like: `%!(EXTRA *fs.PathError=open /photos: read-only file system)`

I am trying to put images into a specific folder in Golang. Here is the code below.
This is the function where I create a folder called photos in the root directory.
func createPhotoFolder(folderName string) {
err := os.Mkdir(folderName, 777)
if err != nil {
fmt.Println("Error creating folder: ", err)
return
}
fmt.Println(folderName, " created successfully in the root directory")
}
This is the function where I make get request to fetch image and try to put them into a photos folder I created earlier.
func downloadImages(urls []string) {
for i, url := range urls {
resp, err := http.Get(url)
fmt.Printf("%d inside for loop\n", i)
if err != nil {
log.Fatal("error fetching image: ", err)
}
defer resp.Body.Close()
out, err := os.Create("photos")
if err != nil {
log.Printf("Can't put image into folder: ", err)
}
defer out.Close()
}
}
This is the error I get when I run the program.
1- If the folder name is written in this way os.Create("photos") without forwardslash I get the error message as below.
Can't put image into folder: %!(EXTRA *fs.PathError=open photos: is a directory)
2- If I write it like os.Create("/photos"). I get the error as below.
Can't put image into folder: %!(EXTRA *fs.PathError=open /photos: read-only file system)
I gave all the permission while creating the photos folder in the way of chmod.
I did try using io.Copy() but it requires a file parameter which I don't get while creating one using os.Create()
How should I create the folder and put the images inside it properly?
Here, in your code, in os.Create, it should have the complete address of the file to be created along with the name of the file to be created. Like:
gopath := "C:/Users/<username>/go/src/photos/" //where photos is the folder you created
filename := "photo1.jpg"
out, err := os.Create(gopath + filename)
Also, as #steven-penny gave in his answer, create a filename from the image name directly from the url. So that you don't have to give the filename for each image you download.
out, err := os.create(gopath + filepath.Base(link))
And save the image to your system with,
out.Readfrom(resp.Body)
Here is a small program that does what I think you are trying to do:
package main
import (
"net/http"
"os"
"path/filepath"
)
func downloadImages(links []string) error {
tmp := os.TempDir()
for _, link := range links {
println(link)
res, err := http.Get(link)
if err != nil { return err }
defer res.Body.Close()
file, err := os.Create(filepath.Join(tmp, filepath.Base(link)))
if err != nil { return err }
defer file.Close()
file.ReadFrom(res.Body)
}
return nil
}
func main() {
links := []string{
"http://cdn.sstatic.net/Sites/stackoverflow/Img/apple-touch-icon.png",
"http://cdn.sstatic.net/Sites/stackoverflow/Img/favicon.ico",
}
err := downloadImages(links)
if err != nil {
panic(err)
}
}
You'll want to modify it, as you were using a different directory, but it should get you started.
https://golang.org/pkg/os#File.ReadFrom

Editing a zip file in memory

I am trying to edit a zip file in memory in Go and return the zipped file through a HTTP response
The goal is to add a few files to a path in the zip file example
I add a log.txt file in my path/to/file route in the zipped folder
All this should be done without saving the file or editing the original file.
I have implemented a simple version of real-time stream compression, which can correctly compress a single file. If you want it to run efficiently, you need a lot of optimization.
This is only for reference. If you need more information, you should set more useful HTTP header information before compression so that the client can correctly process the response data.
package main
import (
"archive/zip"
"io"
"net/http"
"os"
"github.com/gin-gonic/gin"
)
func main() {
engine := gin.Default()
engine.GET("/log.zip", func(c *gin.Context) {
f, err := os.Open("./log.txt")
if err != nil {
c.String(http.StatusInternalServerError, err.Error())
return
}
defer f.Close()
info, err := f.Stat()
if err != nil {
c.String(http.StatusInternalServerError, err.Error())
return
}
z := zip.NewWriter(c.Writer)
head, err := zip.FileInfoHeader(info)
if err != nil {
c.String(http.StatusInternalServerError, err.Error())
return
}
defer z.Close()
w, err := z.CreateHeader(head)
if err != nil {
c.String(http.StatusInternalServerError, err.Error())
return
}
_, err = io.Copy(w, f)
if err != nil {
c.String(http.StatusInternalServerError, err.Error())
return
}
})
engine.Run("127.0.0.1:8080")
}
So after hours of tireless work i figured out my approach was bad or maybe not possible with the level of my knowledge so here is a not so optimal solution but it works and fill ur file is not large it should be okay for you.
So you have a file template.zip and u want to add extra files, my initial approach was to copy the whole file into memory and edit it from their but i was having complications.
My next approach was to recreate the file in memory, file by file and to do that i need to know every file in the directory i used the code below to get all my files into a list
root := "template"
err = filepath.Walk(root, func(path string, info os.FileInfo, err error) error {
if info.IsDir() {
return nil
}append(files,path)}
now i have all my files and i can create a buffer to hold all this files
buf := new(bytes.Buffer)
// Create a new zip archive.
zipWriter := zip.NewWriter(buf)
now with the zip archive i can write all my old files to it while at the same time copying the contents
for _, file := range files {
zipFile, err := zipWriter.Create(file)
if err != nil {
fmt.Println(err)
}
content, err := ioutil.ReadFile(file)
if err != nil {
log.Fatal(err)
}
// Convert []byte to string and print to screen
// text := string(content)
_, err = zipFile.Write(content)
if err != nil {
fmt.Println(err)
}
}
At this point, we have our file in buf.bytes()
The remaining cold adds the new files and sends the response back to the client
for _, appCode := range appPageCodeText {
f, err := zipWriter.Create(filepath.fileextension)
if err != nil {
log.Fatal(err)
}
_, err = f.Write([]byte(appCode.Content))
}
err = zipWriter.Close()
if err != nil {
fmt.Println(err)
}
w.Header().Set("Content-Disposition", "attachment; filename="+"template.zip")
w.Header().Set("Content-Type", "application/zip")
w.Write(buf.Bytes()) //'Copy' the file to the client

Issue when writing file to disk after downloading it by FTP

The file that is written to disk is empty, but the reader is not.
I do not understand where the issue is.
I tried to play with a Buffer and then String() method and I can confirm that the content is fine, but using the Read() method of this library is not working.
The library I use is github.com/jlaffaye/ftp
// pullFileByFTP
func pullFileByFTP(fileID, server string, port int64, username, password, path, file string) error {
// Connect to the server
client, err := ftp.Dial(fmt.Sprintf("%s:%d", server, port))
if err != nil {
return err
}
// Log in the server
err = client.Login(username, password)
if err != nil {
return err
}
// Retrieve the file
reader, err := client.Retr(fmt.Sprintf("%s%s", path, file))
if err != nil {
return err
}
// Read the file
var srcFile []byte
_, err = reader.Read(srcFile)
if err != nil {
return err
}
// Create the destination file
dstFile, err := os.Create(fmt.Sprintf("%s/%s", shared.TmpDir, fileID))
if err != nil {
return fmt.Errorf("Error while creating the destination file : %s", err)
}
defer dstFile.Close()
// Copy the file
dstFile.Write(srcFile)
return nil
}
You are using Read and Write wrong:
var srcFile []byte
_, err = reader.Read(srcFile)
Read puts the read bytes into its argument. Since srcFile is a nil slice, this instructs the reader to read zero bytes. Use ioutil.ReadAll to read all bytes.
Next up is your use of Write. Write(b) writes up to len(b) bytes, but not necessarily all of it. You must check the return values and call Write repeatedly if necessary.
However, in your case you just want to connect an io.Reader (*Response implements io.Reader) and io.Writer (*os.File). That's what io.Copy is for:
reader, err := client.Retr(path + file)
dstFile, err := ioutil.TempFile("", fileID)
_, err := io.Copy(dstFile, reader)
err := dstFile.Close()

Resources