ClamAv not detecting eicar signature in a zip file - go

I have a zip file (considerably large for ClamAV) that has EICAR file in it and for whatever reason, clam av is unable to detect it. When I unzip the file and pass the folder path, it is able to detect the EICAR signature. It is also able to detect eicar signatures on small zip files consistently but not so consistent with large files. I have also observed that ClamAV is not able to detect EICAR signatures on some golang and java lib compressed files but is able to detect them when compressed using the zip command line util.
Max file size and scan size are set to 0 to disable any limit.
Steps to reproduce: Please clone the repo here and compress using golang's archive/zip. Pass this on to ClamAV to find that the EICAR signature is not detected.
Here is what I have used to compress the file in golang.
package main
import (
"archive/zip"
"io"
"log"
"os"
"path/filepath"
)
func zipSource(source, target string) error {
// 1. Create a ZIP file and zip.Writer
f, err := os.Create(target)
if err != nil {
return err
}
defer f.Close()
writer := zip.NewWriter(f)
defer writer.Close()
// 2. Go through all the files of the source
return filepath.Walk(source, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
// 3. Create a local file header
header, err := zip.FileInfoHeader(info)
if err != nil {
return err
}
// set compression
header.Method = zip.Deflate
// 4. Set relative path of a file as the header name
header.Name, err = filepath.Rel(filepath.Dir(source), path)
if err != nil {
return err
}
if info.IsDir() {
header.Name += "/"
}
// 5. Create writer for the file header and save content of the file
headerWriter, err := writer.CreateHeader(header)
if err != nil {
return err
}
if info.IsDir() {
return nil
}
f, err := os.Open(path)
if err != nil {
return err
}
defer f.Close()
_, err = io.Copy(headerWriter, f)
return err
})
}
func main() {
if err := zipSource({sourcefolderLocation}, {targetZipFileName}); err != nil {
log.Fatal(err)
}
}
Any help in understanding this unpredictable behavior is highly appreciated.

Related

Golang: Facing error while creating .tar.gz file having large name

I am trying to create a .tar.gz file from folder that contains multiple files / folders. Once the .tar.gz file gets created, while extracting, the files are not not properly extracted. Mostly I think its because of large names or path exceeding some n characters, because same thing works when the filename/path is small. I referred this https://github.com/golang/go/issues/17630 and tried to add below code but it did not help.
header.Uid = 0
header.Gid = 0
I am using simple code seen below to create .tar.gz. The approach is, I create a temp folder, do some processing on the files and from that temp path, I create the .tar.gz file hence in the path below I am using pre-defined temp folder path.
package main
import (
"archive/tar"
"compress/gzip"
"fmt"
"io"
"log"
"os"
fp "path/filepath"
)
func main() {
// Create output file
out, err := os.Create("output.tar.gz")
if err != nil {
log.Fatalln("Error writing archive:", err)
}
defer out.Close()
// Create the archive and write the output to the "out" Writer
tmpDir := "C:/Users/USERNAME~1/AppData/Local/Temp/temp-241232063"
err = createArchive1(tmpDir, out)
if err != nil {
log.Fatalln("Error creating archive:", err)
}
fmt.Println("Archive created successfully")
}
func createArchive1(path string, targetFile *os.File) error {
gw := gzip.NewWriter(targetFile)
defer gw.Close()
tw := tar.NewWriter(gw)
defer tw.Close()
// walk through every file in the folder
err := fp.Walk(path, func(filePath string, info os.FileInfo, err error) error {
// ensure the src actually exists before trying to tar it
if _, err := os.Stat(filePath); err != nil {
return err
}
if err != nil {
return err
}
if info.IsDir() {
return nil
}
file, err := os.Open(filePath)
if err != nil {
return err
}
defer file.Close()
// generate tar header
header, err := tar.FileInfoHeader(info, info.Name())
header.Uid = 0
header.Gid = 0
if err != nil {
return err
}
header.Name = filePath //strings.TrimPrefix(filePath, fmt.Sprintf("%s/", fp.Dir(path))) //info.Name()
// write header
if err := tw.WriteHeader(header); err != nil {
return err
}
if _, err := io.Copy(tw, file); err != nil {
return err
}
return nil
})
return err
}
Please let me know what wrong I am doing.

Uploading for to internet site

With the below code I can download a file from internet asking with monitoring the downloaded percentage.
How can I do something to upload file to internet as well as monitoring the upload progress. I want to upload executable file at github assets
package main
import (
"fmt"
"io"
"net/http"
"os"
"strings"
"github.com/dustin/go-humanize"
)
// WriteCounter counts the number of bytes written to it. It implements to the io.Writer interface
// and we can pass this into io.TeeReader() which will report progress on each write cycle.
type WriteCounter struct {
Total uint64
}
func (wc *WriteCounter) Write(p []byte) (int, error) {
n := len(p)
wc.Total += uint64(n)
wc.PrintProgress()
return n, nil
}
func (wc WriteCounter) PrintProgress() {
// Clear the line by using a character return to go back to the start and remove
// the remaining characters by filling it with spaces
fmt.Printf("\r%s", strings.Repeat(" ", 35))
// Return again and print current status of download
// We use the humanize package to print the bytes in a meaningful way (e.g. 10 MB)
fmt.Printf("\rDownloading... %s complete", humanize.Bytes(wc.Total))
}
func main() {
fmt.Println("Download Started")
fileUrl := "https://upload.wikimedia.org/wikipedia/commons/d/d6/Wp-w4-big.jpg"
err := DownloadFile("avatar.jpg", fileUrl)
if err != nil {
panic(err)
}
fmt.Println("Download Finished")
}
// DownloadFile will download a url to a local file. It's efficient because it will
// write as it downloads and not load the whole file into memory. We pass an io.TeeReader
// into Copy() to report progress on the download.
func DownloadFile(filepath string, url string) error {
// Create the file, but give it a tmp file extension, this means we won't overwrite a
// file until it's downloaded, but we'll remove the tmp extension once downloaded.
out, err := os.Create(filepath + ".tmp")
if err != nil {
return err
}
// Get the data
resp, err := http.Get(url)
if err != nil {
out.Close()
return err
}
defer resp.Body.Close()
// Create our progress reporter and pass it to be used alongside our writer
counter := &WriteCounter{}
if _, err = io.Copy(out, io.TeeReader(resp.Body, counter)); err != nil {
out.Close()
return err
}
// The progress use the same line so print a new line once it's finished downloading
fmt.Print("\n")
// Close the file without defer so it can happen before Rename()
out.Close()
if err = os.Rename(filepath+".tmp", filepath); err != nil {
return err
}
return nil
}
I just modify your code. It works for my file server.
func UploadFile(filepath string, url string) error {
// Create the file, but give it a tmp file extension, this means we won't overwrite a
// file until it's downloaded, but we'll remove the tmp extension once downloaded.
out, err := os.Open(filepath)
if err != nil {
return err
}
// Create our progress reporter and pass it to be used alongside our writer
counter := &WriteCounter{}
// Get the data
resp, err := http.Post(url, "multipart/form-data", io.TeeReader(out, counter))
if err != nil {
out.Close()
log.Println(err.Error())
return err
}
defer resp.Body.Close()
// The progress use the same line so print a new line once it's finished downloading
fmt.Print("\n")
// Close the file without defer so it can happen before Rename()
out.Close()
return nil
}

Editing a zip file in memory

I am trying to edit a zip file in memory in Go and return the zipped file through a HTTP response
The goal is to add a few files to a path in the zip file example
I add a log.txt file in my path/to/file route in the zipped folder
All this should be done without saving the file or editing the original file.
I have implemented a simple version of real-time stream compression, which can correctly compress a single file. If you want it to run efficiently, you need a lot of optimization.
This is only for reference. If you need more information, you should set more useful HTTP header information before compression so that the client can correctly process the response data.
package main
import (
"archive/zip"
"io"
"net/http"
"os"
"github.com/gin-gonic/gin"
)
func main() {
engine := gin.Default()
engine.GET("/log.zip", func(c *gin.Context) {
f, err := os.Open("./log.txt")
if err != nil {
c.String(http.StatusInternalServerError, err.Error())
return
}
defer f.Close()
info, err := f.Stat()
if err != nil {
c.String(http.StatusInternalServerError, err.Error())
return
}
z := zip.NewWriter(c.Writer)
head, err := zip.FileInfoHeader(info)
if err != nil {
c.String(http.StatusInternalServerError, err.Error())
return
}
defer z.Close()
w, err := z.CreateHeader(head)
if err != nil {
c.String(http.StatusInternalServerError, err.Error())
return
}
_, err = io.Copy(w, f)
if err != nil {
c.String(http.StatusInternalServerError, err.Error())
return
}
})
engine.Run("127.0.0.1:8080")
}
So after hours of tireless work i figured out my approach was bad or maybe not possible with the level of my knowledge so here is a not so optimal solution but it works and fill ur file is not large it should be okay for you.
So you have a file template.zip and u want to add extra files, my initial approach was to copy the whole file into memory and edit it from their but i was having complications.
My next approach was to recreate the file in memory, file by file and to do that i need to know every file in the directory i used the code below to get all my files into a list
root := "template"
err = filepath.Walk(root, func(path string, info os.FileInfo, err error) error {
if info.IsDir() {
return nil
}append(files,path)}
now i have all my files and i can create a buffer to hold all this files
buf := new(bytes.Buffer)
// Create a new zip archive.
zipWriter := zip.NewWriter(buf)
now with the zip archive i can write all my old files to it while at the same time copying the contents
for _, file := range files {
zipFile, err := zipWriter.Create(file)
if err != nil {
fmt.Println(err)
}
content, err := ioutil.ReadFile(file)
if err != nil {
log.Fatal(err)
}
// Convert []byte to string and print to screen
// text := string(content)
_, err = zipFile.Write(content)
if err != nil {
fmt.Println(err)
}
}
At this point, we have our file in buf.bytes()
The remaining cold adds the new files and sends the response back to the client
for _, appCode := range appPageCodeText {
f, err := zipWriter.Create(filepath.fileextension)
if err != nil {
log.Fatal(err)
}
_, err = f.Write([]byte(appCode.Content))
}
err = zipWriter.Close()
if err != nil {
fmt.Println(err)
}
w.Header().Set("Content-Disposition", "attachment; filename="+"template.zip")
w.Header().Set("Content-Type", "application/zip")
w.Write(buf.Bytes()) //'Copy' the file to the client

How to compress a file to .zip without directory folder in Go

There're examples about compressing a file to .zip in Go. However, the file they generate include the directory folder. When I decompress the .zip file, there will be a new folder.
So, how can I compress a file to .zip without getting the directory folder included?
An example:
https://golangcode.com/create-zip-files-in-go/
package main
import (
"archive/zip"
"fmt"
"io"
"os"
)
func main() {
// List of Files to Zip
files := []string{"example.csv", "data.csv"}
output := "done.zip"
if err := ZipFiles(output, files); err != nil {
panic(err)
}
fmt.Println("Zipped File:", output)
}
// ZipFiles compresses one or many files into a single zip archive file.
// Param 1: filename is the output zip file's name.
// Param 2: files is a list of files to add to the zip.
func ZipFiles(filename string, files []string) error {
newZipFile, err := os.Create(filename)
if err != nil {
return err
}
defer newZipFile.Close()
zipWriter := zip.NewWriter(newZipFile)
defer zipWriter.Close()
// Add files to zip
for _, file := range files {
if err = AddFileToZip(zipWriter, file); err != nil {
return err
}
}
return nil
}
func AddFileToZip(zipWriter *zip.Writer, filename string) error {
fileToZip, err := os.Open(filename)
if err != nil {
return err
}
defer fileToZip.Close()
// Get the file information
info, err := fileToZip.Stat()
if err != nil {
return err
}
header, err := zip.FileInfoHeader(info)
if err != nil {
return err
}
// Using FileInfoHeader() above only uses the basename of the file. If we want
// to preserve the folder structure we can overwrite this with the full path.
header.Name = filename
// Change to deflate to gain better compression
// see http://golang.org/pkg/archive/zip/#pkg-constants
header.Method = zip.Deflate
writer, err := zipWriter.CreateHeader(header)
if err != nil {
return err
}
_, err = io.Copy(writer, fileToZip)
return err
}
Just use a base name of the file in the zip header.
header.Name = filepath.Base(filename)
^^^^^^^^^^^^^^
Here is a version that does the same thing
package main
import (
"archive/zip"
"io"
"log"
"os"
"path/filepath"
)
func createFlatZip(w io.Writer, files ...string) error {
z := zip.NewWriter(w)
for _, file := range files {
src, err := os.Open(file)
if err != nil {
return err
}
info, err := src.Stat()
if err != nil {
return err
}
hdr, err := zip.FileInfoHeader(info)
if err != nil {
return err
}
hdr.Name = filepath.Base(file) // Write only the base name in the header
dst, err := z.CreateHeader(hdr)
if err != nil {
return err
}
_, err = io.Copy(dst, src)
if err != nil {
return err
}
src.Close()
}
return z.Close()
}
func main() {
if len(os.Args) < 3 {
log.Fatalf("archive name and at least one file are required")
}
a, err := os.Create(os.Args[1])
if err != nil {
log.Fatal(err)
}
defer a.Close()
err = createFlatZip(a, os.Args[2:]...)
if err != nil {
log.Fatal(err)
}
}
Results:
~/src/gozip
➜ go build
~/src/gozip
➜ mkdir test && echo 1 > test/1.txt # create a test file in a subfolder
~/src/gozip
➜ ./gozip 1.zip test/1.txt
~/src/gozip
➜ unzip -l 1.zip
Archive: 1.zip
Length Date Time Name
--------- ---------- ----- ----
2 08-15-2019 01:29 1.txt
--------- -------
2 1 file

How do I zip a directory containing sub directories or files in Golang?

I know it will have to do with the zip package I just have no idea how I would implement such a thing.
Here's solution that uses Go's built-in recursive file-walker since the top answer so far has implemented their own file-walker:
Aside, some findings I've found while generating zip files today that might save someone else a headache:
The argument to w.Create(zippath) should not start with a "/", and should be relative to the zip root (aka the folder that would be created if you unzipped the archive). So a top-level "manifest.xml" file should be w.Create("manifest.xml"). Nested files should be w.Create("a/b/c.css). If you are generating bad/surprising archives, check first to ensure you aren't breaking this rule. My code does not try to enforce this.
Some specs (like epub) want files in a certain order, but Go's filepath.Walk will crawl in lexical order. (For that matter, I've found all epub parsers to be lenient here so far from Calibre to Books.app on macOS). If you need a specific order, then #LeTigre's solution with ReadDir will let you sort the files at each level of descent.
package main
import (
"archive/zip"
"fmt"
"io"
"os"
"path/filepath"
)
// Zips "./input" into "./output.zip"
func main() {
file, err := os.Create("output.zip")
if err != nil {
panic(err)
}
defer file.Close()
w := zip.NewWriter(file)
defer w.Close()
walker := func(path string, info os.FileInfo, err error) error {
fmt.Printf("Crawling: %#v\n", path)
if err != nil {
return err
}
if info.IsDir() {
return nil
}
file, err := os.Open(path)
if err != nil {
return err
}
defer file.Close()
// Ensure that `path` is not absolute; it should not start with "/".
// This snippet happens to work because I don't use
// absolute paths, but ensure your real-world code
// transforms path into a zip-root relative path.
f, err := w.Create(path)
if err != nil {
return err
}
_, err = io.Copy(f, file)
if err != nil {
return err
}
return nil
}
err = filepath.Walk("input", walker)
if err != nil {
panic(err)
}
}
To do it manually, you could modify the code linked above:
ExampleZipWriter
To give you a simple example, that has many flaws, but might be easily understood:
func ZipWriter() {
baseFolder := "/Users/tom/Desktop/testing/"
// Get a Buffer to Write To
outFile, err := os.Create(`/Users/tom/Desktop/zip.zip`)
if err != nil {
fmt.Println(err)
}
defer outFile.Close()
// Create a new zip archive.
w := zip.NewWriter(outFile)
// Add some files to the archive.
addFiles(w, baseFolder, "")
if err != nil {
fmt.Println(err)
}
// Make sure to check the error on Close.
err = w.Close()
if err != nil {
fmt.Println(err)
}
}
We use this to iterate on files recursively to generate folders too:
func addFiles(w *zip.Writer, basePath, baseInZip string) {
// Open the Directory
files, err := ioutil.ReadDir(basePath)
if err != nil {
fmt.Println(err)
}
for _, file := range files {
fmt.Println(basePath + file.Name())
if !file.IsDir() {
dat, err := ioutil.ReadFile(basePath + file.Name())
if err != nil {
fmt.Println(err)
}
// Add some files to the archive.
f, err := w.Create(baseInZip + file.Name())
if err != nil {
fmt.Println(err)
}
_, err = f.Write(dat)
if err != nil {
fmt.Println(err)
}
} else if file.IsDir() {
// Recurse
newBase := basePath + file.Name() + "/"
fmt.Println("Recursing and Adding SubDir: " + file.Name())
fmt.Println("Recursing and Adding SubDir: " + newBase)
addFiles(w, newBase, baseInZip + file.Name() + "/")
}
}
}
#danneu's answer is good. But if the directory contains empty subdirectories, the code doesn't work and it will ignore them.
So instead of just returning nil, we should create a directory:
if info.IsDir() {
// add a trailing slash for creating dir
path = fmt.Sprintf("%s%c", path, os.PathSeparator)
_, err = w.Create(path)
return err
}
an ez way to zip directory, you can run command of server:
out, err := exec.Command("zip", "-r", "-D", "ideaz.zip", ".idea/*").Output()
https://play.golang.org/p/Mn-HmUjm5q2

Resources