I'm attempting to create a compressed tar archive using the Go standard library, specifically compress/gzip and archive/tar. I can successfully create a tar archive, but when I try to compress said archive, the resulting tarball can't be decompressed. On OSX, I get "Error 1 - Operation Not Permitted"
To run this code, you'll need a file named foo.txt in the same directory.
package main
import (
"archive/tar"
"bytes"
"compress/gzip"
"io/ioutil"
"log"
"os"
)
func main() {
var b bytes.Buffer
// Create a new zip archive.
w := tar.NewWriter(gzip.NewWriter(&b))
fi, err := os.Stat("foo.txt")
if err != nil {
log.Fatal(err)
}
header, err := tar.FileInfoHeader(fi, "")
if err != nil {
log.Fatal(err)
}
err = w.WriteHeader(header)
if err != nil {
log.Fatal(err)
}
contents, err := ioutil.ReadFile("foo.txt")
if err != nil {
log.Fatal(err)
}
_, err = w.Write(contents)
if err != nil {
log.Fatal(err)
}
err = w.Close()
// Make sure to check the error on Close.
err = ioutil.WriteFile("foo.tar.gz", b.Bytes(), 0666)
if err != nil {
log.Fatal(err)
}
}
You need to close the underlying gzip writer so that it you are guaranteed all bytes are flushed out to the file. Like so:
// gzip writer
gz := gzip.NewWriter(f)
// Create a new tar archive.
w := tar.NewWriter(gz)
// add things to the tar archive
// ...
// make sure the gzip writer flushes any pending bytes
if err = gz.Close(); err != nil {
log.Fatal(err)
}
Related
I have a zip file (considerably large for ClamAV) that has EICAR file in it and for whatever reason, clam av is unable to detect it. When I unzip the file and pass the folder path, it is able to detect the EICAR signature. It is also able to detect eicar signatures on small zip files consistently but not so consistent with large files. I have also observed that ClamAV is not able to detect EICAR signatures on some golang and java lib compressed files but is able to detect them when compressed using the zip command line util.
Max file size and scan size are set to 0 to disable any limit.
Steps to reproduce: Please clone the repo here and compress using golang's archive/zip. Pass this on to ClamAV to find that the EICAR signature is not detected.
Here is what I have used to compress the file in golang.
package main
import (
"archive/zip"
"io"
"log"
"os"
"path/filepath"
)
func zipSource(source, target string) error {
// 1. Create a ZIP file and zip.Writer
f, err := os.Create(target)
if err != nil {
return err
}
defer f.Close()
writer := zip.NewWriter(f)
defer writer.Close()
// 2. Go through all the files of the source
return filepath.Walk(source, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
// 3. Create a local file header
header, err := zip.FileInfoHeader(info)
if err != nil {
return err
}
// set compression
header.Method = zip.Deflate
// 4. Set relative path of a file as the header name
header.Name, err = filepath.Rel(filepath.Dir(source), path)
if err != nil {
return err
}
if info.IsDir() {
header.Name += "/"
}
// 5. Create writer for the file header and save content of the file
headerWriter, err := writer.CreateHeader(header)
if err != nil {
return err
}
if info.IsDir() {
return nil
}
f, err := os.Open(path)
if err != nil {
return err
}
defer f.Close()
_, err = io.Copy(headerWriter, f)
return err
})
}
func main() {
if err := zipSource({sourcefolderLocation}, {targetZipFileName}); err != nil {
log.Fatal(err)
}
}
Any help in understanding this unpredictable behavior is highly appreciated.
I am trying to create a .tar.gz file from folder that contains multiple files / folders. Once the .tar.gz file gets created, while extracting, the files are not not properly extracted. Mostly I think its because of large names or path exceeding some n characters, because same thing works when the filename/path is small. I referred this https://github.com/golang/go/issues/17630 and tried to add below code but it did not help.
header.Uid = 0
header.Gid = 0
I am using simple code seen below to create .tar.gz. The approach is, I create a temp folder, do some processing on the files and from that temp path, I create the .tar.gz file hence in the path below I am using pre-defined temp folder path.
package main
import (
"archive/tar"
"compress/gzip"
"fmt"
"io"
"log"
"os"
fp "path/filepath"
)
func main() {
// Create output file
out, err := os.Create("output.tar.gz")
if err != nil {
log.Fatalln("Error writing archive:", err)
}
defer out.Close()
// Create the archive and write the output to the "out" Writer
tmpDir := "C:/Users/USERNAME~1/AppData/Local/Temp/temp-241232063"
err = createArchive1(tmpDir, out)
if err != nil {
log.Fatalln("Error creating archive:", err)
}
fmt.Println("Archive created successfully")
}
func createArchive1(path string, targetFile *os.File) error {
gw := gzip.NewWriter(targetFile)
defer gw.Close()
tw := tar.NewWriter(gw)
defer tw.Close()
// walk through every file in the folder
err := fp.Walk(path, func(filePath string, info os.FileInfo, err error) error {
// ensure the src actually exists before trying to tar it
if _, err := os.Stat(filePath); err != nil {
return err
}
if err != nil {
return err
}
if info.IsDir() {
return nil
}
file, err := os.Open(filePath)
if err != nil {
return err
}
defer file.Close()
// generate tar header
header, err := tar.FileInfoHeader(info, info.Name())
header.Uid = 0
header.Gid = 0
if err != nil {
return err
}
header.Name = filePath //strings.TrimPrefix(filePath, fmt.Sprintf("%s/", fp.Dir(path))) //info.Name()
// write header
if err := tw.WriteHeader(header); err != nil {
return err
}
if _, err := io.Copy(tw, file); err != nil {
return err
}
return nil
})
return err
}
Please let me know what wrong I am doing.
There're examples about compressing a file to .zip in Go. However, the file they generate include the directory folder. When I decompress the .zip file, there will be a new folder.
So, how can I compress a file to .zip without getting the directory folder included?
An example:
https://golangcode.com/create-zip-files-in-go/
package main
import (
"archive/zip"
"fmt"
"io"
"os"
)
func main() {
// List of Files to Zip
files := []string{"example.csv", "data.csv"}
output := "done.zip"
if err := ZipFiles(output, files); err != nil {
panic(err)
}
fmt.Println("Zipped File:", output)
}
// ZipFiles compresses one or many files into a single zip archive file.
// Param 1: filename is the output zip file's name.
// Param 2: files is a list of files to add to the zip.
func ZipFiles(filename string, files []string) error {
newZipFile, err := os.Create(filename)
if err != nil {
return err
}
defer newZipFile.Close()
zipWriter := zip.NewWriter(newZipFile)
defer zipWriter.Close()
// Add files to zip
for _, file := range files {
if err = AddFileToZip(zipWriter, file); err != nil {
return err
}
}
return nil
}
func AddFileToZip(zipWriter *zip.Writer, filename string) error {
fileToZip, err := os.Open(filename)
if err != nil {
return err
}
defer fileToZip.Close()
// Get the file information
info, err := fileToZip.Stat()
if err != nil {
return err
}
header, err := zip.FileInfoHeader(info)
if err != nil {
return err
}
// Using FileInfoHeader() above only uses the basename of the file. If we want
// to preserve the folder structure we can overwrite this with the full path.
header.Name = filename
// Change to deflate to gain better compression
// see http://golang.org/pkg/archive/zip/#pkg-constants
header.Method = zip.Deflate
writer, err := zipWriter.CreateHeader(header)
if err != nil {
return err
}
_, err = io.Copy(writer, fileToZip)
return err
}
Just use a base name of the file in the zip header.
header.Name = filepath.Base(filename)
^^^^^^^^^^^^^^
Here is a version that does the same thing
package main
import (
"archive/zip"
"io"
"log"
"os"
"path/filepath"
)
func createFlatZip(w io.Writer, files ...string) error {
z := zip.NewWriter(w)
for _, file := range files {
src, err := os.Open(file)
if err != nil {
return err
}
info, err := src.Stat()
if err != nil {
return err
}
hdr, err := zip.FileInfoHeader(info)
if err != nil {
return err
}
hdr.Name = filepath.Base(file) // Write only the base name in the header
dst, err := z.CreateHeader(hdr)
if err != nil {
return err
}
_, err = io.Copy(dst, src)
if err != nil {
return err
}
src.Close()
}
return z.Close()
}
func main() {
if len(os.Args) < 3 {
log.Fatalf("archive name and at least one file are required")
}
a, err := os.Create(os.Args[1])
if err != nil {
log.Fatal(err)
}
defer a.Close()
err = createFlatZip(a, os.Args[2:]...)
if err != nil {
log.Fatal(err)
}
}
Results:
~/src/gozip
➜ go build
~/src/gozip
➜ mkdir test && echo 1 > test/1.txt # create a test file in a subfolder
~/src/gozip
➜ ./gozip 1.zip test/1.txt
~/src/gozip
➜ unzip -l 1.zip
Archive: 1.zip
Length Date Time Name
--------- ---------- ----- ----
2 08-15-2019 01:29 1.txt
--------- -------
2 1 file
I've written the following code to tar a file, code works but strangely if I untar the archive the file permissions are gone so I can't read it unless I then chmod the file:
package main
import (
"archive/tar"
"io/ioutil"
"log"
"os"
)
func main() {
c, err := os.Create("/path/to/tar/file/test.tar")
if err != nil {
log.Fatalln(err)
}
tw := tar.NewWriter(c)
f, err := os.Open("sample.txt")
if err != nil {
log.Fatalln(err)
}
fi, err := f.Stat()
if err != nil {
log.Fatalln(err)
}
hdr := &tar.Header{Name: f.Name(),
Size: fi.Size(),
}
if err := tw.WriteHeader(hdr); err != nil {
log.Fatalln(err)
}
r, err := ioutil.ReadFile("sample.txt")
if err != nil {
log.Fatalln(err)
}
if _, err := tw.Write(r); err != nil {
log.Fatalln(err)
}
if err := tw.Close(); err != nil {
log.Fatalln(err)
}
}
Any idea what I'm doing wrong?
You're not preserving the original permissions of the file. You're manually creating a header, and specifying only the name and size. Instead, use tar.FileInfoHeader to build the header.
package main
import (
"archive/tar"
"io/ioutil"
"log"
"os"
)
func main() {
c, err := os.Create("/path/to/tar/file/test.tar")
if err != nil {
log.Fatalln(err)
}
tw := tar.NewWriter(c)
f, err := os.Open("sample.txt")
if err != nil {
log.Fatalln(err)
}
fi, err := f.Stat()
if err != nil {
log.Fatalln(err)
}
// create header from FileInfo
hdr, err := tar.FileInfoHeader(fi, "")
if err != nil {
log.Fatalln(err)
}
if err := tw.WriteHeader(hdr); err != nil {
log.Fatalln(err)
}
// instead of reading the whole file into memory, prefer io.Copy
r, err := io.Copy(tw, f)
if err != nil {
log.Fatalln(err)
}
log.Printf("Wrote %d bytes\n", r)
}
Also note that I used io.Copy to copy data from the file (an io.Reader) to the tar writer (an io.Writer). This will work much better for larger files.
Also - pay special attention to this note from the docs:
Because os.FileInfo's Name method returns only the base name of the file it describes, it may be necessary to modify the Name field of the returned header to provide the full path name of the file.
In this simple example, you're just using sample.txt so you shouldn't run into trouble. If you wanted to preserve a directory structure in your tar, you may have to modify the Name field in the header.
Is it possible to extract a tar.xz package in golang? My understanding is it's possible to use the library for tar and sending it to an xz go library.
I recently created an XZ decompression package so it is now
possible to extract a tar.xz using only Go code.
The following code extracts the file myfile.tar.xz to the current
directory:
package main
import (
"archive/tar"
"fmt"
"io"
"log"
"os"
"github.com/xi2/xz"
)
func main() {
// Open a file
f, err := os.Open("myfile.tar.xz")
if err != nil {
log.Fatal(err)
}
// Create an xz Reader
r, err := xz.NewReader(f, 0)
if err != nil {
log.Fatal(err)
}
// Create a tar Reader
tr := tar.NewReader(r)
// Iterate through the files in the archive.
for {
hdr, err := tr.Next()
if err == io.EOF {
// end of tar archive
break
}
if err != nil {
log.Fatal(err)
}
switch hdr.Typeflag {
case tar.TypeDir:
// create a directory
fmt.Println("creating: " + hdr.Name)
err = os.MkdirAll(hdr.Name, 0777)
if err != nil {
log.Fatal(err)
}
case tar.TypeReg, tar.TypeRegA:
// write a file
fmt.Println("extracting: " + hdr.Name)
w, err := os.Create(hdr.Name)
if err != nil {
log.Fatal(err)
}
_, err = io.Copy(w, tr)
if err != nil {
log.Fatal(err)
}
w.Close()
}
}
f.Close()
}
http://golang.org/pkg/archive/tar/#example_
also you can do
import "os/exec"
cmd := exec.Command("tar", "-x", "/your/archive.tar.xz")
err := cmd.Run()
There is no Lempel-Ziv-Markow encoder or decoder in the Go standard library. If you are allowed to assume that the platform your code runs on provides the xz utility, you could use stub functions like these:
import "os/exec"
// decompress xz compressed data stream r.
func UnxzReader(r io.Reader) (io.ReadCloser, error) {
unxz := exec.Command("xz", "-d")
unxz.Stdin = r
out, err := unxz.StdoutPipe()
if err != nil {
return nil, err
}
err = unxz.Start()
if err != nil {
return nil, err
}
// we are not interested in the exit status, but we should really collect
// that zombie process
go unxz.Wait()
return out, nil
}