I have the following function I'v tried to simplify to just adding a single file to a .zip archive.
Whatever I try, the resulting .zip file has no files listed in it. The size of the archive is correct. But when I try to extract all (windows), the archive is empty.
go version go1.10.1 windows/amd64
func Zip(src string, dst string) error {
destinationFile, err := os.Create(dst)
if err != nil {
return err
}
myZip := zip.NewWriter(destinationFile)
file := `C:\MA\testing\cldeploy-local.json`
zipFile, err := myZip.Create(file)
fsFile, err := os.Open(file)
if err != nil {
return err
}
_, err = io.Copy(zipFile, fsFile)
if err != nil {
return err
}
return nil
if err != nil {
return err
}
err = myZip.Close()
if err != nil {
return err
}
return nil
}
When I extract the file an error message appears: The compressed (zipped) Folder ... is invalid.
As answered by #JimB: file needs to be added as relative path
only forward slashes. can not start with slash
func Zip(src string, dst string) error {
destinationFile, err := os.Create(dst)
if err != nil {
return err
}
myZip := zip.NewWriter(destinationFile)
file := `C:\MA\testing\cldeploy-local.json`
// file needs to be added as relative path
// only forward slashes. can not start with slash
relPath := strings.TrimPrefix(file, filepath.Dir(src))
relPath = strings.Replace(relPath, `\`, `/`, -1)
relPath = strings.TrimLeft(relPath, `/`)
zipFile, err := myZip.Create(relPath)
fsFile, err := os.Open(file)
if err != nil {
return err
}
_, err = io.Copy(zipFile, fsFile)
if err != nil {
return err
}
return nil
if err != nil {
return err
}
err = myZip.Close()
if err != nil {
return err
}
return nil
}
I faced same issue.
Zipfile created, but empty if you open it with Windows Explorer. But if you open it with 7-zip filename looks like
Actual:
C:\Users\user\AppData\Local\Temp\products_list_432735630.zip\C:\Users\user\AppData\Local\Temp
Expected
C:\Users\user\AppData\Local\Temp\products_list_432735630.zip\
So if zip file and CSV file in one folder you must use relative path.
For Example
relativeFilename := filepath.Base(csvFile.Name())
zippedFile, err := zipper.Create(relativeFilename)
Related
I wrote a function that makes a .zip archive from a directory (and subdir-s, files in it)
func ZipDirectory(backupName string) error {
file, err := os.Create(backupName)
if err != nil {
return err
}
defer file.Close()
w := zip.NewWriter(file)
defer w.Close()
walker := func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
if info.IsDir() {
return nil
}
file, err := os.Open(path)
if err != nil {
return fmt.Errorf("error opening file %q: %w", path, err)
}
defer file.Close()
f, err := w.Create(path)
if err != nil {
return fmt.Errorf("error creating file %q: %w", path, err)
}
_, err = io.Copy(f, file)
if err != nil {
return fmt.Errorf("error copying file %q: %w", path, err)
}
return nil
}
err = filepath.Walk(config.FilePath, walker)
if err != nil {
return err
}
return nil
}
but if directory weights more then 3GB it creates empty archive which weights exactly as it should be.
it looks like this:
help, don't understand this mystic result....
Big zip archives shows empty after archivation because you don't have installed software for unzipiing it (like 7Zip)
After installation and openning with 7Zip everything works
I want to zip some directory inside content into zip file
e.g. assume i’ve this directory structure
dir1
file1.html
file2.go
Now I want to zip it to dir1.zip which is working
when I extract it I got the same structure...
I want to zip the content inside that when I unzip it I get the files inside without the `dir1' folder as root after extracting it
file1.html
file2.go
I try to play with the path’s with this code and it doesn’t work,
Any idea what I miss here ?
i've tryied
func Zipit(source, target string) error {
zipfile, err := os.Create(target)
if err != nil {
return err
}
defer zipfile.Close()
archive := zip.NewWriter(zipfile)
defer archive.Close()
info, err := os.Stat(source)
if err != nil {
return nil
}
var baseDir string
if info.IsDir() {
baseDir = filepath.Base(source)
}
filepath.Walk(source, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
header, err := zip.FileInfoHeader(info)
if err != nil {
return err
}
if baseDir != "" {
header.Name = filepath.Join(baseDir, strings.TrimPrefix(path, source))
}
if info.IsDir() {
header.Name += "/"
} else {
header.Method = zip.Deflate
}
writer, err := archive.CreateHeader(header)
if err != nil {
return err
}
if info.IsDir() {
return nil
}
file, err := os.Open(path)
if err != nil {
return err
}
defer file.Close()
_, err = io.Copy(writer, file)
return err
})
return err
}
dir.Zipit("path/dir1" +"/", "test"+".zip")
Or maybe there is simpler way in GO to achieve this?
Assuming you are calling your function as follows:
Zipit("dir1/", "dir1.zip")
All you need to do is remove the baseDir that is being added to the filename inside the archive.
You currently have the following code:
if baseDir != "" {
header.Name = filepath.Join(baseDir, strings.TrimPrefix(path, source))
}
baseDir here is dir1.
Simply omit the baseDir (but to keep trimming the prefix):
header.Name = strings.TrimPrefix(path, source)
This is very similar to Unzip artifacts with different name where all you need to do is modify header.Name as you see fit. It sounds like you need to examine the various filepath functions to see how they can help you.
I want to zip some directory inside content into zip file
e.g. assume i’ve this directory structure
dir1
file1.html
file2.go
Now I want to zip it to dir1.zip which is working
when I extract it I got the same structure...
I want to zip the content inside that when I unzip it I get the files inside without the `dir1' folder as root after extracting it
file1.html
file2.go
I try to play with the path’s with this code and it doesn’t work,
Any idea what I miss here ?
i've tryied
func Zipit(source, target string) error {
zipfile, err := os.Create(target)
if err != nil {
return err
}
defer zipfile.Close()
archive := zip.NewWriter(zipfile)
defer archive.Close()
info, err := os.Stat(source)
if err != nil {
return nil
}
var baseDir string
if info.IsDir() {
baseDir = filepath.Base(source)
}
filepath.Walk(source, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
header, err := zip.FileInfoHeader(info)
if err != nil {
return err
}
if baseDir != "" {
header.Name = filepath.Join(baseDir, strings.TrimPrefix(path, source))
}
if info.IsDir() {
header.Name += "/"
} else {
header.Method = zip.Deflate
}
writer, err := archive.CreateHeader(header)
if err != nil {
return err
}
if info.IsDir() {
return nil
}
file, err := os.Open(path)
if err != nil {
return err
}
defer file.Close()
_, err = io.Copy(writer, file)
return err
})
return err
}
dir.Zipit("path/dir1" +"/", "test"+".zip")
Or maybe there is simpler way in GO to achieve this?
Assuming you are calling your function as follows:
Zipit("dir1/", "dir1.zip")
All you need to do is remove the baseDir that is being added to the filename inside the archive.
You currently have the following code:
if baseDir != "" {
header.Name = filepath.Join(baseDir, strings.TrimPrefix(path, source))
}
baseDir here is dir1.
Simply omit the baseDir (but to keep trimming the prefix):
header.Name = strings.TrimPrefix(path, source)
This is very similar to Unzip artifacts with different name where all you need to do is modify header.Name as you see fit. It sounds like you need to examine the various filepath functions to see how they can help you.
So I'm trying to crate a tar.gz file file from multiple directories and files. Something with the same usage as:
tar -cvzf sometarfile.tar.gz somedir/ someotherdir/ somefile.json somefile.xml
Assuming the directories have other directories inside of them.
I have this as an input:
paths := []string{
"somedir/",
"someotherdir/",
"somefile.json",
"somefile.xml",
}
and using these:
func TarFilesDirs(paths []string, tarFilePath string ) error {
// set up the output file
file, err := os.Create(tarFilePath)
if err != nil {
return err
}
defer file.Close()
// set up the gzip writer
gz := gzip.NewWriter(file)
defer gz.Close()
tw := tar.NewWriter(gz)
defer tw.Close()
// add each file/dir as needed into the current tar archive
for _,i := range paths {
if err := tarit(i, tw); err != nil {
return err
}
}
return nil
}
func tarit(source string, tw *tar.Writer) error {
info, err := os.Stat(source)
if err != nil {
return nil
}
var baseDir string
if info.IsDir() {
baseDir = filepath.Base(source)
}
return filepath.Walk(source,
func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
header, err := tar.FileInfoHeader(info, info.Name())
if err != nil {
return err
}
if baseDir != "" {
header.Name = filepath.Join(baseDir, strings.TrimPrefix(path, source))
}
if err := tw.WriteHeader(header); err != nil {
return err
}
if info.IsDir() {
return nil
}
file, err := os.Open(path)
if err != nil {
return err
}
defer file.Close()
_, err = io.Copy(tw, file)
if err != nil {
log.Println("failing here")
return err
}
return err
})
}
Problem: if a directory is large I'm getting:
archive/tar: write too long
error, when I remove it everything works.
Ran out of ideas and wasted many hours on this trying to find a solution...
Any ideas?
Thanks
I was having a similar issue until I looked more closely at the tar.FileInfoHeader doc:
FileInfoHeader creates a partially-populated Header from fi. If fi describes a symlink, FileInfoHeader records link as the link target. If fi describes a directory, a slash is appended to the name. Because os.FileInfo's Name method returns only the base name of the file it describes, it may be necessary to modify the Name field of the returned header to provide the full path name of the file.
Essentially, FileInfoHeader isn't guaranteed to fill out all the header fields before you write it with WriteHeader, and if you look at the implementation the Size field is only set on regular files. Your code snippet seems to only handle directories, this means if you come across any other non regular file, you write the header with a size of zero then attempt to copy a potentially non-zero sized special file on disk into the tar. Go returns ErrWriteTooLong to stop you from creating a broken tar.
I came up with this and haven't had the issue since.
if err := filepath.Walk(directory, func(path string, info os.FileInfo, err error) error {
if err != nil {
return check(err)
}
var link string
if info.Mode()&os.ModeSymlink == os.ModeSymlink {
if link, err = os.Readlink(path); err != nil {
return check(err)
}
}
header, err := tar.FileInfoHeader(info, link)
if err != nil {
return check(err)
}
header.Name = filepath.Join(baseDir, strings.TrimPrefix(path, directory))
if err = tw.WriteHeader(header); err != nil {
return check(err)
}
if !info.Mode().IsRegular() { //nothing more to do for non-regular
return nil
}
fh, err := os.Open(path)
if err != nil {
return check(err)
}
defer fh.Close()
if _, err = io.CopyBuffer(tw, fh, buf); err != nil {
return check(err)
}
return nil
})
Since you are seeing this issue with a large directory only, I think the following fix might not help, but this will address the issue of creating a tar from files that might be continuously growing.
In my case the issue was that when we were creating the tar header, the header.Size (inside tar.FileInfoHeader) was getting setting to the file size (info.Size()) at that instant of time.
When we later in the code, try to open the concerned file (os.Open) and copy its contents (io.Copy) we risk copying more data than what we earlier set the tar header size to because the file could have grown in the meantime.
This piece of code will ensure we ONLY copy that much data as we set the tar header size to:
_, err = io.**CopyN**(tw, file, info.Size())
if err != nil {
log.Println("failing here")
return err
}
Write writes to the current entry in the tar archive. Write returns the error ErrWriteTooLong if more than hdr.Size bytes are written after WriteHeader.
There is a Size option you can add to the Header. Haven't tried it but maybe that helps...
See also https://golang.org/pkg/archive/tar/
I wrote code that unzips a file in a particular location then copies the contents of the folder to outside where the folder is unzipped then it removes the folder.
This is the Code I wrote:
package main
import (
"os"
"flag"
"fmt"
"io"
"path/filepath"
"os/exec"
"archive/zip"
"time"
)
func RemoveContents(dir string) error {
d, err := os.Open(dir)
if err != nil {
return err
}
names, err := d.Readdirnames(-1)
if err != nil {
return err
}
for _, name := range names {
err = os.RemoveAll(filepath.Join(dir, name))
if err != nil {
return err
}
}
d.Close()
return nil
}
func CopyFile(source string, dest string) (err error) {
sourcefile, err := os.Open(source)
if err != nil {
return err
}
defer sourcefile.Close()
destfile, err := os.Create(dest)
if err != nil {
return err
}
defer destfile.Close()
_, err = io.Copy(destfile, sourcefile)
if err == nil {
sourceinfo, err := os.Stat(source)
if err != nil {
err = os.Chmod(dest, sourceinfo.Mode())
}
}
return
}
func CopyDir(source string, dest string) (err error) {
// get properties of source dir
sourceinfo, err := os.Stat(source)
if err != nil {
return err
}
// create dest dir
err = os.MkdirAll(dest, sourceinfo.Mode())
if err != nil {
return err
}
directory, _ := os.Open(source)
objects, err := directory.Readdir(-1)
for _, obj := range objects {
sourcefilepointer := source + "/" + obj.Name()
destinationfilepointer := dest + "/" + obj.Name()
if obj.IsDir() {
// create sub-directories - recursively
err = CopyDir(sourcefilepointer, destinationfilepointer)
if err != nil {
fmt.Println(err)
}
} else {
// perform copy
err = CopyFile(sourcefilepointer, destinationfilepointer)
if err != nil {
fmt.Println(err)
}
}
}
return
}
func main() {
flag.Parse() // get the source and destination directory
source_dir := flag.Arg(0) // get the source directory from 1st argument
dest_dir := flag.Arg(1) // get the destination directory from the 2nd argument
os.MkdirAll("E:\\go\\copyDirectory\\myFile.zip",0777)
zipFilePath := "E:\\go\\copyDirectory\\myFile.zip"
tempWrkDir := "E:\\go\\copyDirectory\\"
//Read zip file and get path handle.
fileHandleReader, err := zip.OpenReader(zipFilePath)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
//open zip file and read all the folder and files inside
for _, fileReadHandler := range fileHandleReader.Reader.File {
//read the file or folder handle inside zip
fileOpenHandle, err := fileReadHandler.Open()
if err != nil {
fmt.Println(err)
os.Exit(1)
}
defer fileOpenHandle.Close()
targetUnZipPath := filepath.Join(tempWrkDir, fileReadHandler.Name)
if fileReadHandler.FileInfo().IsDir() {
os.MkdirAll(targetUnZipPath, fileReadHandler.Mode())
//fmt.Println("Creating directory", path)
}else {
// create new dummy file to copy original file.
newTempFileHandle, err := os.OpenFile(targetUnZipPath, os.O_WRONLY|os.O_CREATE, fileReadHandler.Mode())
if err != nil {
fmt.Println(err)
os.Exit(1)
}
defer newTempFileHandle.Close()
//copying original file to dummy file.
if _, err = io.Copy(newTempFileHandle, fileOpenHandle); err != nil {
fmt.Println(err)
os.Exit(1)
}
}
}
time.Sleep(1000*time.Millisecond)
fmt.Println("Source :" + source_dir)
// check if the source dir exist
src, err := os.Stat(source_dir)
if err != nil {
panic(err)
}
if !src.IsDir() {
fmt.Println("Source is not a directory")
os.Exit(1)
}
// create the destination directory
fmt.Println("Destination :"+ dest_dir)
/*_, err = os.Open(dest_dir)
if !os.IsNotExist(err) {
fmt.Println("Destination directory already exists. Abort!")
os.Exit(1)
}*/
err = CopyDir(source_dir, dest_dir)
if err != nil {
fmt.Println(err)
} else {
fmt.Println("Directory copied")
}
err = RemoveContents("./myFiles")
if err != nil {
fmt.Println("ERRR:::",err)
}
//time.Sleep(10000*time.Millisecond)
}
The problem is that everything works fine except for deleting the folder. The folder has only one file in it. The location of the file is as follows:
E:\go\copyDirectory\myfile\mytextfile.txt
The Location of the zip file is as follows:
E:\go\copyDirectory\myfile.zip
The zip file has only one text file. The File inside the zip file is as follows:
E:\go\copyDirectory\myfile.zip\myfile\mytextfile.txt
The error I get is:
ERRR::: remove myfile\mytextfile.txt: The process cannot
access the file because it is being used by another process.
Thanks in advance.
You aren't closing the file. This:
defer newTempFileHandle.Close()
Is run when main finishes, which is after:
err = RemoveContents("./myFiles")
You can wrap that bit of code in an unnamed function:
func() {
//read the file or folder handle inside zip
fileOpenHandle, err := fileReadHandler.Open()
if err != nil {
fmt.Println(err)
os.Exit(1)
}
defer fileOpenHandle.Close()
targetUnZipPath := filepath.Join(tempWrkDir, fileReadHandler.Name)
if fileReadHandler.FileInfo().IsDir() {
os.MkdirAll(targetUnZipPath, fileReadHandler.Mode())
//fmt.Println("Creating directory", path)
} else {
// create new dummy file to copy original file.
newTempFileHandle, err := os.OpenFile(targetUnZipPath, os.O_WRONLY|os.O_CREATE, fileReadHandler.Mode())
if err != nil {
fmt.Println(err)
os.Exit(1)
}
defer newTempFileHandle.Close()
//copying original file to dummy file.
if _, err = io.Copy(newTempFileHandle, fileOpenHandle); err != nil {
fmt.Println(err)
os.Exit(1)
}
}
}()
And then your defer will happen before you try and remove the files. I would recommend pulling this out into a named function though.