I want to zip some directory inside content into zip file
e.g. assume i’ve this directory structure
dir1
file1.html
file2.go
Now I want to zip it to dir1.zip which is working
when I extract it I got the same structure...
I want to zip the content inside that when I unzip it I get the files inside without the `dir1' folder as root after extracting it
file1.html
file2.go
I try to play with the path’s with this code and it doesn’t work,
Any idea what I miss here ?
i've tryied
func Zipit(source, target string) error {
zipfile, err := os.Create(target)
if err != nil {
return err
}
defer zipfile.Close()
archive := zip.NewWriter(zipfile)
defer archive.Close()
info, err := os.Stat(source)
if err != nil {
return nil
}
var baseDir string
if info.IsDir() {
baseDir = filepath.Base(source)
}
filepath.Walk(source, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
header, err := zip.FileInfoHeader(info)
if err != nil {
return err
}
if baseDir != "" {
header.Name = filepath.Join(baseDir, strings.TrimPrefix(path, source))
}
if info.IsDir() {
header.Name += "/"
} else {
header.Method = zip.Deflate
}
writer, err := archive.CreateHeader(header)
if err != nil {
return err
}
if info.IsDir() {
return nil
}
file, err := os.Open(path)
if err != nil {
return err
}
defer file.Close()
_, err = io.Copy(writer, file)
return err
})
return err
}
dir.Zipit("path/dir1" +"/", "test"+".zip")
Or maybe there is simpler way in GO to achieve this?
Assuming you are calling your function as follows:
Zipit("dir1/", "dir1.zip")
All you need to do is remove the baseDir that is being added to the filename inside the archive.
You currently have the following code:
if baseDir != "" {
header.Name = filepath.Join(baseDir, strings.TrimPrefix(path, source))
}
baseDir here is dir1.
Simply omit the baseDir (but to keep trimming the prefix):
header.Name = strings.TrimPrefix(path, source)
This is very similar to Unzip artifacts with different name where all you need to do is modify header.Name as you see fit. It sounds like you need to examine the various filepath functions to see how they can help you.
Related
I have the following function I'v tried to simplify to just adding a single file to a .zip archive.
Whatever I try, the resulting .zip file has no files listed in it. The size of the archive is correct. But when I try to extract all (windows), the archive is empty.
go version go1.10.1 windows/amd64
func Zip(src string, dst string) error {
destinationFile, err := os.Create(dst)
if err != nil {
return err
}
myZip := zip.NewWriter(destinationFile)
file := `C:\MA\testing\cldeploy-local.json`
zipFile, err := myZip.Create(file)
fsFile, err := os.Open(file)
if err != nil {
return err
}
_, err = io.Copy(zipFile, fsFile)
if err != nil {
return err
}
return nil
if err != nil {
return err
}
err = myZip.Close()
if err != nil {
return err
}
return nil
}
When I extract the file an error message appears: The compressed (zipped) Folder ... is invalid.
As answered by #JimB: file needs to be added as relative path
only forward slashes. can not start with slash
func Zip(src string, dst string) error {
destinationFile, err := os.Create(dst)
if err != nil {
return err
}
myZip := zip.NewWriter(destinationFile)
file := `C:\MA\testing\cldeploy-local.json`
// file needs to be added as relative path
// only forward slashes. can not start with slash
relPath := strings.TrimPrefix(file, filepath.Dir(src))
relPath = strings.Replace(relPath, `\`, `/`, -1)
relPath = strings.TrimLeft(relPath, `/`)
zipFile, err := myZip.Create(relPath)
fsFile, err := os.Open(file)
if err != nil {
return err
}
_, err = io.Copy(zipFile, fsFile)
if err != nil {
return err
}
return nil
if err != nil {
return err
}
err = myZip.Close()
if err != nil {
return err
}
return nil
}
I faced same issue.
Zipfile created, but empty if you open it with Windows Explorer. But if you open it with 7-zip filename looks like
Actual:
C:\Users\user\AppData\Local\Temp\products_list_432735630.zip\C:\Users\user\AppData\Local\Temp
Expected
C:\Users\user\AppData\Local\Temp\products_list_432735630.zip\
So if zip file and CSV file in one folder you must use relative path.
For Example
relativeFilename := filepath.Base(csvFile.Name())
zippedFile, err := zipper.Create(relativeFilename)
I want to zip some directory inside content into zip file
e.g. assume i’ve this directory structure
dir1
file1.html
file2.go
Now I want to zip it to dir1.zip which is working
when I extract it I got the same structure...
I want to zip the content inside that when I unzip it I get the files inside without the `dir1' folder as root after extracting it
file1.html
file2.go
I try to play with the path’s with this code and it doesn’t work,
Any idea what I miss here ?
i've tryied
func Zipit(source, target string) error {
zipfile, err := os.Create(target)
if err != nil {
return err
}
defer zipfile.Close()
archive := zip.NewWriter(zipfile)
defer archive.Close()
info, err := os.Stat(source)
if err != nil {
return nil
}
var baseDir string
if info.IsDir() {
baseDir = filepath.Base(source)
}
filepath.Walk(source, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
header, err := zip.FileInfoHeader(info)
if err != nil {
return err
}
if baseDir != "" {
header.Name = filepath.Join(baseDir, strings.TrimPrefix(path, source))
}
if info.IsDir() {
header.Name += "/"
} else {
header.Method = zip.Deflate
}
writer, err := archive.CreateHeader(header)
if err != nil {
return err
}
if info.IsDir() {
return nil
}
file, err := os.Open(path)
if err != nil {
return err
}
defer file.Close()
_, err = io.Copy(writer, file)
return err
})
return err
}
dir.Zipit("path/dir1" +"/", "test"+".zip")
Or maybe there is simpler way in GO to achieve this?
Assuming you are calling your function as follows:
Zipit("dir1/", "dir1.zip")
All you need to do is remove the baseDir that is being added to the filename inside the archive.
You currently have the following code:
if baseDir != "" {
header.Name = filepath.Join(baseDir, strings.TrimPrefix(path, source))
}
baseDir here is dir1.
Simply omit the baseDir (but to keep trimming the prefix):
header.Name = strings.TrimPrefix(path, source)
This is very similar to Unzip artifacts with different name where all you need to do is modify header.Name as you see fit. It sounds like you need to examine the various filepath functions to see how they can help you.
So I'm trying to crate a tar.gz file file from multiple directories and files. Something with the same usage as:
tar -cvzf sometarfile.tar.gz somedir/ someotherdir/ somefile.json somefile.xml
Assuming the directories have other directories inside of them.
I have this as an input:
paths := []string{
"somedir/",
"someotherdir/",
"somefile.json",
"somefile.xml",
}
and using these:
func TarFilesDirs(paths []string, tarFilePath string ) error {
// set up the output file
file, err := os.Create(tarFilePath)
if err != nil {
return err
}
defer file.Close()
// set up the gzip writer
gz := gzip.NewWriter(file)
defer gz.Close()
tw := tar.NewWriter(gz)
defer tw.Close()
// add each file/dir as needed into the current tar archive
for _,i := range paths {
if err := tarit(i, tw); err != nil {
return err
}
}
return nil
}
func tarit(source string, tw *tar.Writer) error {
info, err := os.Stat(source)
if err != nil {
return nil
}
var baseDir string
if info.IsDir() {
baseDir = filepath.Base(source)
}
return filepath.Walk(source,
func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
header, err := tar.FileInfoHeader(info, info.Name())
if err != nil {
return err
}
if baseDir != "" {
header.Name = filepath.Join(baseDir, strings.TrimPrefix(path, source))
}
if err := tw.WriteHeader(header); err != nil {
return err
}
if info.IsDir() {
return nil
}
file, err := os.Open(path)
if err != nil {
return err
}
defer file.Close()
_, err = io.Copy(tw, file)
if err != nil {
log.Println("failing here")
return err
}
return err
})
}
Problem: if a directory is large I'm getting:
archive/tar: write too long
error, when I remove it everything works.
Ran out of ideas and wasted many hours on this trying to find a solution...
Any ideas?
Thanks
I was having a similar issue until I looked more closely at the tar.FileInfoHeader doc:
FileInfoHeader creates a partially-populated Header from fi. If fi describes a symlink, FileInfoHeader records link as the link target. If fi describes a directory, a slash is appended to the name. Because os.FileInfo's Name method returns only the base name of the file it describes, it may be necessary to modify the Name field of the returned header to provide the full path name of the file.
Essentially, FileInfoHeader isn't guaranteed to fill out all the header fields before you write it with WriteHeader, and if you look at the implementation the Size field is only set on regular files. Your code snippet seems to only handle directories, this means if you come across any other non regular file, you write the header with a size of zero then attempt to copy a potentially non-zero sized special file on disk into the tar. Go returns ErrWriteTooLong to stop you from creating a broken tar.
I came up with this and haven't had the issue since.
if err := filepath.Walk(directory, func(path string, info os.FileInfo, err error) error {
if err != nil {
return check(err)
}
var link string
if info.Mode()&os.ModeSymlink == os.ModeSymlink {
if link, err = os.Readlink(path); err != nil {
return check(err)
}
}
header, err := tar.FileInfoHeader(info, link)
if err != nil {
return check(err)
}
header.Name = filepath.Join(baseDir, strings.TrimPrefix(path, directory))
if err = tw.WriteHeader(header); err != nil {
return check(err)
}
if !info.Mode().IsRegular() { //nothing more to do for non-regular
return nil
}
fh, err := os.Open(path)
if err != nil {
return check(err)
}
defer fh.Close()
if _, err = io.CopyBuffer(tw, fh, buf); err != nil {
return check(err)
}
return nil
})
Since you are seeing this issue with a large directory only, I think the following fix might not help, but this will address the issue of creating a tar from files that might be continuously growing.
In my case the issue was that when we were creating the tar header, the header.Size (inside tar.FileInfoHeader) was getting setting to the file size (info.Size()) at that instant of time.
When we later in the code, try to open the concerned file (os.Open) and copy its contents (io.Copy) we risk copying more data than what we earlier set the tar header size to because the file could have grown in the meantime.
This piece of code will ensure we ONLY copy that much data as we set the tar header size to:
_, err = io.**CopyN**(tw, file, info.Size())
if err != nil {
log.Println("failing here")
return err
}
Write writes to the current entry in the tar archive. Write returns the error ErrWriteTooLong if more than hdr.Size bytes are written after WriteHeader.
There is a Size option you can add to the Header. Haven't tried it but maybe that helps...
See also https://golang.org/pkg/archive/tar/
I know it will have to do with the zip package I just have no idea how I would implement such a thing.
Here's solution that uses Go's built-in recursive file-walker since the top answer so far has implemented their own file-walker:
Aside, some findings I've found while generating zip files today that might save someone else a headache:
The argument to w.Create(zippath) should not start with a "/", and should be relative to the zip root (aka the folder that would be created if you unzipped the archive). So a top-level "manifest.xml" file should be w.Create("manifest.xml"). Nested files should be w.Create("a/b/c.css). If you are generating bad/surprising archives, check first to ensure you aren't breaking this rule. My code does not try to enforce this.
Some specs (like epub) want files in a certain order, but Go's filepath.Walk will crawl in lexical order. (For that matter, I've found all epub parsers to be lenient here so far from Calibre to Books.app on macOS). If you need a specific order, then #LeTigre's solution with ReadDir will let you sort the files at each level of descent.
package main
import (
"archive/zip"
"fmt"
"io"
"os"
"path/filepath"
)
// Zips "./input" into "./output.zip"
func main() {
file, err := os.Create("output.zip")
if err != nil {
panic(err)
}
defer file.Close()
w := zip.NewWriter(file)
defer w.Close()
walker := func(path string, info os.FileInfo, err error) error {
fmt.Printf("Crawling: %#v\n", path)
if err != nil {
return err
}
if info.IsDir() {
return nil
}
file, err := os.Open(path)
if err != nil {
return err
}
defer file.Close()
// Ensure that `path` is not absolute; it should not start with "/".
// This snippet happens to work because I don't use
// absolute paths, but ensure your real-world code
// transforms path into a zip-root relative path.
f, err := w.Create(path)
if err != nil {
return err
}
_, err = io.Copy(f, file)
if err != nil {
return err
}
return nil
}
err = filepath.Walk("input", walker)
if err != nil {
panic(err)
}
}
To do it manually, you could modify the code linked above:
ExampleZipWriter
To give you a simple example, that has many flaws, but might be easily understood:
func ZipWriter() {
baseFolder := "/Users/tom/Desktop/testing/"
// Get a Buffer to Write To
outFile, err := os.Create(`/Users/tom/Desktop/zip.zip`)
if err != nil {
fmt.Println(err)
}
defer outFile.Close()
// Create a new zip archive.
w := zip.NewWriter(outFile)
// Add some files to the archive.
addFiles(w, baseFolder, "")
if err != nil {
fmt.Println(err)
}
// Make sure to check the error on Close.
err = w.Close()
if err != nil {
fmt.Println(err)
}
}
We use this to iterate on files recursively to generate folders too:
func addFiles(w *zip.Writer, basePath, baseInZip string) {
// Open the Directory
files, err := ioutil.ReadDir(basePath)
if err != nil {
fmt.Println(err)
}
for _, file := range files {
fmt.Println(basePath + file.Name())
if !file.IsDir() {
dat, err := ioutil.ReadFile(basePath + file.Name())
if err != nil {
fmt.Println(err)
}
// Add some files to the archive.
f, err := w.Create(baseInZip + file.Name())
if err != nil {
fmt.Println(err)
}
_, err = f.Write(dat)
if err != nil {
fmt.Println(err)
}
} else if file.IsDir() {
// Recurse
newBase := basePath + file.Name() + "/"
fmt.Println("Recursing and Adding SubDir: " + file.Name())
fmt.Println("Recursing and Adding SubDir: " + newBase)
addFiles(w, newBase, baseInZip + file.Name() + "/")
}
}
}
#danneu's answer is good. But if the directory contains empty subdirectories, the code doesn't work and it will ignore them.
So instead of just returning nil, we should create a directory:
if info.IsDir() {
// add a trailing slash for creating dir
path = fmt.Sprintf("%s%c", path, os.PathSeparator)
_, err = w.Create(path)
return err
}
an ez way to zip directory, you can run command of server:
out, err := exec.Command("zip", "-r", "-D", "ideaz.zip", ".idea/*").Output()
https://play.golang.org/p/Mn-HmUjm5q2
I wrote code that unzips a file in a particular location then copies the contents of the folder to outside where the folder is unzipped then it removes the folder.
This is the Code I wrote:
package main
import (
"os"
"flag"
"fmt"
"io"
"path/filepath"
"os/exec"
"archive/zip"
"time"
)
func RemoveContents(dir string) error {
d, err := os.Open(dir)
if err != nil {
return err
}
names, err := d.Readdirnames(-1)
if err != nil {
return err
}
for _, name := range names {
err = os.RemoveAll(filepath.Join(dir, name))
if err != nil {
return err
}
}
d.Close()
return nil
}
func CopyFile(source string, dest string) (err error) {
sourcefile, err := os.Open(source)
if err != nil {
return err
}
defer sourcefile.Close()
destfile, err := os.Create(dest)
if err != nil {
return err
}
defer destfile.Close()
_, err = io.Copy(destfile, sourcefile)
if err == nil {
sourceinfo, err := os.Stat(source)
if err != nil {
err = os.Chmod(dest, sourceinfo.Mode())
}
}
return
}
func CopyDir(source string, dest string) (err error) {
// get properties of source dir
sourceinfo, err := os.Stat(source)
if err != nil {
return err
}
// create dest dir
err = os.MkdirAll(dest, sourceinfo.Mode())
if err != nil {
return err
}
directory, _ := os.Open(source)
objects, err := directory.Readdir(-1)
for _, obj := range objects {
sourcefilepointer := source + "/" + obj.Name()
destinationfilepointer := dest + "/" + obj.Name()
if obj.IsDir() {
// create sub-directories - recursively
err = CopyDir(sourcefilepointer, destinationfilepointer)
if err != nil {
fmt.Println(err)
}
} else {
// perform copy
err = CopyFile(sourcefilepointer, destinationfilepointer)
if err != nil {
fmt.Println(err)
}
}
}
return
}
func main() {
flag.Parse() // get the source and destination directory
source_dir := flag.Arg(0) // get the source directory from 1st argument
dest_dir := flag.Arg(1) // get the destination directory from the 2nd argument
os.MkdirAll("E:\\go\\copyDirectory\\myFile.zip",0777)
zipFilePath := "E:\\go\\copyDirectory\\myFile.zip"
tempWrkDir := "E:\\go\\copyDirectory\\"
//Read zip file and get path handle.
fileHandleReader, err := zip.OpenReader(zipFilePath)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
//open zip file and read all the folder and files inside
for _, fileReadHandler := range fileHandleReader.Reader.File {
//read the file or folder handle inside zip
fileOpenHandle, err := fileReadHandler.Open()
if err != nil {
fmt.Println(err)
os.Exit(1)
}
defer fileOpenHandle.Close()
targetUnZipPath := filepath.Join(tempWrkDir, fileReadHandler.Name)
if fileReadHandler.FileInfo().IsDir() {
os.MkdirAll(targetUnZipPath, fileReadHandler.Mode())
//fmt.Println("Creating directory", path)
}else {
// create new dummy file to copy original file.
newTempFileHandle, err := os.OpenFile(targetUnZipPath, os.O_WRONLY|os.O_CREATE, fileReadHandler.Mode())
if err != nil {
fmt.Println(err)
os.Exit(1)
}
defer newTempFileHandle.Close()
//copying original file to dummy file.
if _, err = io.Copy(newTempFileHandle, fileOpenHandle); err != nil {
fmt.Println(err)
os.Exit(1)
}
}
}
time.Sleep(1000*time.Millisecond)
fmt.Println("Source :" + source_dir)
// check if the source dir exist
src, err := os.Stat(source_dir)
if err != nil {
panic(err)
}
if !src.IsDir() {
fmt.Println("Source is not a directory")
os.Exit(1)
}
// create the destination directory
fmt.Println("Destination :"+ dest_dir)
/*_, err = os.Open(dest_dir)
if !os.IsNotExist(err) {
fmt.Println("Destination directory already exists. Abort!")
os.Exit(1)
}*/
err = CopyDir(source_dir, dest_dir)
if err != nil {
fmt.Println(err)
} else {
fmt.Println("Directory copied")
}
err = RemoveContents("./myFiles")
if err != nil {
fmt.Println("ERRR:::",err)
}
//time.Sleep(10000*time.Millisecond)
}
The problem is that everything works fine except for deleting the folder. The folder has only one file in it. The location of the file is as follows:
E:\go\copyDirectory\myfile\mytextfile.txt
The Location of the zip file is as follows:
E:\go\copyDirectory\myfile.zip
The zip file has only one text file. The File inside the zip file is as follows:
E:\go\copyDirectory\myfile.zip\myfile\mytextfile.txt
The error I get is:
ERRR::: remove myfile\mytextfile.txt: The process cannot
access the file because it is being used by another process.
Thanks in advance.
You aren't closing the file. This:
defer newTempFileHandle.Close()
Is run when main finishes, which is after:
err = RemoveContents("./myFiles")
You can wrap that bit of code in an unnamed function:
func() {
//read the file or folder handle inside zip
fileOpenHandle, err := fileReadHandler.Open()
if err != nil {
fmt.Println(err)
os.Exit(1)
}
defer fileOpenHandle.Close()
targetUnZipPath := filepath.Join(tempWrkDir, fileReadHandler.Name)
if fileReadHandler.FileInfo().IsDir() {
os.MkdirAll(targetUnZipPath, fileReadHandler.Mode())
//fmt.Println("Creating directory", path)
} else {
// create new dummy file to copy original file.
newTempFileHandle, err := os.OpenFile(targetUnZipPath, os.O_WRONLY|os.O_CREATE, fileReadHandler.Mode())
if err != nil {
fmt.Println(err)
os.Exit(1)
}
defer newTempFileHandle.Close()
//copying original file to dummy file.
if _, err = io.Copy(newTempFileHandle, fileOpenHandle); err != nil {
fmt.Println(err)
os.Exit(1)
}
}
}()
And then your defer will happen before you try and remove the files. I would recommend pulling this out into a named function though.