I am trying to zip a folder containing files using following code
func zipFolder(source string) error {
target := "uniq.zip"
zipfile, err := os.Create(target)
if err != nil {
return err
}
defer zipfile.Close()
archive := zip.NewWriter(zipfile)
defer archive.Close()
info, err := os.Stat(source)
if err != nil {
return nil
}
var baseDir string
if info.IsDir() {
baseDir = filepath.Base(source)
}
filepath.Walk(source, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
header, err := zip.FileInfoHeader(info)
if err != nil {
return err
}
if baseDir != "" {
header.Name = filepath.Join(baseDir, strings.TrimPrefix(path, source))
}
if info.IsDir() {
header.Name += "/"
} else {
header.Method = zip.Store
}
writer, err := archive.CreateHeader(header)
if err != nil {
return err
}
if info.IsDir() {
return nil
}
file, err := os.Open(path)
if err != nil {
return err
}
defer file.Close()
_, err = io.Copy(writer, file)
return err
})
return err
}
This successfully generates a zip while but when i unzip it (I am using mac), the compressed file is of 0 bytes without any content.
Any idea what am i doing wrong here?
Related
I have a tar.gz file that I received via HTTP request. Inside this tar is a folder, which has a file i'm interested in. I am unsure what parameter to give as root to WalkDir, most attempts end in an ENOENT error. My code:
func untarFiles(tarBytes io.ReadCloser) (*os.File, error) {
var dbFile os.File
gzipReader, err := gzip.NewReader(tarBytes)
if err != nil {
return nil, err
}
defer gzipReader.Close()
tarReader := tar.NewReader(gzipReader)
for {
header, err := tarReader.Next()
switch {
case err == io.EOF:
break
case header.FileInfo().IsDir():
err = filepath.WalkDir(header.Name, func(path string, info fs.DirEntry, err error) error {
if err != nil {
return err
}
if info.IsDir() {
return nil
}
if !strings.HasSuffix(path, "my_suffix") {
return nil
}
dirFile, err := os.Open(path)
if err != nil {
return err
}
defer dirFile.Close()
dbFile = *dirFile
return nil
})
break
}
break
}
return &dbFile, nil
I've tried header.Name, header.FileInfo().Name(). Do I have to save this as a temporary directory and delete it later?
Thanks!
Is there an easy way to copy a directory in go?
I have the following function:
err = CopyDir("sourceFolder","destinationFolder")
Nothing so far has worked, including libraries such as github.com/cf-guardian/guardian/kernel/fileutils
One important thing to note is that I need to preserve directory structure, including the sourceFolder itself, not simply copy all contents of the folder.
I believe that docker implementation can be considered as complete solution for handling edge cases:
https://github.com/moby/moby/blob/master/daemon/graphdriver/copy/copy.go
There are following good things:
unsupported file type rise error
preserving permissions and ownership
preserving extended attributes
preserving timestamp
but because of a lot of imports your tiny application becomes huge.
I've tried to combine several solutions but use stdlib and for Linux only:
func CopyDirectory(scrDir, dest string) error {
entries, err := os.ReadDir(scrDir)
if err != nil {
return err
}
for _, entry := range entries {
sourcePath := filepath.Join(scrDir, entry.Name())
destPath := filepath.Join(dest, entry.Name())
fileInfo, err := os.Stat(sourcePath)
if err != nil {
return err
}
stat, ok := fileInfo.Sys().(*syscall.Stat_t)
if !ok {
return fmt.Errorf("failed to get raw syscall.Stat_t data for '%s'", sourcePath)
}
switch fileInfo.Mode() & os.ModeType{
case os.ModeDir:
if err := CreateIfNotExists(destPath, 0755); err != nil {
return err
}
if err := CopyDirectory(sourcePath, destPath); err != nil {
return err
}
case os.ModeSymlink:
if err := CopySymLink(sourcePath, destPath); err != nil {
return err
}
default:
if err := Copy(sourcePath, destPath); err != nil {
return err
}
}
if err := os.Lchown(destPath, int(stat.Uid), int(stat.Gid)); err != nil {
return err
}
fInfo, err := entry.Info()
if err != nil {
return err
}
isSymlink := fInfo.Mode()&os.ModeSymlink != 0
if !isSymlink {
if err := os.Chmod(destPath, fInfo.Mode()); err != nil {
return err
}
}
}
return nil
}
func Copy(srcFile, dstFile string) error {
out, err := os.Create(dstFile)
if err != nil {
return err
}
defer out.Close()
in, err := os.Open(srcFile)
defer in.Close()
if err != nil {
return err
}
_, err = io.Copy(out, in)
if err != nil {
return err
}
return nil
}
func Exists(filePath string) bool {
if _, err := os.Stat(filePath); os.IsNotExist(err) {
return false
}
return true
}
func CreateIfNotExists(dir string, perm os.FileMode) error {
if Exists(dir) {
return nil
}
if err := os.MkdirAll(dir, perm); err != nil {
return fmt.Errorf("failed to create directory: '%s', error: '%s'", dir, err.Error())
}
return nil
}
func CopySymLink(source, dest string) error {
link, err := os.Readlink(source)
if err != nil {
return err
}
return os.Symlink(link, dest)
}
This package seems to do exactly what you want to do, give it a try.
From the readme:
err := Copy("your/source/directory", "your/destination/directory")
Not satisfied with the already listed options which include using sketchy libraries, or vastly bloated libraries.
In my case, I opted to do things the old fashioned way. With shell commands!
import (
"os/exec"
)
func main() {
// completely arbitrary paths
oldDir := "/home/arshbot/"
newDir := "/tmp/"
cmd := exec.Command("cp", "--recursive", oldDir, newDir)
cmd.Run()
}
This solution copies a directory recursively, including symbolic links. Trying to be efficient in the actual copy stage using streams.
Also it's fairly easy to handle more of irregular files if needed.
// CopyDir copies the content of src to dst. src should be a full path.
func CopyDir(dst, src string) error {
return filepath.Walk(src, func(path string, info fs.FileInfo, err error) error {
if err != nil {
return err
}
// copy to this path
outpath := filepath.Join(dst, strings.TrimPrefix(path, src))
if info.IsDir() {
os.MkdirAll(outpath, info.Mode())
return nil // means recursive
}
// handle irregular files
if !info.Mode().IsRegular() {
switch info.Mode().Type() & os.ModeType {
case os.ModeSymlink:
link, err := os.Readlink(path)
if err != nil {
return err
}
return os.Symlink(link, outpath)
}
return nil
}
// copy contents of regular file efficiently
// open input
in, _ := os.Open(path)
if err != nil {
return err
}
defer in.Close()
// create output
fh, err := os.Create(outpath)
if err != nil {
return err
}
defer fh.Close()
// make it the same
fh.Chmod(info.Mode())
// copy content
_, err = io.Copy(fh, in)
return err
})
}
I've come up with a relatively shorter answer which uses path/filepath's Walk method:
import (
"io/ioutil"
"path/filepath"
"os"
"strings"
)
func copy(source, destination string) error {
var err error = filepath.Walk(source, func(path string, info os.FileInfo, err error) error {
var relPath string = strings.Replace(path, source, "", 1)
if relPath == "" {
return nil
}
if info.IsDir() {
return os.Mkdir(filepath.Join(destination, relPath), 0755)
} else {
var data, err1 = ioutil.ReadFile(filepath.Join(source, relPath))
if err1 != nil {
return err1
}
return ioutil.WriteFile(filepath.Join(destination, relPath), data, 0777)
}
})
return err
}
Also this might be a solution:
available on github.com/floscodes/golang-tools
import (
"fmt"
"io/ioutil"
"os"
)
func CopyDir(src string, dest string) error {
if dest[:len(src)] == src {
return fmt.Errorf("Cannot copy a folder into the folder itself!")
}
f, err := os.Open(src)
if err != nil {
return err
}
file, err := f.Stat()
if err != nil {
return err
}
if !file.IsDir() {
return fmt.Errorf("Source " + file.Name() + " is not a directory!")
}
err = os.Mkdir(dest, 0755)
if err != nil {
return err
}
files, err := ioutil.ReadDir(src)
if err != nil {
return err
}
for _, f := range files {
if f.IsDir() {
err = CopyDir(src+"/"+f.Name(), dest+"/"+f.Name())
if err != nil {
return err
}
}
if !f.IsDir() {
content, err := ioutil.ReadFile(src + "/" + f.Name())
if err != nil {
return err
}
err = ioutil.WriteFile(dest+"/"+f.Name(), content, 0755)
if err != nil {
return err
}
}
}
return nil
}
I'm trying to move a file from my C-drive to my H-drive using os.Replace().
The code looks as follows:
func MoveFile(source string, destination string) {
err := os.Rename(source, destination)
if err != nil {
fmt.Println(err)
}
}
However, when I run the code I get the following error:
rename C:\old\path\to\file.txt H:\new\path\to\file.txt: The system cannot move the file to a different disk drive.
I found this issue on GitHub that specifies the problem but it appears that they will not change this function to allow it to move file on different disk drives.
I already searched for other possibilities to move files, but found nothing in the standard documentation or the internet.
So, what should I do now to be able to move files on different disk drives?
As the comment said, you'll need to create a new file on the other disk, copy the contents, and then remove the original. It's straightforward using os.Create, io.Copy, and os.Remove:
import (
"fmt"
"io"
"os"
)
func MoveFile(sourcePath, destPath string) error {
inputFile, err := os.Open(sourcePath)
if err != nil {
return fmt.Errorf("Couldn't open source file: %s", err)
}
outputFile, err := os.Create(destPath)
if err != nil {
inputFile.Close()
return fmt.Errorf("Couldn't open dest file: %s", err)
}
defer outputFile.Close()
_, err = io.Copy(outputFile, inputFile)
inputFile.Close()
if err != nil {
return fmt.Errorf("Writing to output file failed: %s", err)
}
// The copy was successful, so now delete the original file
err = os.Remove(sourcePath)
if err != nil {
return fmt.Errorf("Failed removing original file: %s", err)
}
return nil
}
You need to make sure that you handle all cases on both Linux and Windows. For example, for any size file,
package main
import (
"flag"
"fmt"
"io"
"os"
)
func MoveFile(source, destination string) (err error) {
src, err := os.Open(source)
if err != nil {
return err
}
defer src.Close()
fi, err := src.Stat()
if err != nil {
return err
}
flag := os.O_WRONLY | os.O_CREATE | os.O_TRUNC
perm := fi.Mode() & os.ModePerm
dst, err := os.OpenFile(destination, flag, perm)
if err != nil {
return err
}
defer dst.Close()
_, err = io.Copy(dst, src)
if err != nil {
dst.Close()
os.Remove(destination)
return err
}
err = dst.Close()
if err != nil {
return err
}
err = src.Close()
if err != nil {
return err
}
err = os.Remove(source)
if err != nil {
return err
}
return nil
}
func main() {
var src, dst string
flag.StringVar(&src, "src", "", "source file")
flag.StringVar(&dst, "dst", "", "destination file")
flag.Parse()
if src == "" || dst == "" {
flag.Usage()
os.Exit(1)
}
err := MoveFile(src, dst)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Printf("moved %q to %q\n", src, dst)
}
Output (Linux):
$ cp move.file src.file && go build movefile.go && ./movefile -src=src.file -dst=dst.file
moved "src.file" to "dst.file"
$
Output (Windows):
>copy /Y move.file src.file && go build movefile.go && movefile -src=src.file -dst=dst.file
moved "src.file" to "dst.file"
>
This solution Moves the file and preserves permissions:
func MoveFile(src, dst string) error {
in, err := os.Open(src)
if err != nil {
return fmt.Errorf("Couldn't open source file: %s", err)
}
out, err := os.Create(dst)
if err != nil {
in.Close()
return fmt.Errorf("Couldn't open dest file: %s", err)
}
defer out.Close()
_, err = io.Copy(out, in)
in.Close()
if err != nil {
return fmt.Errorf("Writing to output file failed: %s", err)
}
err = out.Sync()
if err != nil {
return fmt.Errorf("Sync error: %s", err)
}
si, err := os.Stat(src)
if err != nil {
return fmt.Errorf("Stat error: %s", err)
}
err = os.Chmod(dst, si.Mode())
if err != nil {
return fmt.Errorf("Chmod error: %s", err)
}
err = os.Remove(src)
if err != nil {
return fmt.Errorf("Failed removing original file: %s", err)
}
return nil
}
If only want to Copy the file without remove the original:
func CopyFile(src, dst string) error {
in, err := os.Open(src)
if err != nil {
return fmt.Errorf("Couldn't open source file: %s", err)
}
out, err := os.Create(dst)
if err != nil {
in.Close()
return fmt.Errorf("Couldn't open dest file: %s", err)
}
defer out.Close()
_, err = io.Copy(out, in)
in.Close()
if err != nil {
return fmt.Errorf("Writing to output file failed: %s", err)
}
err = out.Sync()
if err != nil {
return fmt.Errorf("Sync error: %s", err)
}
si, err := os.Stat(src)
if err != nil {
return fmt.Errorf("Stat error: %s", err)
}
err = os.Chmod(dst, si.Mode())
if err != nil {
return fmt.Errorf("Chmod error: %s", err)
}
return nil
}
Maybe you can use a magic approach, just using the syscall.MoveFile as follows.
func main() {
oldpath := "D:\\black.txt"
newpath := "E:\\black-new.txt"
from, _ := syscall.UTF16PtrFromString(oldpath)
to, _ := syscall.UTF16PtrFromString(newpath)
fmt.Println(*from, *to)
err := syscall.MoveFile(from, to)
if err != nil {
panic(err)
}
}
the program works.
if you want a cross-platform compatibility program, you can implement your own MoveFile.
func MoveFile(src string, dst string) error {
if runtime.GOOS == "windows" {
from, _ := syscall.UTF16PtrFromString(src)
to, _ := syscall.UTF16PtrFromString(dst)
return syscall.MoveFile(from, to)
} else {
return os.Rename(src, dst)
}
}
I was trying to unzip a .docx file. First file of name "[Content_Types].xml" has been extracted. But then I encountered an error as follows:
open frontend/uploads/doc_data/_rels/.rels: permission denied
How can we set permission for this?
The unzip function I use is as follows:
func Unzip(src, dest string) error {
r, err := zip.OpenReader(src)
if err != nil {
return err
}
defer r.Close()
for _, f := range r.File {
rc, err := f.Open()
if err != nil {
return err
}
defer rc.Close()
fpath := filepath.Join(dest, f.Name)
if f.FileInfo().IsDir() {
os.MkdirAll(fpath, f.Mode())
} else {
var fdir string
if lastIndex := strings.LastIndex(fpath,string(os.PathSeparator)); lastIndex > -1 {
fdir = fpath[:lastIndex]
}
err = os.MkdirAll(fdir, f.Mode())
if err != nil {
log.Fatal(err)
return err
}
f, err := os.OpenFile(
fpath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, f.Mode())
if err != nil {
return err
}
defer f.Close()
_, err = io.Copy(f, rc)
if err != nil {
return err
}
}
}
return nil
}
Try using the Chmod function of the os package to set the file permissions to something that allows you to do what you want.
err := os.Chmod("file.txt", 0777)
if err != nil {
fmt.Println(err)
}
more info here : https://socketloop.com/tutorials/golang-change-file-read-or-write-permission-example
I wrote code that unzips a file in a particular location then copies the contents of the folder to outside where the folder is unzipped then it removes the folder.
This is the Code I wrote:
package main
import (
"os"
"flag"
"fmt"
"io"
"path/filepath"
"os/exec"
"archive/zip"
"time"
)
func RemoveContents(dir string) error {
d, err := os.Open(dir)
if err != nil {
return err
}
names, err := d.Readdirnames(-1)
if err != nil {
return err
}
for _, name := range names {
err = os.RemoveAll(filepath.Join(dir, name))
if err != nil {
return err
}
}
d.Close()
return nil
}
func CopyFile(source string, dest string) (err error) {
sourcefile, err := os.Open(source)
if err != nil {
return err
}
defer sourcefile.Close()
destfile, err := os.Create(dest)
if err != nil {
return err
}
defer destfile.Close()
_, err = io.Copy(destfile, sourcefile)
if err == nil {
sourceinfo, err := os.Stat(source)
if err != nil {
err = os.Chmod(dest, sourceinfo.Mode())
}
}
return
}
func CopyDir(source string, dest string) (err error) {
// get properties of source dir
sourceinfo, err := os.Stat(source)
if err != nil {
return err
}
// create dest dir
err = os.MkdirAll(dest, sourceinfo.Mode())
if err != nil {
return err
}
directory, _ := os.Open(source)
objects, err := directory.Readdir(-1)
for _, obj := range objects {
sourcefilepointer := source + "/" + obj.Name()
destinationfilepointer := dest + "/" + obj.Name()
if obj.IsDir() {
// create sub-directories - recursively
err = CopyDir(sourcefilepointer, destinationfilepointer)
if err != nil {
fmt.Println(err)
}
} else {
// perform copy
err = CopyFile(sourcefilepointer, destinationfilepointer)
if err != nil {
fmt.Println(err)
}
}
}
return
}
func main() {
flag.Parse() // get the source and destination directory
source_dir := flag.Arg(0) // get the source directory from 1st argument
dest_dir := flag.Arg(1) // get the destination directory from the 2nd argument
os.MkdirAll("E:\\go\\copyDirectory\\myFile.zip",0777)
zipFilePath := "E:\\go\\copyDirectory\\myFile.zip"
tempWrkDir := "E:\\go\\copyDirectory\\"
//Read zip file and get path handle.
fileHandleReader, err := zip.OpenReader(zipFilePath)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
//open zip file and read all the folder and files inside
for _, fileReadHandler := range fileHandleReader.Reader.File {
//read the file or folder handle inside zip
fileOpenHandle, err := fileReadHandler.Open()
if err != nil {
fmt.Println(err)
os.Exit(1)
}
defer fileOpenHandle.Close()
targetUnZipPath := filepath.Join(tempWrkDir, fileReadHandler.Name)
if fileReadHandler.FileInfo().IsDir() {
os.MkdirAll(targetUnZipPath, fileReadHandler.Mode())
//fmt.Println("Creating directory", path)
}else {
// create new dummy file to copy original file.
newTempFileHandle, err := os.OpenFile(targetUnZipPath, os.O_WRONLY|os.O_CREATE, fileReadHandler.Mode())
if err != nil {
fmt.Println(err)
os.Exit(1)
}
defer newTempFileHandle.Close()
//copying original file to dummy file.
if _, err = io.Copy(newTempFileHandle, fileOpenHandle); err != nil {
fmt.Println(err)
os.Exit(1)
}
}
}
time.Sleep(1000*time.Millisecond)
fmt.Println("Source :" + source_dir)
// check if the source dir exist
src, err := os.Stat(source_dir)
if err != nil {
panic(err)
}
if !src.IsDir() {
fmt.Println("Source is not a directory")
os.Exit(1)
}
// create the destination directory
fmt.Println("Destination :"+ dest_dir)
/*_, err = os.Open(dest_dir)
if !os.IsNotExist(err) {
fmt.Println("Destination directory already exists. Abort!")
os.Exit(1)
}*/
err = CopyDir(source_dir, dest_dir)
if err != nil {
fmt.Println(err)
} else {
fmt.Println("Directory copied")
}
err = RemoveContents("./myFiles")
if err != nil {
fmt.Println("ERRR:::",err)
}
//time.Sleep(10000*time.Millisecond)
}
The problem is that everything works fine except for deleting the folder. The folder has only one file in it. The location of the file is as follows:
E:\go\copyDirectory\myfile\mytextfile.txt
The Location of the zip file is as follows:
E:\go\copyDirectory\myfile.zip
The zip file has only one text file. The File inside the zip file is as follows:
E:\go\copyDirectory\myfile.zip\myfile\mytextfile.txt
The error I get is:
ERRR::: remove myfile\mytextfile.txt: The process cannot
access the file because it is being used by another process.
Thanks in advance.
You aren't closing the file. This:
defer newTempFileHandle.Close()
Is run when main finishes, which is after:
err = RemoveContents("./myFiles")
You can wrap that bit of code in an unnamed function:
func() {
//read the file or folder handle inside zip
fileOpenHandle, err := fileReadHandler.Open()
if err != nil {
fmt.Println(err)
os.Exit(1)
}
defer fileOpenHandle.Close()
targetUnZipPath := filepath.Join(tempWrkDir, fileReadHandler.Name)
if fileReadHandler.FileInfo().IsDir() {
os.MkdirAll(targetUnZipPath, fileReadHandler.Mode())
//fmt.Println("Creating directory", path)
} else {
// create new dummy file to copy original file.
newTempFileHandle, err := os.OpenFile(targetUnZipPath, os.O_WRONLY|os.O_CREATE, fileReadHandler.Mode())
if err != nil {
fmt.Println(err)
os.Exit(1)
}
defer newTempFileHandle.Close()
//copying original file to dummy file.
if _, err = io.Copy(newTempFileHandle, fileOpenHandle); err != nil {
fmt.Println(err)
os.Exit(1)
}
}
}()
And then your defer will happen before you try and remove the files. I would recommend pulling this out into a named function though.