I am trying to traverse the file structure in Breadth first fashion. I am getting segmentation violation - go

I know about ioutil.ReadDir and os.filePath but none of them traverse the directory in Breadth first fashion.
My approach is to call ioutil.ReadDir and append all the contents of the root dir into a slice. Then I am iterating over the contents and checking if it IsDir[] and calling the function recursively if true.
package main
import (
"io/ioutil"
"os"
)
var files []string
var path string
func appendFiles(root string) {
fileInfo, err := ioutil.ReadDir(root)
if err != nil {
return
}
for _, file := range fileInfo {
files = append(files, file.Name())
}
for _, file := range fileInfo {
fileStat, _ := os.Stat(file.Name())
if fileStat.Mode().IsDir() {
// path = path + "/" + file.Name()
appendFiles(file.Name())
}
}
}
func main() {
appendFiles(".")
}

The problem is that os.Stat() may return an error which you omit. When that happens, fileStat may be nil, so calling fileStat.Mode() in the next line panics.
And the reason os.Stat() fails is because file.Name() is relative to root, file.Name() by itself has little chance to exist, it must be joined with root. If os.Stat() is called with a file name that doesn't exist, it returns a nil file info and a non-nil error.
You may use filepath.Join() to construct a valid path for files that os.Stat() will work with. And it would be better to handle errors, e.g. return them, which you can inspect in main().
func appendFiles(root string) error {
fileInfo, err := ioutil.ReadDir(root)
if err != nil {
return fmt.Errorf("ReadDir error: %w", err)
}
for _, file := range fileInfo {
files = append(files, filepath.Join(root, file.Name()))
}
for _, file := range fileInfo {
fullName := filepath.Join(root, file.Name())
fileStat, err := os.Stat(fullName)
if err != nil {
return fmt.Errorf("Stat error: %w", err)
}
if fileStat.Mode().IsDir() {
if err := appendFiles(fullName); err != nil {
return fmt.Errorf("appendFiles error: %w", err)
}
}
}
return nil
}
func main() {
if err := appendFiles("."); err != nil {
fmt.Println(err)
}
}

Related

Golang: Facing error while creating .tar.gz file having large name

I am trying to create a .tar.gz file from folder that contains multiple files / folders. Once the .tar.gz file gets created, while extracting, the files are not not properly extracted. Mostly I think its because of large names or path exceeding some n characters, because same thing works when the filename/path is small. I referred this https://github.com/golang/go/issues/17630 and tried to add below code but it did not help.
header.Uid = 0
header.Gid = 0
I am using simple code seen below to create .tar.gz. The approach is, I create a temp folder, do some processing on the files and from that temp path, I create the .tar.gz file hence in the path below I am using pre-defined temp folder path.
package main
import (
"archive/tar"
"compress/gzip"
"fmt"
"io"
"log"
"os"
fp "path/filepath"
)
func main() {
// Create output file
out, err := os.Create("output.tar.gz")
if err != nil {
log.Fatalln("Error writing archive:", err)
}
defer out.Close()
// Create the archive and write the output to the "out" Writer
tmpDir := "C:/Users/USERNAME~1/AppData/Local/Temp/temp-241232063"
err = createArchive1(tmpDir, out)
if err != nil {
log.Fatalln("Error creating archive:", err)
}
fmt.Println("Archive created successfully")
}
func createArchive1(path string, targetFile *os.File) error {
gw := gzip.NewWriter(targetFile)
defer gw.Close()
tw := tar.NewWriter(gw)
defer tw.Close()
// walk through every file in the folder
err := fp.Walk(path, func(filePath string, info os.FileInfo, err error) error {
// ensure the src actually exists before trying to tar it
if _, err := os.Stat(filePath); err != nil {
return err
}
if err != nil {
return err
}
if info.IsDir() {
return nil
}
file, err := os.Open(filePath)
if err != nil {
return err
}
defer file.Close()
// generate tar header
header, err := tar.FileInfoHeader(info, info.Name())
header.Uid = 0
header.Gid = 0
if err != nil {
return err
}
header.Name = filePath //strings.TrimPrefix(filePath, fmt.Sprintf("%s/", fp.Dir(path))) //info.Name()
// write header
if err := tw.WriteHeader(header); err != nil {
return err
}
if _, err := io.Copy(tw, file); err != nil {
return err
}
return nil
})
return err
}
Please let me know what wrong I am doing.

How to compress a file to .zip without directory folder in Go

There're examples about compressing a file to .zip in Go. However, the file they generate include the directory folder. When I decompress the .zip file, there will be a new folder.
So, how can I compress a file to .zip without getting the directory folder included?
An example:
https://golangcode.com/create-zip-files-in-go/
package main
import (
"archive/zip"
"fmt"
"io"
"os"
)
func main() {
// List of Files to Zip
files := []string{"example.csv", "data.csv"}
output := "done.zip"
if err := ZipFiles(output, files); err != nil {
panic(err)
}
fmt.Println("Zipped File:", output)
}
// ZipFiles compresses one or many files into a single zip archive file.
// Param 1: filename is the output zip file's name.
// Param 2: files is a list of files to add to the zip.
func ZipFiles(filename string, files []string) error {
newZipFile, err := os.Create(filename)
if err != nil {
return err
}
defer newZipFile.Close()
zipWriter := zip.NewWriter(newZipFile)
defer zipWriter.Close()
// Add files to zip
for _, file := range files {
if err = AddFileToZip(zipWriter, file); err != nil {
return err
}
}
return nil
}
func AddFileToZip(zipWriter *zip.Writer, filename string) error {
fileToZip, err := os.Open(filename)
if err != nil {
return err
}
defer fileToZip.Close()
// Get the file information
info, err := fileToZip.Stat()
if err != nil {
return err
}
header, err := zip.FileInfoHeader(info)
if err != nil {
return err
}
// Using FileInfoHeader() above only uses the basename of the file. If we want
// to preserve the folder structure we can overwrite this with the full path.
header.Name = filename
// Change to deflate to gain better compression
// see http://golang.org/pkg/archive/zip/#pkg-constants
header.Method = zip.Deflate
writer, err := zipWriter.CreateHeader(header)
if err != nil {
return err
}
_, err = io.Copy(writer, fileToZip)
return err
}
Just use a base name of the file in the zip header.
header.Name = filepath.Base(filename)
^^^^^^^^^^^^^^
Here is a version that does the same thing
package main
import (
"archive/zip"
"io"
"log"
"os"
"path/filepath"
)
func createFlatZip(w io.Writer, files ...string) error {
z := zip.NewWriter(w)
for _, file := range files {
src, err := os.Open(file)
if err != nil {
return err
}
info, err := src.Stat()
if err != nil {
return err
}
hdr, err := zip.FileInfoHeader(info)
if err != nil {
return err
}
hdr.Name = filepath.Base(file) // Write only the base name in the header
dst, err := z.CreateHeader(hdr)
if err != nil {
return err
}
_, err = io.Copy(dst, src)
if err != nil {
return err
}
src.Close()
}
return z.Close()
}
func main() {
if len(os.Args) < 3 {
log.Fatalf("archive name and at least one file are required")
}
a, err := os.Create(os.Args[1])
if err != nil {
log.Fatal(err)
}
defer a.Close()
err = createFlatZip(a, os.Args[2:]...)
if err != nil {
log.Fatal(err)
}
}
Results:
~/src/gozip
➜ go build
~/src/gozip
➜ mkdir test && echo 1 > test/1.txt # create a test file in a subfolder
~/src/gozip
➜ ./gozip 1.zip test/1.txt
~/src/gozip
➜ unzip -l 1.zip
Archive: 1.zip
Length Date Time Name
--------- ---------- ----- ----
2 08-15-2019 01:29 1.txt
--------- -------
2 1 file

Golang: Parse all templates in directory and subdirectories?

This is my directory structure:
app/
template/
layout/
base.tmpl
index.tmpl
template.ParseGlob("*/*.tmpl") parses index.tmpl but not base.tmpl in the layout subdirectory. Is there a way to parse all templates recursively?
Not without implementing your own function to do it, I've been using something like this
func ParseTemplates() *template.Template {
templ := template.New("")
err := filepath.Walk("./views", func(path string, info os.FileInfo, err error) error {
if strings.Contains(path, ".html") {
_, err = templ.ParseFiles(path)
if err != nil {
log.Println(err)
}
}
return err
})
if err != nil {
panic(err)
}
return templ
}
This will parse all your templates then you can render them by calling their names e.g.
template.ExecuteTemplate(w, "home", nil)
if it is not deeply nested (if you know names of sub-directories beforehand) you can just do this:
t := template.Must(template.ParseGlob("template/*.tmpl"))
template.Must(t.ParseGlob("template/layout/*.tmpl"))
Then for each sub-directory do the same as for 'layout'
Datsik's answer has the drawback that there are name collision issues when several directories contain many templates. If two templates in different directories have the same filename it won't work properly: only the second of them will be usable.
This is caused by the implementation of template.ParseFiles, so we can solve it by avoiding template.ParseFiles. Here's a modified walk algorithm that does this by using template.Parse directly instead.
func findAndParseTemplates(rootDir string, funcMap template.FuncMap) (*template.Template, error) {
cleanRoot := filepath.Clean(rootDir)
pfx := len(cleanRoot)+1
root := template.New("")
err := filepath.Walk(cleanRoot, func(path string, info os.FileInfo, e1 error) error {
if !info.IsDir() && strings.HasSuffix(path, ".html") {
if e1 != nil {
return e1
}
b, e2 := ioutil.ReadFile(path)
if e2 != nil {
return e2
}
name := path[pfx:]
t := root.New(name).Funcs(funcMap)
_, e2 = t.Parse(string(b))
if e2 != nil {
return e2
}
}
return nil
})
return root, err
}
This will parse all your templates then you can render them by calling their names e.g.
template.ExecuteTemplate(w, "a/home.html", nil)
You can load multiple subdirectories this way. Here we ignore if subdirectories do not exist. But we want to make sure that the first directory can be loaded.
func ParseTemplates() (*template.Template, error) {
templateBuilder := template.New("")
if t, _ := templateBuilder.ParseGlob("/*/*/*/*/*.tmpl"); t != nil {
templateBuilder = t
}
if t, _ := templateBuilder.ParseGlob("/*/*/*/*.tmpl"); t != nil {
templateBuilder = t
}
if t, _ := templateBuilder.ParseGlob("/*/*/*.tmpl"); t != nil {
templateBuilder = t
}
if t, _ := templateBuilder.ParseGlob("/*/*.tmpl"); t != nil {
templateBuilder = t
}
return templateBuilder.ParseGlob("/*.tmpl")
}
I made a package that solves exactly this problem.
https://github.com/karelbilek/template-parse-recursive
package main
import (
"html/template"
"os"
recurparse "github.com/karelbilek/template-parse-recursive"
)
func main() {
t, err := recurparse.HTMLParse(
template.New("templates"),
"path/to/templates",
"*.html",
)
if err != nil {
panic(err)
}
templateUnder := t.Lookup("subdir/subdir/template.html")
templateUnder.Execute(os.Stdout, nil)
}
With the answer of pete911 I could create this function to parse all the html templates on multiple subdirectories.
// parse.go
func ParseHtmlTemplates() *template.Template {
var directories []string
var filenames []string
// Root directory of template files
root := "./templates"
// Get all directories on /templates and check if there's repeated files
err := filepath.Walk(root, func(path string, info os.FileInfo, err error) error {
if !info.IsDir() {
// Is file
filename := info.Name()
hasRepeatedFiles := contains(filenames, filename)
if hasRepeatedFiles {
return fmt.Errorf("You can't have repeated template files: %s", filename)
}
filenames = append(filenames, filename)
} else {
// Is directory
directories = append(directories, path)
}
return nil
})
if err != nil {
fmt.Println(err)
os.Exit(1)
}
// Create a template for parsing all directories
tmpl := template.Must(template.ParseGlob(root + "/*.html"))
// Parse all directories (minus the root directory)
for _, path := range directories[1:] {
pattern := path + "/*.html"
template.Must(tmpl.ParseGlob(pattern))
}
return tmpl
}
// contains private method
func contains(filenames []string, filename string) bool {
for _, f := range filenames {
if f == filename {
return true
}
}
return false
}
// main.go
func main() {
tmpl = ParseHtmlTemplates()
if err := tmpl.Execute(os.Stdout, ""); err != nil {
panic(err)
}
}

How do I zip a directory containing sub directories or files in Golang?

I know it will have to do with the zip package I just have no idea how I would implement such a thing.
Here's solution that uses Go's built-in recursive file-walker since the top answer so far has implemented their own file-walker:
Aside, some findings I've found while generating zip files today that might save someone else a headache:
The argument to w.Create(zippath) should not start with a "/", and should be relative to the zip root (aka the folder that would be created if you unzipped the archive). So a top-level "manifest.xml" file should be w.Create("manifest.xml"). Nested files should be w.Create("a/b/c.css). If you are generating bad/surprising archives, check first to ensure you aren't breaking this rule. My code does not try to enforce this.
Some specs (like epub) want files in a certain order, but Go's filepath.Walk will crawl in lexical order. (For that matter, I've found all epub parsers to be lenient here so far from Calibre to Books.app on macOS). If you need a specific order, then #LeTigre's solution with ReadDir will let you sort the files at each level of descent.
package main
import (
"archive/zip"
"fmt"
"io"
"os"
"path/filepath"
)
// Zips "./input" into "./output.zip"
func main() {
file, err := os.Create("output.zip")
if err != nil {
panic(err)
}
defer file.Close()
w := zip.NewWriter(file)
defer w.Close()
walker := func(path string, info os.FileInfo, err error) error {
fmt.Printf("Crawling: %#v\n", path)
if err != nil {
return err
}
if info.IsDir() {
return nil
}
file, err := os.Open(path)
if err != nil {
return err
}
defer file.Close()
// Ensure that `path` is not absolute; it should not start with "/".
// This snippet happens to work because I don't use
// absolute paths, but ensure your real-world code
// transforms path into a zip-root relative path.
f, err := w.Create(path)
if err != nil {
return err
}
_, err = io.Copy(f, file)
if err != nil {
return err
}
return nil
}
err = filepath.Walk("input", walker)
if err != nil {
panic(err)
}
}
To do it manually, you could modify the code linked above:
ExampleZipWriter
To give you a simple example, that has many flaws, but might be easily understood:
func ZipWriter() {
baseFolder := "/Users/tom/Desktop/testing/"
// Get a Buffer to Write To
outFile, err := os.Create(`/Users/tom/Desktop/zip.zip`)
if err != nil {
fmt.Println(err)
}
defer outFile.Close()
// Create a new zip archive.
w := zip.NewWriter(outFile)
// Add some files to the archive.
addFiles(w, baseFolder, "")
if err != nil {
fmt.Println(err)
}
// Make sure to check the error on Close.
err = w.Close()
if err != nil {
fmt.Println(err)
}
}
We use this to iterate on files recursively to generate folders too:
func addFiles(w *zip.Writer, basePath, baseInZip string) {
// Open the Directory
files, err := ioutil.ReadDir(basePath)
if err != nil {
fmt.Println(err)
}
for _, file := range files {
fmt.Println(basePath + file.Name())
if !file.IsDir() {
dat, err := ioutil.ReadFile(basePath + file.Name())
if err != nil {
fmt.Println(err)
}
// Add some files to the archive.
f, err := w.Create(baseInZip + file.Name())
if err != nil {
fmt.Println(err)
}
_, err = f.Write(dat)
if err != nil {
fmt.Println(err)
}
} else if file.IsDir() {
// Recurse
newBase := basePath + file.Name() + "/"
fmt.Println("Recursing and Adding SubDir: " + file.Name())
fmt.Println("Recursing and Adding SubDir: " + newBase)
addFiles(w, newBase, baseInZip + file.Name() + "/")
}
}
}
#danneu's answer is good. But if the directory contains empty subdirectories, the code doesn't work and it will ignore them.
So instead of just returning nil, we should create a directory:
if info.IsDir() {
// add a trailing slash for creating dir
path = fmt.Sprintf("%s%c", path, os.PathSeparator)
_, err = w.Create(path)
return err
}
an ez way to zip directory, you can run command of server:
out, err := exec.Command("zip", "-r", "-D", "ideaz.zip", ".idea/*").Output()
https://play.golang.org/p/Mn-HmUjm5q2

Unable to delete an unzipped folder using golang

I wrote code that unzips a file in a particular location then copies the contents of the folder to outside where the folder is unzipped then it removes the folder.
This is the Code I wrote:
package main
import (
"os"
"flag"
"fmt"
"io"
"path/filepath"
"os/exec"
"archive/zip"
"time"
)
func RemoveContents(dir string) error {
d, err := os.Open(dir)
if err != nil {
return err
}
names, err := d.Readdirnames(-1)
if err != nil {
return err
}
for _, name := range names {
err = os.RemoveAll(filepath.Join(dir, name))
if err != nil {
return err
}
}
d.Close()
return nil
}
func CopyFile(source string, dest string) (err error) {
sourcefile, err := os.Open(source)
if err != nil {
return err
}
defer sourcefile.Close()
destfile, err := os.Create(dest)
if err != nil {
return err
}
defer destfile.Close()
_, err = io.Copy(destfile, sourcefile)
if err == nil {
sourceinfo, err := os.Stat(source)
if err != nil {
err = os.Chmod(dest, sourceinfo.Mode())
}
}
return
}
func CopyDir(source string, dest string) (err error) {
// get properties of source dir
sourceinfo, err := os.Stat(source)
if err != nil {
return err
}
// create dest dir
err = os.MkdirAll(dest, sourceinfo.Mode())
if err != nil {
return err
}
directory, _ := os.Open(source)
objects, err := directory.Readdir(-1)
for _, obj := range objects {
sourcefilepointer := source + "/" + obj.Name()
destinationfilepointer := dest + "/" + obj.Name()
if obj.IsDir() {
// create sub-directories - recursively
err = CopyDir(sourcefilepointer, destinationfilepointer)
if err != nil {
fmt.Println(err)
}
} else {
// perform copy
err = CopyFile(sourcefilepointer, destinationfilepointer)
if err != nil {
fmt.Println(err)
}
}
}
return
}
func main() {
flag.Parse() // get the source and destination directory
source_dir := flag.Arg(0) // get the source directory from 1st argument
dest_dir := flag.Arg(1) // get the destination directory from the 2nd argument
os.MkdirAll("E:\\go\\copyDirectory\\myFile.zip",0777)
zipFilePath := "E:\\go\\copyDirectory\\myFile.zip"
tempWrkDir := "E:\\go\\copyDirectory\\"
//Read zip file and get path handle.
fileHandleReader, err := zip.OpenReader(zipFilePath)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
//open zip file and read all the folder and files inside
for _, fileReadHandler := range fileHandleReader.Reader.File {
//read the file or folder handle inside zip
fileOpenHandle, err := fileReadHandler.Open()
if err != nil {
fmt.Println(err)
os.Exit(1)
}
defer fileOpenHandle.Close()
targetUnZipPath := filepath.Join(tempWrkDir, fileReadHandler.Name)
if fileReadHandler.FileInfo().IsDir() {
os.MkdirAll(targetUnZipPath, fileReadHandler.Mode())
//fmt.Println("Creating directory", path)
}else {
// create new dummy file to copy original file.
newTempFileHandle, err := os.OpenFile(targetUnZipPath, os.O_WRONLY|os.O_CREATE, fileReadHandler.Mode())
if err != nil {
fmt.Println(err)
os.Exit(1)
}
defer newTempFileHandle.Close()
//copying original file to dummy file.
if _, err = io.Copy(newTempFileHandle, fileOpenHandle); err != nil {
fmt.Println(err)
os.Exit(1)
}
}
}
time.Sleep(1000*time.Millisecond)
fmt.Println("Source :" + source_dir)
// check if the source dir exist
src, err := os.Stat(source_dir)
if err != nil {
panic(err)
}
if !src.IsDir() {
fmt.Println("Source is not a directory")
os.Exit(1)
}
// create the destination directory
fmt.Println("Destination :"+ dest_dir)
/*_, err = os.Open(dest_dir)
if !os.IsNotExist(err) {
fmt.Println("Destination directory already exists. Abort!")
os.Exit(1)
}*/
err = CopyDir(source_dir, dest_dir)
if err != nil {
fmt.Println(err)
} else {
fmt.Println("Directory copied")
}
err = RemoveContents("./myFiles")
if err != nil {
fmt.Println("ERRR:::",err)
}
//time.Sleep(10000*time.Millisecond)
}
The problem is that everything works fine except for deleting the folder. The folder has only one file in it. The location of the file is as follows:
E:\go\copyDirectory\myfile\mytextfile.txt
The Location of the zip file is as follows:
E:\go\copyDirectory\myfile.zip
The zip file has only one text file. The File inside the zip file is as follows:
E:\go\copyDirectory\myfile.zip\myfile\mytextfile.txt
The error I get is:
ERRR::: remove myfile\mytextfile.txt: The process cannot
access the file because it is being used by another process.
Thanks in advance.
You aren't closing the file. This:
defer newTempFileHandle.Close()
Is run when main finishes, which is after:
err = RemoveContents("./myFiles")
You can wrap that bit of code in an unnamed function:
func() {
//read the file or folder handle inside zip
fileOpenHandle, err := fileReadHandler.Open()
if err != nil {
fmt.Println(err)
os.Exit(1)
}
defer fileOpenHandle.Close()
targetUnZipPath := filepath.Join(tempWrkDir, fileReadHandler.Name)
if fileReadHandler.FileInfo().IsDir() {
os.MkdirAll(targetUnZipPath, fileReadHandler.Mode())
//fmt.Println("Creating directory", path)
} else {
// create new dummy file to copy original file.
newTempFileHandle, err := os.OpenFile(targetUnZipPath, os.O_WRONLY|os.O_CREATE, fileReadHandler.Mode())
if err != nil {
fmt.Println(err)
os.Exit(1)
}
defer newTempFileHandle.Close()
//copying original file to dummy file.
if _, err = io.Copy(newTempFileHandle, fileOpenHandle); err != nil {
fmt.Println(err)
os.Exit(1)
}
}
}()
And then your defer will happen before you try and remove the files. I would recommend pulling this out into a named function though.

Resources