Refactor function that reading files from folders tree - go

I have the following structure of my project like that:
main.go
/data
/1
1.json
/2
2.json
/3
3.json
And I have the following function that works well
const (
data = "./data"
)
type Hello struct {
S string `json:"hello"`
}
func myFunc() {
folders, err := os.ReadDir(data)
if err != nil {
log.Fatalf("read dir: %s", err)
}
for _, folder := range folders {
path := filepath.Join(data, folder.Name())
files, err := os.ReadDir(path)
if err != nil {
log.Fatalf("read dir: %s", err)
}
for _, file := range files {
pathToFile := filepath.Join(path, file.Name())
b, err := os.ReadFile(pathToFile)
if err != nil {
log.Println(err)
}
h := Hello{}
err = json.Unmarshal(b, &h)
if err != nil {
log.Printf("unmarshall erro: %w\n", err)
}
fmt.Println(h)
}
}
}
I should refactor my data tree and divide data folder into two additional folders. So my structure should be like this:
main.go
/data
/dog
/1
1.json
/2
2.json
/cat
/3
3.json
So I need to fix my function according to this structure of the files.
I don't want to hardcode and make the solution with additional for loop. But I think it is not good solution.
func myfunc() {
subFolders, err := os.ReadDir(data)
if err != nil {
log.Fatalf("read dir: %s", err)
}
for _, subFolder := range subFolders {
pathToSubFolder := filepath.Join(data, subFolder.Name())
folders, err := os.ReadDir(pathToSubFolder)
if err != nil {
log.Fatalf("read dir1: %s", err)
}
for _, folder := range folders {
path := filepath.Join(pathToSubFolder, folder.Name())
files, err := os.ReadDir(path)
if err != nil {
log.Fatalf("read dir2: %s", err)
}
for _, file := range files {
pathToFile := filepath.Join(path, file.Name())
b, err := os.ReadFile(pathToFile)
if err != nil {
log.Println(err)
}
h := Hello{}
err = json.Unmarshal(b, &h)
if err != nil {
log.Printf("unmarshall erro: %w\n", err)
}
fmt.Println(h)
}
}
}
}
This soultion works but can anyone suggest another solution that could be better?
Thanks in advance

Related

Unzipping from a shared smb folder golang

I'm using the code below to iterate through a folder (unzipping .zip files locally, performing actions on them etc).
func extractZipWithPassword(localPath string, pathToServer string) {
filepath.Walk(localPath, func(path string, info os.FileInfo, err error) error {
extension := filepath.Ext(path)
fileName := filepath.Base(path)
if extension == ".zip" {
zip_password := "password"
outDirZIP := filepath.Dir(path) + "/" + strings.TrimSuffix(fileName, extension)
os.Mkdir(outDirZIP, 0777)
commandString := fmt.Sprintf(`unzip -P %s %s -d %s`, zip_password, path, outDirZIP)
commandSlice := strings.Fields(commandString)
fmt.Println(commandString)
c := exec.Command(commandSlice[0], commandSlice[1:]...)
c.Run()
zipCount++
iterateThroughPE(outDirZIP)
parsePKCS7(outDirZIP)
splitter(start_string, end_string, outDirZIP)
return nil
} else {
fmt.Println("")
return nil
}
})
}
However now I need to extract archives from an SMB folder that requires login+password combination to access.
I tried using this lib github.com/hirochachacha/go-smb2 to access the shared folder, but to no success.
Is there a way to straight up unzip .zip files to a local folder of my choice with Go?
If there's not, what are my options?
UPDATE
Got the share mounted with go-smb2, not sure how to proceed next.
func listShares(path string) {
conn, err := net.Dial("tcp", "ip:port")
if err != nil {
panic(err)
}
defer conn.Close()
d := &smb2.Dialer{
Initiator: &smb2.NTLMInitiator{
User: "user",
Password: "pw",
},
}
s, err := d.Dial(conn)
if err != nil {
panic(err)
}
defer s.Logoff()
names, err := s.ListSharenames()
if err != nil {
panic(err)
}
for _, name := range names {
fmt.Println(name)
}
fs, err := s.Mount("sharename")
if err != nil {
panic(err)
}
defer fs.Umount()
}

Go SMB2 , reading and listing directory

I'm trying to list directory contents on a share from Linux to Windows using Go.
So far I've managed to Create/Remove new files inside a share with full Read/Write permissions.
Go module: https://godoc.org/github.com/hirochachacha/go-smb2#Client
Functions:
func connect_client(host string, share string, session map[string]string) *smb2.Client {
//Checks for a connection on port
conn, err := net.Dial("tcp", host+":445")
if err != nil {
panic(err)
}
//smb auth
d := &smb2.Dialer{
Initiator: &smb2.NTLMInitiator{
User: session["Username"],
Password: session["Password"],
Domain: session["Domain"],
},
}
//Returns a client session
client, err := d.Dial(conn)
if err != nil {
fmt.Println("Connection failed")
client.Logoff()
} else {
fmt.Println("Connection Succeeded")
}
return client
}
func check_write(host string, client *smb2.Client) {
file := "asdasdas.txt"
fs, err := client.Mount(host)
if err != nil {
fmt.Println(err)
os.Exit(0)
}
defer fs.Umount()
share := strings.Split(host, `\\`)
f, err := fs.Create(file)
if err != nil {
fmt.Println("You do not have write permissions on directory:%s ! \n", strings.Split(share[1], `\`)[1])
os.Exit(0)
}
defer fs.Remove(file)
defer f.Close()
fmt.Printf("You have write permissions to directory: %s \n", strings.Split(share[1], `\`)[1]))
}
func list_all(client *smb2.Client, host string) {
fs, err := client.Mount(host)
if err != nil {
fmt.Println(err)
os.Exit(0)
}
defer fs.Umount()
_, err = fs.Open(`Test.txt`)
if err != nil {
fmt.Println(err)
os.Exit(0)
}
}
func main() {
host, share, action, session := get_flags()
client := connect_client(host, share, session)
full_host := `\\` + host + `\` + share
//File create
if action == "check_write" {
check_write(full_host, client)
}
if action == "list_files" {
list_all(client, full_host)
}
}
In the function list_all() everything works, but when I am trying to access \\192.168.1.19\Sharing only..
When I input just a host with directory name it seas it can not list the directory path because it can not find the object specified.
I can't understand how I can get pointer used for *RemoteFile in order to use the functions:
f.Readdir()
f.Name()
etc....
So far I managed to use *RemoteFileSystem only for all other actions but I want to list all contents of the directory..
Help would be much appreciated!
Edit:
If it wasn't clear enough, in order to use functions like:
f.Readdir()
f.Name()
I need to get a pointer for *RemoteFile, this is my main issue
https://godoc.org/github.com/hirochachacha/go-smb2#RemoteFileSystem.Open
Use Open on a RemoteFileSystem and either a directory name, or empty string for the directory at the root of the filesystem.
e.g.
client, err := d.Dial(conn)
if err != nil {
return err
}
rfs, err := client.Mount("jrwren")
if err != nil {
return err
}
// cat the NOTES file.
f, err := rfs.Open("NOTES")
if err != nil {
return err
}
defer f.Close()
io.Copy(os.Stdout, f)
// List all the files
dir, err := rfs.Open("")
if err != nil {
return err
}
fis, err := dir.Readdir(10)
if err != nil {
return err
}
for i := range fis {
fmt.Println(fis[i].Name())
}
func list_all(client *smb2.Client, host string) {
fs, err := client.Mount(host)
if err != nil {
fmt.Println(err)
os.Exit(0)
}
dir, err := fs.Open("")
if err != nil {
fmt.Println(err)
os.Exit(0)
}
fis, err := dir.Readdir(-1)
if err != nil {
fmt.Println(err)
os.Exit(0)
}
for i := range fis {
fmt.Println(fis[i].Name())
}
}
This would be the answer thank you!

Making a filename truncator in go

I am new to Go, and I've been trying to make a program that truncates filenames after (and including) the character "-" of all files in its directory. It's not working, and I don't know where I'm going wrong.
func changeFilename() {
file, err := os.Open(".")
if err != nil {
log.Fatalf("failed opening directory: %s", err)
}
defer file.Close()
oldNames, _ := file.Readdirnames(0)
var s string
for _, i := range oldNames {
for _, j := range i {
for j != '-' {
k := strconv.QuoteRune(j)
s += k
j++
}
}
err := os.Rename(i, s)
if err != nil {
log.Fatal("failed to rename: %s", err)
}
}
}
I expected filenames in the test directory to be changed when I ran the executable, but they didn't. The program builds without any error messages.
Fixing your logic to correspond to your description of the problem:
package main
import (
"fmt"
"log"
"os"
"path/filepath"
)
// changeFilename truncates filenames after (and including) the character "-"
// of all files in the directory. File extensions are preserved.
func changeFilename(dir string) {
file, err := os.Open(dir)
if err != nil {
log.Fatalf("failed opening directory: %s", err)
}
defer file.Close()
fis, err := file.Readdir(0)
if err != nil {
log.Fatalf("failed reading directory: %s", err)
}
for _, fi := range fis {
if !fi.Mode().IsRegular() {
continue
}
old := fi.Name()
ext := filepath.Ext(old)
for i, r := range old[:len(old)-len(ext)] {
if r == '-' {
new := old[:i] + ext
err := os.Rename(old, new)
if err != nil {
log.Fatalf("failed to rename: %s", err)
}
fmt.Printf("%q %q\n", old, new)
break
}
}
}
}
func main() {
changeFilename(".")
}
Output:
"trunc-ate.typ" "trunc.typ"
"hyp-hen.ext" "hyp.ext"
Your s variable should be moved into the outermost loop since its lifetime should be tied to a single oldName. The innermost loop has no exit condition once a non - character is encountered. That should be an if statement with a corresponding break if a - is found. As for building up the new filename, you ought to use += string(j) instead of quoting the rune.
func changeFilename() {
file, err := os.Open(".")
if err != nil {
log.Fatalf("failed opening directory: %s", err)
}
defer file.Close()
oldNames, _ := file.Readdirnames(0)
for _, i := range oldNames {
var s string
for _, j := range i {
if j != '-' {
s += string(j)
} else {
break
}
}
err := os.Rename(i, s)
if err != nil {
log.Fatal("failed to rename: %s", err)
}
}
}

Move a file to a different drive with Go

I'm trying to move a file from my C-drive to my H-drive using os.Replace().
The code looks as follows:
func MoveFile(source string, destination string) {
err := os.Rename(source, destination)
if err != nil {
fmt.Println(err)
}
}
However, when I run the code I get the following error:
rename C:\old\path\to\file.txt H:\new\path\to\file.txt: The system cannot move the file to a different disk drive.
I found this issue on GitHub that specifies the problem but it appears that they will not change this function to allow it to move file on different disk drives.
I already searched for other possibilities to move files, but found nothing in the standard documentation or the internet.
So, what should I do now to be able to move files on different disk drives?
As the comment said, you'll need to create a new file on the other disk, copy the contents, and then remove the original. It's straightforward using os.Create, io.Copy, and os.Remove:
import (
"fmt"
"io"
"os"
)
func MoveFile(sourcePath, destPath string) error {
inputFile, err := os.Open(sourcePath)
if err != nil {
return fmt.Errorf("Couldn't open source file: %s", err)
}
outputFile, err := os.Create(destPath)
if err != nil {
inputFile.Close()
return fmt.Errorf("Couldn't open dest file: %s", err)
}
defer outputFile.Close()
_, err = io.Copy(outputFile, inputFile)
inputFile.Close()
if err != nil {
return fmt.Errorf("Writing to output file failed: %s", err)
}
// The copy was successful, so now delete the original file
err = os.Remove(sourcePath)
if err != nil {
return fmt.Errorf("Failed removing original file: %s", err)
}
return nil
}
You need to make sure that you handle all cases on both Linux and Windows. For example, for any size file,
package main
import (
"flag"
"fmt"
"io"
"os"
)
func MoveFile(source, destination string) (err error) {
src, err := os.Open(source)
if err != nil {
return err
}
defer src.Close()
fi, err := src.Stat()
if err != nil {
return err
}
flag := os.O_WRONLY | os.O_CREATE | os.O_TRUNC
perm := fi.Mode() & os.ModePerm
dst, err := os.OpenFile(destination, flag, perm)
if err != nil {
return err
}
defer dst.Close()
_, err = io.Copy(dst, src)
if err != nil {
dst.Close()
os.Remove(destination)
return err
}
err = dst.Close()
if err != nil {
return err
}
err = src.Close()
if err != nil {
return err
}
err = os.Remove(source)
if err != nil {
return err
}
return nil
}
func main() {
var src, dst string
flag.StringVar(&src, "src", "", "source file")
flag.StringVar(&dst, "dst", "", "destination file")
flag.Parse()
if src == "" || dst == "" {
flag.Usage()
os.Exit(1)
}
err := MoveFile(src, dst)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Printf("moved %q to %q\n", src, dst)
}
Output (Linux):
$ cp move.file src.file && go build movefile.go && ./movefile -src=src.file -dst=dst.file
moved "src.file" to "dst.file"
$
Output (Windows):
>copy /Y move.file src.file && go build movefile.go && movefile -src=src.file -dst=dst.file
moved "src.file" to "dst.file"
>
This solution Moves the file and preserves permissions:
func MoveFile(src, dst string) error {
in, err := os.Open(src)
if err != nil {
return fmt.Errorf("Couldn't open source file: %s", err)
}
out, err := os.Create(dst)
if err != nil {
in.Close()
return fmt.Errorf("Couldn't open dest file: %s", err)
}
defer out.Close()
_, err = io.Copy(out, in)
in.Close()
if err != nil {
return fmt.Errorf("Writing to output file failed: %s", err)
}
err = out.Sync()
if err != nil {
return fmt.Errorf("Sync error: %s", err)
}
si, err := os.Stat(src)
if err != nil {
return fmt.Errorf("Stat error: %s", err)
}
err = os.Chmod(dst, si.Mode())
if err != nil {
return fmt.Errorf("Chmod error: %s", err)
}
err = os.Remove(src)
if err != nil {
return fmt.Errorf("Failed removing original file: %s", err)
}
return nil
}
If only want to Copy the file without remove the original:
func CopyFile(src, dst string) error {
in, err := os.Open(src)
if err != nil {
return fmt.Errorf("Couldn't open source file: %s", err)
}
out, err := os.Create(dst)
if err != nil {
in.Close()
return fmt.Errorf("Couldn't open dest file: %s", err)
}
defer out.Close()
_, err = io.Copy(out, in)
in.Close()
if err != nil {
return fmt.Errorf("Writing to output file failed: %s", err)
}
err = out.Sync()
if err != nil {
return fmt.Errorf("Sync error: %s", err)
}
si, err := os.Stat(src)
if err != nil {
return fmt.Errorf("Stat error: %s", err)
}
err = os.Chmod(dst, si.Mode())
if err != nil {
return fmt.Errorf("Chmod error: %s", err)
}
return nil
}
Maybe you can use a magic approach, just using the syscall.MoveFile as follows.
func main() {
oldpath := "D:\\black.txt"
newpath := "E:\\black-new.txt"
from, _ := syscall.UTF16PtrFromString(oldpath)
to, _ := syscall.UTF16PtrFromString(newpath)
fmt.Println(*from, *to)
err := syscall.MoveFile(from, to)
if err != nil {
panic(err)
}
}
the program works.
if you want a cross-platform compatibility program, you can implement your own MoveFile.
func MoveFile(src string, dst string) error {
if runtime.GOOS == "windows" {
from, _ := syscall.UTF16PtrFromString(src)
to, _ := syscall.UTF16PtrFromString(dst)
return syscall.MoveFile(from, to)
} else {
return os.Rename(src, dst)
}
}

Unable to delete an unzipped folder using golang

I wrote code that unzips a file in a particular location then copies the contents of the folder to outside where the folder is unzipped then it removes the folder.
This is the Code I wrote:
package main
import (
"os"
"flag"
"fmt"
"io"
"path/filepath"
"os/exec"
"archive/zip"
"time"
)
func RemoveContents(dir string) error {
d, err := os.Open(dir)
if err != nil {
return err
}
names, err := d.Readdirnames(-1)
if err != nil {
return err
}
for _, name := range names {
err = os.RemoveAll(filepath.Join(dir, name))
if err != nil {
return err
}
}
d.Close()
return nil
}
func CopyFile(source string, dest string) (err error) {
sourcefile, err := os.Open(source)
if err != nil {
return err
}
defer sourcefile.Close()
destfile, err := os.Create(dest)
if err != nil {
return err
}
defer destfile.Close()
_, err = io.Copy(destfile, sourcefile)
if err == nil {
sourceinfo, err := os.Stat(source)
if err != nil {
err = os.Chmod(dest, sourceinfo.Mode())
}
}
return
}
func CopyDir(source string, dest string) (err error) {
// get properties of source dir
sourceinfo, err := os.Stat(source)
if err != nil {
return err
}
// create dest dir
err = os.MkdirAll(dest, sourceinfo.Mode())
if err != nil {
return err
}
directory, _ := os.Open(source)
objects, err := directory.Readdir(-1)
for _, obj := range objects {
sourcefilepointer := source + "/" + obj.Name()
destinationfilepointer := dest + "/" + obj.Name()
if obj.IsDir() {
// create sub-directories - recursively
err = CopyDir(sourcefilepointer, destinationfilepointer)
if err != nil {
fmt.Println(err)
}
} else {
// perform copy
err = CopyFile(sourcefilepointer, destinationfilepointer)
if err != nil {
fmt.Println(err)
}
}
}
return
}
func main() {
flag.Parse() // get the source and destination directory
source_dir := flag.Arg(0) // get the source directory from 1st argument
dest_dir := flag.Arg(1) // get the destination directory from the 2nd argument
os.MkdirAll("E:\\go\\copyDirectory\\myFile.zip",0777)
zipFilePath := "E:\\go\\copyDirectory\\myFile.zip"
tempWrkDir := "E:\\go\\copyDirectory\\"
//Read zip file and get path handle.
fileHandleReader, err := zip.OpenReader(zipFilePath)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
//open zip file and read all the folder and files inside
for _, fileReadHandler := range fileHandleReader.Reader.File {
//read the file or folder handle inside zip
fileOpenHandle, err := fileReadHandler.Open()
if err != nil {
fmt.Println(err)
os.Exit(1)
}
defer fileOpenHandle.Close()
targetUnZipPath := filepath.Join(tempWrkDir, fileReadHandler.Name)
if fileReadHandler.FileInfo().IsDir() {
os.MkdirAll(targetUnZipPath, fileReadHandler.Mode())
//fmt.Println("Creating directory", path)
}else {
// create new dummy file to copy original file.
newTempFileHandle, err := os.OpenFile(targetUnZipPath, os.O_WRONLY|os.O_CREATE, fileReadHandler.Mode())
if err != nil {
fmt.Println(err)
os.Exit(1)
}
defer newTempFileHandle.Close()
//copying original file to dummy file.
if _, err = io.Copy(newTempFileHandle, fileOpenHandle); err != nil {
fmt.Println(err)
os.Exit(1)
}
}
}
time.Sleep(1000*time.Millisecond)
fmt.Println("Source :" + source_dir)
// check if the source dir exist
src, err := os.Stat(source_dir)
if err != nil {
panic(err)
}
if !src.IsDir() {
fmt.Println("Source is not a directory")
os.Exit(1)
}
// create the destination directory
fmt.Println("Destination :"+ dest_dir)
/*_, err = os.Open(dest_dir)
if !os.IsNotExist(err) {
fmt.Println("Destination directory already exists. Abort!")
os.Exit(1)
}*/
err = CopyDir(source_dir, dest_dir)
if err != nil {
fmt.Println(err)
} else {
fmt.Println("Directory copied")
}
err = RemoveContents("./myFiles")
if err != nil {
fmt.Println("ERRR:::",err)
}
//time.Sleep(10000*time.Millisecond)
}
The problem is that everything works fine except for deleting the folder. The folder has only one file in it. The location of the file is as follows:
E:\go\copyDirectory\myfile\mytextfile.txt
The Location of the zip file is as follows:
E:\go\copyDirectory\myfile.zip
The zip file has only one text file. The File inside the zip file is as follows:
E:\go\copyDirectory\myfile.zip\myfile\mytextfile.txt
The error I get is:
ERRR::: remove myfile\mytextfile.txt: The process cannot
access the file because it is being used by another process.
Thanks in advance.
You aren't closing the file. This:
defer newTempFileHandle.Close()
Is run when main finishes, which is after:
err = RemoveContents("./myFiles")
You can wrap that bit of code in an unnamed function:
func() {
//read the file or folder handle inside zip
fileOpenHandle, err := fileReadHandler.Open()
if err != nil {
fmt.Println(err)
os.Exit(1)
}
defer fileOpenHandle.Close()
targetUnZipPath := filepath.Join(tempWrkDir, fileReadHandler.Name)
if fileReadHandler.FileInfo().IsDir() {
os.MkdirAll(targetUnZipPath, fileReadHandler.Mode())
//fmt.Println("Creating directory", path)
} else {
// create new dummy file to copy original file.
newTempFileHandle, err := os.OpenFile(targetUnZipPath, os.O_WRONLY|os.O_CREATE, fileReadHandler.Mode())
if err != nil {
fmt.Println(err)
os.Exit(1)
}
defer newTempFileHandle.Close()
//copying original file to dummy file.
if _, err = io.Copy(newTempFileHandle, fileOpenHandle); err != nil {
fmt.Println(err)
os.Exit(1)
}
}
}()
And then your defer will happen before you try and remove the files. I would recommend pulling this out into a named function though.

Resources