Windows: FlushFileBuffers system call fails on network mapped drive - windows

fsync doc states
Calling fsync() does not necessarily ensure that the entry in the directory containing the file has also reached disk. For that an explicit fsync() on a file descriptor for the directory is
also needed.
I'm trying to sync a directory to the network mapped drive using SMB on windows, similar to what fsync does on Linux.
The following go code snippet which works fine if the directory is stored on a local drive but fails if it is on a network mapped folder.
func main() {
dir := "Z:\\smb-test" // Path to network mapped drive
f, err := openDir(dir)
if err != nil {
log.Fatal(err)
}
// Works fine if the path is located on a local disk but
// fails if the directory is on a network mapped drive
if err := f.Sync(); err != nil {
log.Fatal(err)
}
if err := f.Close(); err != nil {
log.Fatal(err)
}
}
func openDir(path string) (*os.File, error) {
fd, err := openDirWin(path)
if err != nil {
return nil, err
}
return os.NewFile(uintptr(fd), path), nil
}
func openDirWin(path string) (fd syscall.Handle, err error) {
pathp, err := syscall.UTF16PtrFromString(path)
if err != nil {
return syscall.InvalidHandle, err
}
access := uint32(syscall.GENERIC_READ | syscall.GENERIC_WRITE)
sharemode := uint32(syscall.FILE_SHARE_READ | syscall.FILE_SHARE_WRITE)
createmode := uint32(syscall.OPEN_EXISTING)
fl := uint32(syscall.FILE_FLAG_BACKUP_SEMANTICS)
return syscall.CreateFile(pathp, access, sharemode, nil, createmode, fl, 0)
}
The program fails with
Z:\\smb-test Incorrect function.
MSDN states that the handle passed in should either be a handle to a file, or to a volume, but says nothing about directories.
https://learn.microsoft.com/en-us/windows/desktop/api/FileAPI/nf-fileapi-flushfilebuffers
And FlushFileBuffers is not listed as a function that accepts directory handles
https://learn.microsoft.com/en-us/windows/desktop/fileio/obtaining-a-handle-to-a-directory
I also noticed the flushfilebuffers system call fails with Invalid Device Request for a network mapped drive
So the question is how do I sync a directory on windows?
Or is window's buffered I/O fundamentally different from POSIX and we don't need to flush a directory when a file inside it is modified?

The short answer to my question is there's no way to sync directories on windows using system calls.FlushFileBuffers works for files and volumes, NOT Directories.
Read https://social.msdn.microsoft.com/Forums/Windowsdesktop/en-US/847a735b-f21a-4be2-880b-12660e5b98b4/flushfilebuffers-system-call-fails-on-network-mapped-drive and
https://social.msdn.microsoft.com/Forums/windowsdesktop/en-US/0e0b734b-2b73-414d-8833-8d2eed7043f6/sync-directories-on-windows
for more details.

Related

How to execute file created by my program?

I'm trying to execute .ics file that my program just created. Basically, my program is simple CLI calendar app, which generates .ics file. It would be nice if my program would execute this file and add it straight to OS calendar app, without unnecessary searching and executing through OS GUI.
I paste main function to better understanding.
func main() {
serialized, name := cal()
f, err := os.Create(name + ".ics")
if err != nil {
log.Fatal(err)
}
defer f.Close()
_, err2 := f.WriteString(serialized)
if err2 != nil {
log.Fatal(err2)
}
cmd := exec.Command(name + ".ics")
err = cmd.Run()
if err != nil {
log.Fatal(err)
}
}
As it's shown I tried with exec.Command, but it doesnt' work. I was even trying with additional prefixes like ./ or ~/, but it as well didn't work.
Error messages:
fork/exec ./Meeting.ics: permission denied
exec: "Meeting.ics": executable file not found in $PATH
So to sum it up - I want to skip the process where the user has to find a file and open it. I want to make it automatically as a part of my application.
Here's my repository if it would help https://github.com/lenoopaleno/golang-calendar
I'm working on WSL 2, Ubuntu 22.04
Beside the comments above, you might have a problem in your code with the defer f.Close()
The defer runs when the function ends. Until that time your file might or might not be closed and accessible by a different process.
Second you will most likely have to set an execute flag on the a program to run under unix style operating systems.
Program adjustment:
func main() {
serialized, name := cal()
f, err := os.Create(name + ".ics")
if err != nil {
log.Fatal(err)
}
_, err2 := f.WriteString(serialized)
if err2 != nil {
log.Fatal(err2)
}
f.Sync()
f.Close()
exec.Command(`chmod +x `+name+".ics").Run() // This can be done prettier
cmd := exec.Command(name + ".ics")
err = cmd.Run()
if err != nil {
log.Fatal(err)
}
}

How to extract .7z files in Go

I have a 7z archive of a number of .txt files. I am trying to list all the files in the archive and upload them to an s3 bucket. But I'm having trouble with extracting .7z archives on Go. To do this, I found a package github.com/gen2brain/go-unarr (imported as extractor) and this is what I have so far
content, err := ioutil.ReadFile("sample_archive.7z")
if err != nil {
fmt.Printf("err: %+v", err)
}
a, err := extractor.NewArchiveFromMemory(content)
if err != nil {
fmt.Printf("err: %+v", err)
}
lst, _ := a.List()
fmt.Printf("lst: %+v", last)
This prints a list of all the files in the archive. But this has two issues.
It reads files from local using ioutil and the input of NewArchiveFromMemory must be of type []byte. But I can't read from local and will have to use a file from memory of type os.file. So I will either have to find a different method or convert the os.file to []byte. There's another method NewArchiveFromReader(r io.Reader). But this is returning an error saying Bad File Descriptor.
file, err := os.OpenFile(
path,
os.O_WRONLY|os.O_TRUNC|os.O_CREATE,
0666,
)
a, err := extractor.NewArchiveFromReader(file)
if err != nil {
fmt.Printf("ERROR: %+v", err)
}
lst, _ := a.List()
fmt.Printf("files: %+v\n", lst)
I am able to get the list of the files in the archive. And using Extract(destinaltion_path string), I can also extract it to a local directory. But I want the extracted files also in os.file format ( ie. a list of os.file since there will be multiple files ).
How can I change my current code to achieve both the above targets? Is there any other library to do this?
os.File implements the io.Reader interface (because it has a Read([]byte) (int, error) method defined), so you can use NewArchiveFromReader(file) without any conversions needed. You can read up on Go interfaces for more background on why that works.
If you're okay with extracting to a local directory, you can do that and then read the files back in (warning, may contain typos):
func extractAndOpenAll(*extractor.Archive) ([]*os.File, error) {
err := a.Extract("/tmp/path") // consider using ioutil.TempDir()
if err != nil {
return nil, err
}
filestats, err := ioutil.ReadDir("/tmp/path")
if err != nil {
return nil, err
}
# warning: all these file handles must be closed by the caller,
# which is why even the error case here returns the list of files.
# if you forget, your process might leak file handles.
files := make([]*os.File, 0)
for _, fs := range(filestats) {
file, err := os.Open(fs.Name())
if err != nil {
return files, err
}
files = append(files, file)
}
return files, nil
}
It is possible to use the archived files without writing back to disk (https://github.com/gen2brain/go-unarr#read-all-entries-from-archive), but whether or not you should do that instead depends on what your next step is.

Fprintf Error when writing to network mapped drive

I have the following piece of code which creates an output file on a local drive and required to do the same on a network mapped drive let's call it [H:].
The file name (full path name) entered from command line as argument[1].
I am using Windows 10/Server 2016
// The following will create and append to the file when required.
sourcefile, errf := os.OpenFile(os.Args[1], s.O_CREATE|os.O_APPEND|os.O_RDWR, 0666)
if erro != nil {
panic(erro)
}
defer outfile.Close()
I use the following function to write a map into this file.
func map2Linpro(inp map[string][]string, outfile io.Writer) {
for k, v := range inp {
_, err := fmt.Fprintf(outfile, "%s %s=%s %s\n", v[0], k, v[1], v[2])
if err != nil {
fmt.Println("Error Writing to File: ", err)
}
}
}
Everything is working just fine if the output file is on the local Drive, but when using full path with the Mapped Drive letter, I received the following error:
Error: write h://00_sc//dest01.txt: The parameter is incorrect.
I searched for any reason, but could not find one.
I would appreciate if someone help
The following is the Error I got after adding Panic(erro) after OpenFile.
Which proves that the error source is fmt.Fprintf
Error Writing to File: write H:/00_sc/dest01.txt: The parameter is incorrect.
Thanks to all.
outfile, _ := os.OpenFile(os.Args[2], os.O_CREATE|os.O_APPEND, 0666)
should read
outfile, err := os.OpenFile(os.Args[2], os.O_CREATE|os.O_APPEND, 0666)
if err != nil {
panic(err)
}
rewrite those lines and the resulting error message should give a clue as to the cause

Is there a way to determine if current step is a directory?

I need to implement sftp client that connects to a host, read all available files in a specified folder, then check if a particular file matches a pattern and copy it to according local directory. Problem is that i can't find a way to.
I tried to use client.Walk but cannot figure out a way to understand if this is a directory and skip it:
walker := client.Walk(startDir)
for walker.Step() {
if err := walker.Err(); err != nil {
fmt.Fprintln(os.Stderr, err)
continue
}
filePath := walker.Path()
}
How can I determine if the current iteration is directory?
You may use Walker.Stat() to obtain info about the most recent file or directory visited by a call to Walker.Step(). It returns you a value of type os.FileInfo which has an IsDir() method.
For example:
for walker.Step() {
if err := walker.Err(); err != nil {
fmt.Fprintln(os.Stderr, err)
continue
}
if fi := walker.Stat(); fi.IsDir() {
continue // Skip dir
}
// ...
}

how to get the location of the current file in revel

I am using golang revel web framework and
I am trying to create a sqlite db in the current working directory.
model.go
func New(dbName string,table string) *Db {
_,filename,_,_ := runtime.Caller(1)
db , err := sql.Open("sqlite3",path.Join(path.Dir(filename),dbName))
if err != nil {
log.Fatal(err)
}
err = db.Ping()
if err != nil {
log.Panic(err)
}
database := &Db{Database:db}
_,err = db.Exec("create table %s" +
"( id integer primary key, " +
"name varchar(100),"+
"email varchar(100),"+
"branch varchar(100),"+
"help varchar(100)",)
if err != nil {
log.Fatal(err)
}
}
I have a test in place which just calls this function.
whenever i run the test using revel test or by going to the localhost:9000/#tests, the function Panics and the error message is
cannot open the database file.
The reason that is happening is because the filename returned by runtime.Caller(1) is /usr/local/go/src/runtime/asm_amd64.s for which the program has no permission.
if i directly write ./foo.db, even then the error shows.
I tried os.Getwd() which return empty string.
I also tried filepath.Abs(filepath.Dir(os.Args[0]))
but that returned /home/girish/GoProjects/bin/revel.d which is the revel binary.
So whats the best way to find the directory of the model.go?
It doesn't make sense to get the directory of the model.go file at runtime, because the compiled executable could be on a completely different filesystem.
You may want to get the directory of where the running executable was started from:
dir, err := filepath.Abs(filepath.Dir(os.Args[0]))
dir will be the folder where the program lives at runtime.

Resources