When I lock a file path with flock then check the file existence it return no error even though there is no file in that path. The code follows:
filePath := filepath.Join(r.path, fmt.Sprintf("%s_event.json", eventId))
fileLock := flock.New(filePath)
fileLock.Lock()
defer fileLock.Close()
_, err = os.Stat(filePath)
if err != nil {
if os.IsNotExist(errs) {
return event, EventNotFound{}
}
return
}
But when at first check the Stat then lock the file it works. I need to check it before that. Every idea is welcome!
Related
I'm trying to lock one file when there's one process writing it. When another process tries reading the file, it needs to make sure that no process is writing on it. The idea is that when the write process dies before unlocking the file, and another read process can detect this and deletes this semi-finished file.
To do that, I built such a FileLock structure:
type FileLock struct {
filePath string
f *os.File
}
func (l *FileLock) Lock() error {
if l.f == nil {
f, err := os.OpenFile(l.filePath, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0200)
if err != nil {
return err
}
l.f = f
}
err := syscall.Flock(int(l.f.Fd()), syscall.LOCK_EX|syscall.LOCK_NB)
if err != nil {
return fmt.Errorf("cannot flock file %s - %s", l.filePath, err)
}
return nil
}
func (l *FileLock) Unlock() error {
defer l.f.Close()
return syscall.Flock(int(l.f.Fd()), syscall.LOCK_UN)
}
Before writing to this localfile, I lock it. And unlock when the write is finished:
func downloadFile(response *http.Response, filePath string) error {
output, _ := os.OpenFile(filePath, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0200)
localFileLock := &FileLock{filePath: filePath, f: output}
// Lock the file before writing.
if err := localFileLock.Lock(); err != nil {
return err
}
if _, err := io.Copy(output, response.Body); err != nil {
return fmt.Errorf("problem saving %s to %s: %s", response.Request.URL, filePath, err)
}
// Unlock the file after writing.
if err := localFileLock.Unlock(); err != nil {
return err
}
return nil
}
But for another process, how to check if that file is locked?
Thanks!
when the write process dies before unlocking the file, and another read process can detect this and deletes this semi-finished file.
A process's file descriptors are closed when the process terminates, so if a process terminates with a file locked, its locks are released.¹
So the way to detect and delete an unfinished file is to actually lock it, then delete it with the file descriptor still open and the lock still held.
If you attempt to delete the file without actually acquiring the lock, the deletion may race: another process may have already deleted the original file and recreated and fully written a file with the same name.
¹ man 2 flock:
the lock is released either by an explicit LOCK_UN operation on any of these duplicate file descriptors, or when all such file descriptors have been closed.
I am trying to write in newly generated file in a specific attached volume to my container (a directory)
however i am not sure about the correct the syntax. Below my code:
// Write the certificates to disk
f, _ := os.Create(filepath.Join("/data/certs/", "chamscertificate.pem"))
f.Write(cert)
f.Close()
f, _ = os.Create("key.pem")
f.Write(key)
f.Close()
}
when executing "go run .", i get the "key.pem" but not the "certificate.pem".
You don't check for errors. If the file wasn't created, the information about why the file wasn't created will be in the error return value from os.Create.
f, err := os.Create(filepath.Join("/data/certs/", "chamscertificate.pem"))
if err != nil {
log.Fatal(err)
}
... etc.
Note that f.Write and f.Close also return errors that should be checked.
I need to implement sftp client that connects to a host, read all available files in a specified folder, then check if a particular file matches a pattern and copy it to according local directory. Problem is that i can't find a way to.
I tried to use client.Walk but cannot figure out a way to understand if this is a directory and skip it:
walker := client.Walk(startDir)
for walker.Step() {
if err := walker.Err(); err != nil {
fmt.Fprintln(os.Stderr, err)
continue
}
filePath := walker.Path()
}
How can I determine if the current iteration is directory?
You may use Walker.Stat() to obtain info about the most recent file or directory visited by a call to Walker.Step(). It returns you a value of type os.FileInfo which has an IsDir() method.
For example:
for walker.Step() {
if err := walker.Err(); err != nil {
fmt.Fprintln(os.Stderr, err)
continue
}
if fi := walker.Stat(); fi.IsDir() {
continue // Skip dir
}
// ...
}
I have a Go function that appends a line to a file:
func AppendLine(p string, s string) error {
f, err := os.OpenFile(p, os.O_APPEND|os.O_WRONLY, 0600)
defer f.Close()
if err != nil {
return errors.WithStack(err)
}
_, err = f.WriteString(s + "\n")
return errors.WithStack(err)
}
I'm wondering if the flags os.O_APPEND|os.O_WRONLY make this a safe operation. Is there a guarantee that no matter what happens (even if the process gets shut off in the middle of writing) the existing file contents cannot be deleted?
os package is a wrapper around systems calls so you have guarantees provided by operation system. In this case linux OS guarantees that file opened with O_APPEND flag would be processed atomically http://man7.org/linux/man-pages/man2/open.2.html
I would like to overwrite a symlink using Go but I couldn't find how to do it.
If I try to create the symlink and it already exists an error is returned.
My code:
err := os.Symlink(filePath, symlinkPath)
if err != nil {
fmt.Println(err)
}
I guess the symlink must be removed and then created again. Is that right? If so, how can I unlink the symlink?
Just check that symlink exists and delete it before creating new one
if _, err := os.Lstat(symlinkPath); err == nil {
os.Remove(symlinkPath)
}
Note that #Vadyus's answer hides actual filesystem errors while running lstat. For example, if your disk is broken and Lstat fails, you will still run os.Remove and ignore its error (DANGEROUS, unless you like to debug things for hours).
The following snippets checks for file existence and other errors correctly:
if _, err := os.Lstat(symlinkPath); err == nil {
if err := os.Remove(symlinkPath); err != nil {
return fmt.Errorf("failed to unlink: %+v", err)
}
} else if os.IsNotExist(err) {
return fmt.Errorf("failed to check symlink: %+v", err)
}
The other answers here are correct...but there are two small issues:
There is a tiny data race where the new symlink will be created elsewhere before here but after the delete, leaving it in a potentially inconsistent state.
If the program were to die / crash before the symlink is created but after the previous one is deleted, it may again leave things in an inconsistent state.
The more atomic way to handle this is to create a temporary symlink and then rename it over the original:
symlinkPathTmp := symlinkPath + ".tmp"
if err := os.Remove(symlinkPathTmp); err != nil && !os.IsNotExist(err) {
return err
}
if err := os.Symlink(filePath, symlinkPathTmp); err != nil {
return err
}
if err := os.Rename(symlinkPathTmp, symlinkPath); err != nil {
return err
}
There is still a small race between deleting the temporary link and re-creating it, but it won't risk leaving the primarily link in an inconsistent state. Ideally, we would be able to work around that by using a randomized name for the temporary link, but Go's TempFile always creates a new file, so it isn't quite as useful. (You could maybe call TempFile, then delete the file name and re-use the name, which would be riskier but still safer then just appending a constant .tmp suffix.)
Even with that race however, you still gain the atomic-ness where any interruptions won't result in a missing link.
Note that this is dependent on Posix behavior and may not work on Windows (why would you be using symlinks on Windows anyway?), but it's a technique shared by many macOS/Linux tools that need atomic symlink replacements.