I'm trying to lock one file when there's one process writing it. When another process tries reading the file, it needs to make sure that no process is writing on it. The idea is that when the write process dies before unlocking the file, and another read process can detect this and deletes this semi-finished file.
To do that, I built such a FileLock structure:
type FileLock struct {
filePath string
f *os.File
}
func (l *FileLock) Lock() error {
if l.f == nil {
f, err := os.OpenFile(l.filePath, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0200)
if err != nil {
return err
}
l.f = f
}
err := syscall.Flock(int(l.f.Fd()), syscall.LOCK_EX|syscall.LOCK_NB)
if err != nil {
return fmt.Errorf("cannot flock file %s - %s", l.filePath, err)
}
return nil
}
func (l *FileLock) Unlock() error {
defer l.f.Close()
return syscall.Flock(int(l.f.Fd()), syscall.LOCK_UN)
}
Before writing to this localfile, I lock it. And unlock when the write is finished:
func downloadFile(response *http.Response, filePath string) error {
output, _ := os.OpenFile(filePath, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0200)
localFileLock := &FileLock{filePath: filePath, f: output}
// Lock the file before writing.
if err := localFileLock.Lock(); err != nil {
return err
}
if _, err := io.Copy(output, response.Body); err != nil {
return fmt.Errorf("problem saving %s to %s: %s", response.Request.URL, filePath, err)
}
// Unlock the file after writing.
if err := localFileLock.Unlock(); err != nil {
return err
}
return nil
}
But for another process, how to check if that file is locked?
Thanks!
when the write process dies before unlocking the file, and another read process can detect this and deletes this semi-finished file.
A process's file descriptors are closed when the process terminates, so if a process terminates with a file locked, its locks are released.¹
So the way to detect and delete an unfinished file is to actually lock it, then delete it with the file descriptor still open and the lock still held.
If you attempt to delete the file without actually acquiring the lock, the deletion may race: another process may have already deleted the original file and recreated and fully written a file with the same name.
¹ man 2 flock:
the lock is released either by an explicit LOCK_UN operation on any of these duplicate file descriptors, or when all such file descriptors have been closed.
Related
When I lock a file path with flock then check the file existence it return no error even though there is no file in that path. The code follows:
filePath := filepath.Join(r.path, fmt.Sprintf("%s_event.json", eventId))
fileLock := flock.New(filePath)
fileLock.Lock()
defer fileLock.Close()
_, err = os.Stat(filePath)
if err != nil {
if os.IsNotExist(errs) {
return event, EventNotFound{}
}
return
}
But when at first check the Stat then lock the file it works. I need to check it before that. Every idea is welcome!
I am currently learning Go and I am trying to send the contents of a directory to another machine over a plain tcp connection using Go's net package.
It works fine with individual files and small folders, but I run into issues if the folder contains many subfolders and larger files. I am using the filepath.Walk function to traverse over all files in the given directory. For each file or directory I send, I also send a header that provides the receiver with file name, file size, isDir properties so I know for how long I need to read for when reading the content. The issue I am having is that after a while when reading the header, I am reading actual file content of the previous file even though I already read that file from the connection
Here is the writer side. I simply traverse over the directory.
func transferDir(session *Session, dir string) error {
return filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
header := Header{Name: info.Name(), Size: info.Size(), Path: path}
if info.IsDir() {
header.SetDirBit()
session.WriteHeader(header)
return nil // nothing more to write
}
// content is a file. write the file now byte by byte
file, err := os.Open(path)
inf, err := file.Stat()
header.Size = inf.Size() // get the true size of the file
session.WriteHeader(header)
defer file.Close()
if err != nil {
return err
}
buf := make([]byte, BUF_SIZE)
for {
n, err := file.Read(buf)
if err != nil {
if err == io.EOF {
session.Write(buf[:n])
session.Flush()
break
} else {
log.Println(err)
return err
}
}
session.Write(buf[:n])
session.Flush()
}
return nil
})
And here is the reader part
func (c *Clone) readFile(h Header) error {
file, err := os.Create(h.Path)
defer file.Close()
if err != nil {
return err
}
var receivedByts int64
fmt.Printf("Reading File: %s Size: %d\n", h.Name, h.Size)
for {
if (h.Size - receivedByts) < BUF_SIZE {
n, err := io.CopyN(file, c.sesh, (h.Size - receivedByts))
fmt.Println("Written: %d err: %s\n", n, err)
break
}
n, err := io.CopyN(file, c.sesh, BUF_SIZE)
fmt.Println("Written: %d err: %s\n", n, err)
receivedByts += BUF_SIZE
fmt.Println("Bytes Read: ", receivedByts)
}
return nil
}
Now the weird part is that when I am looking at the print statements I see something like:
Reading File: test.txt Size: 14024
Written 1024 nil
Bytes Read 1024
... This continues all the way to the break statement
And the total of the Bytes read equals the actual file size. Yet, the subsequent read for the header will return content from the test.txt file. Almost like there is still stuff in the buffer, but I think I read it already....
I have a Go function that appends a line to a file:
func AppendLine(p string, s string) error {
f, err := os.OpenFile(p, os.O_APPEND|os.O_WRONLY, 0600)
defer f.Close()
if err != nil {
return errors.WithStack(err)
}
_, err = f.WriteString(s + "\n")
return errors.WithStack(err)
}
I'm wondering if the flags os.O_APPEND|os.O_WRONLY make this a safe operation. Is there a guarantee that no matter what happens (even if the process gets shut off in the middle of writing) the existing file contents cannot be deleted?
os package is a wrapper around systems calls so you have guarantees provided by operation system. In this case linux OS guarantees that file opened with O_APPEND flag would be processed atomically http://man7.org/linux/man-pages/man2/open.2.html
Consider the following code snippet:
func a(fd int) {
file := os.NewFile(uintptr(fd), "")
defer func() {
if err := file.Close(); err != nil {
fmt.Printf("%v", err)
}
}
This piece of code is legit, and will work OK. Files will be closed upon returning from a()
However, The following will not work correctly:
func a(fd int) {
file := os.NewFile(uintptr(fd), "")
defer func() {
if err := syscall.Close(int(file.Fd()); err != nil {
fmt.Printf("%v", err)
}
}
The error that will be received, occasionally, will be bad file descriptor, due to the fact of NewFile setting a finalizer
which, during garbage collection, will close the file itself.
Whats unclear to me, is that the deferred function still has a reference to the file, so theoretically, it shouldn't be garbage collected yet.
So why is golang runtime behaves that way?
the problems of the code is after file.Fd() return, file is unreachable, so file may be close by the finalizer(garbage collected).
according to runtime.SetFinalizer:
For example, if p points to a struct that contains a file descriptor d, and p has a finalizer that closes that file descriptor, and if the last use of p in a function is a call to syscall.Write(p.d, buf, size), then p may be unreachable as soon as the program enters syscall.Write. The finalizer may run at that moment, closing p.d, causing syscall.Write to fail because it is writing to a closed file descriptor (or, worse, to an entirely different file descriptor opened by a different goroutine). To avoid this problem, call runtime.KeepAlive(p) after the call to syscall.Write.
runtime.KeepAlive usage:
KeepAlive marks its argument as currently reachable. This ensures that the object is not freed, and its finalizer is not run, before the point in the program where KeepAlive is called.
func a(fd int) {
file := os.NewFile(uintptr(fd), "")
defer func() {
if err := syscall.Close(int(file.Fd()); err != nil {
fmt.Printf("%v", err)
}
runtime.KeepAlive(file)
}()
}
I've modified the official documentation example for the zlib package to use an opened file rather than a set of hardcoded bytes (code below).
The code reads in the contents of a source text file and compresses it with the zlib package. I then try to read back the compressed file and print its decompressed contents into stdout.
The code doesn't error, but it also doesn't do what I expect it to do; which is to display the decompressed file contents into stdout.
Also: is there another way of displaying this information, rather than using io.Copy?
package main
import (
"compress/zlib"
"io"
"log"
"os"
)
func main() {
var err error
// This defends against an error preventing `defer` from being called
// As log.Fatal otherwise calls `os.Exit`
defer func() {
if err != nil {
log.Fatalln("\nDeferred log: \n", err)
}
}()
src, err := os.Open("source.txt")
if err != nil {
return
}
defer src.Close()
dest, err := os.Create("new.txt")
if err != nil {
return
}
defer dest.Close()
zdest := zlib.NewWriter(dest)
defer zdest.Close()
if _, err := io.Copy(zdest, src); err != nil {
return
}
n, err := os.Open("new.txt")
if err != nil {
return
}
r, err := zlib.NewReader(n)
if err != nil {
return
}
defer r.Close()
io.Copy(os.Stdout, r)
err = os.Remove("new.txt")
if err != nil {
return
}
}
Your defer func doesn't do anything, because you're shadowing the err variable on every new assignment. If you want a defer to run, return from a separate function, and call log.Fatal after the return statement.
As for why you're not seeing any output, it's because you're deferring all the Close calls. The zlib.Writer isn't flushed until after the function exits, and neither is the destination file. Call Close() explicitly where you need it.
zdest := zlib.NewWriter(dest)
if _, err := io.Copy(zdest, src); err != nil {
log.Fatal(err)
}
zdest.Close()
dest.Close()
I think you messed up the code logic with all this defer stuff and your "trick" err checking.
Files are definitively written when flushed or closed. You just copy into new.txt without closing it before opening it to read it.
Defering the closing of the file is neat inside a function which has multiple exits: It makes sure the file is closed once the function is left. But your main requires the new.txt to be closed after the copy, before re-opening it. So don't defer the close here.
BTW: Your defense against log.Fatal terminating the code without calling your defers is, well, at least strange. The files are all put into some proper state by the OS, there is absolutely no need to complicate the stuff like this.
Check the error from the second Copy:
2015/12/22 19:00:33
Deferred log:
unexpected EOF
exit status 1
The thing is, you need to close zdest immediately after you've done writing. Close it after the first Copy and it works.
I would have suggested to use io.MultiWriter.
In this way you read only once from src. Not much gain for small files but is faster for bigger files.
w := io.MultiWriter(dest, os.Stdout)