I am trying to change directories in a go routine to directory x. I now want to use a different go routine that changes the directory to directory y. Will the execution of my first go routine be affected by this change to the current working directory in the second go routine? The purpose of wanting to do this is to introduce parallelism while doing similar tasks. If it does end up changing the CWD, what should be an alternative approach (forking...)?
Like mentioned in the comments, keeping track of the current working directory in each goroutine will cause problems.
Try using filepath.Abs to capture the absolute directory and store that instead. Then each goroutine can operate on it's own directory without worrying about it being "switched" under the hood. Just be sure you're not modifying the same file accidentally by multiple goroutines.
Edit: Removing a chunk of text per #Evan's comment. Use absolute paths :p
#Evan has identified a fundamental flaw in attempting to use the 'change working directory' (CWD) system call.
I believe that #Evan is correct, and that the CWD is a thread property on some OS's.
As #Evan pointed out, a goroutine could be resheduled (for example at a function call, channel access, or system call) onto a different thread.
The implications are, it may be impossible to change the CWD (if Chdir() could change the threads CWD) because Go's runtime chooses to reschedule the goroutine on a different thread; its CWD could change invisibly and unpredictably.
Edit: I would not expect Chdir() to do anything other than change the CWD for the process. However, the documentation for the package has no mention of 'process'.
Worse, the runtime may change how things work with releases.
Even worse, it would be very hard to debug. It may be a 'Heisenberg problem', where any attempt to debug it (for example by calling a function, which the runtime may use as a reschedule point) may actually changes the behaviour in an unpredictable way.
Keep track of absolute path names. This is explicit, clear, and would even work across goroutines without any need for synchronisation. Hence it is simpler, and easier to test and debug.
Related
For context: I was reading this example of trickery one can do with a shebang. It uses
#!/bin/rm
which ends up deleting the file you executed (very funny; you can extend this to create self-deleting messages). This shows that the program (rm) is invoked with the filename as an argument and therefore has access to the whole file including the shebang that invoked it.
Another trickery I've come up with is to invoke yourself as an interpreter in shebang to create an infinite loop. For example if /usr/bin/loop starts with
#!/usr/bin/loop
t should invoke itself with itself forever. Obviously at some point an error will occur and in my particular case I get:
bash: /usr/bin/loop: /usr/bin/loop: bad interpreter: Too many levels of symbolic links
It looks like it got two levels deep. Can somebody explain to me why this particular error occurs? Or maybe share some other error messages for different shells.
In particular I would like to understand why there are symbolic links involved and whether this is an implementation detail of bash or not.
why this particular error occurs?
Because when kernel tries to run the executable, tries to run /usr/bin/loop, then tries to run /usr/bin/loop, then tries to run /usr/bin/loop, etc. and finally fails with ELOOP error. It checks that.
https://elixir.bootlin.com/linux/latest/source/fs/exec.c#L1767
hare some other error messages for different shells.
While it could be, this particular errno message comes from glibc strerror.
why there are symbolic links involved
Because ELOOP is typically returned when doing input/output operations, the message mentions them. Symbolic links are not involved.
this is an implementation detail of bash or not.
Not.
Reading the man pages for fchown, I find this statement:
The fchown() system call is particularly useful when used in conjunction with the file locking primitives (see flock(2)).
Here's the mystery: the man page for flock(2) makes no mention of fchown or even how ownership affects it in general.
So can anyone explain what happens when fchown and flock are used together and why it's so "useful?"
I'm developing for macOS (Darwin), but I find the same statement (and lack of an explanation) in Linux, BSD, POSIX, and virtually every other *NIX man page I've searched.
Backstory ('cause every great villain has a backstory):
I have a set-UID helper process that gets executed as root, but spends much of its time running as user. While it's running as user, the files it creates belong to user. Good so far.
However, occasionally it needs to create files while running as root. When this happens, the files belong to root and I want them to belong to user. So my plan was to create+open the file, then call fchown() to change the ownership back to user.
But a few of these files are shared and I use flock() to block concurrent access to the file and now I'm wondering what will happen to my flocks.
I'm trying to write a program, calcsize, that calculates the size of all sub directories. I want to create a cache of the result and only re-walk the directory if it has changed since the last time I've run the program.
Something like:
./calcsize
//outputs
/absolute/file/path1/ 1000 Bytes
/absolute/file/path2/ 2000 Bytes
I'm already walking the dirs with my own walk implementation because the built in go filepath.Walk is already calling Lstat on every file.
Is there any way to know if a directory or set of files has changed without calling Lstat on every file? Maybe a system call I'm not aware of?
In general, no. However you might want to look at: https://github.com/mattn/go-zglob/blob/master/fastwalk/fastwalk_unix.go
And using that data you can skip some of the stat calls, if you only care about files.
Whether, and how, this is possible depends heavily on your operating system. But you might take a look at github.com/howeyc/fsnotify which claims to offer this (I've never used it--I just now found it via Google).
In general, look at any Go program that provides a 'watch' feature. GoConvey and GopherJS's serve mode come to mind as examples, but there are others (possibly even in the standard library).
From the doc (ReadDirectoryChangesW):
"Retrieves information that describes the changes within the specified directory. The function does not report changes to the specified directory itself."
My question is: What do I use to report changes to the specified directory itself?
I want to be able to capture changes not only to things and sub-things in the folder itself but also to detect for example, when the folder itself has been deleted.
One strategy would be to actually monitor for changes on the parent of the folder I'm really interested in and then use that to generate an event when the folder I'm interested in is deleted. This works but has the potential to generate thousands of 'uninteresting' events.
A second strategy is to have a recursive monitor for stuff under the folder I'm actually interested in and then a non-recursive monitor on it'a parent. The non-recursive monitor would then be able to tell me when the real folder of interest is deleted.
The latter, second strategy, generates fewer events and is the strategy I would like to use. BUT: It doesn't work 'in process'. That is, if I start to monitor the folder of interest recursively (HANDLE A), and it's parent non-recursively (HANDLE B) and then in the same process, I try and delete the folder of interest, no removal event is generated for it (even though I verify from a console that the thing no longer exists). My suspicion is that this is due to HANDLE A on the folder still being open, and even though I have included the "FILE_SHARE_DELETE" flag in the call to CreateFileW that gave me HANDLE A, it simply can't work.
Note that 'Out of process', i.e. when I delete the folder from within a completely separate process, the above strategy does work.
So, what are my options?
Many thanks,
Ben.
Please note this is not duplicate of File r/w locking and unlink. (The difference - platform. Operations of files like locking and deletion have totally different semantics, thus the sultion would be different).
I have following problem. I want to create a file system based session storage where each session data is stored in simple file named with session ids.
I want following API: write(sid,data,timeout), read(sid,data,timeout), remove(sid)
where sid==file name, Also I want to have some kind of GC that may remove all timed-out sessions.
Quite simple task if you work with single process but absolutly not trivial when working with multiple processes or even over shared folders.
The simplest solution I thought about was:
write/read:
hanlde=CreateFile
LockFile(handle)
read/write data
UnlockFile(handle)
CloseHanlde(handle)
GC (for each file in directory)
hanlde=CreateFile
LockFile(handle)
check if timeout occured
DeleteFile
UnlockFile(handle)
CloseHanlde(handle)
But AFIAK I can't call DeleteFile on opended locked file (unlike in Unix where file locking is
not mandatory and unlink is allowed for opened files.
But if I put DeleteFile outside of Locking loop bad scenario may happen
GC - CreateFile/LockFile/Unlock/CloseHandle,
write - oCreateFile/LockFile/WriteUpdatedData/Unlock/CloseHandle
GC - DeleteFile
Does anybody have an idea how such issue may be solved? Are there any tricks that allow
combine file locking and file removal or make operation on file atomic (Win32)?
Notes:
I don't want to use Database,
I look for a solution for Win32 API for NT 5.01 and above
Thanks.
I don't really understand how this is supposed to work. However, deleting a file that's opened by another process is possible. The process that creates the file has to use the FILE_SHARE_DELETE flag for the dwShareMode argument of CreateFile(). A subsequent DeleteFile() call will succeed. The file doesn't actually get removed from the file system until the last handle on it is closed.
You currently have data in the record that allows the GC to determine if the record is timed out. How about extending that housekeeping info with a "TooLateWeAlreadyTimedItOut" flag.
GC sets TooLateWeAlreadyTimedItOut = true
Release lock
<== writer comes in here, sees the "TooLate" flag and so does not write
GC deletes
In other words we're using a kind of optimistic locking approach. This does require some additional complexity in the Writer, but now you're not dependent upon any OS-specifc wrinkles.
I'm not clear what happens in the case:
GC checks timeout
GC deletes
Writer attempts write, and finds no file ...
Whatever you have planned for this case can also be used in the "TooLate" case
Edited to add:
You have said that it's valid for this sequence to occur:
GC Deletes
(Very slightly later) Writer attempts a write, sees no file, creates a new one
The writer can treat "tooLate" flag as a identical to this case. It just creates a new file, with a different name, use a version number as a trailing part of it's name. Opening a session file the first time requires a directory search, but then you can stash the latest name in the session.
This does assume that there can only be one Writer thread for a given session, or that we can mediate between two Writer threads creating the file, but that must be true for your simple GC/Writer case to work.
For Windows, you can use the FILE_FLAG_DELETE_ON_CLOSE option to CreateFile - that will cause the file to be deleted when you close the handle. But I'm not sure that this satisfies your semantics (because I don't believe you can clear the delete-on-close attribute.
Here's another thought. What about renaming the file before you delete it? You simply can't close the window where the write comes in after you decided to delete the file but what if you rename the file before deleting it? Then when the write comes in it'll see that the session file doesn't exist and recreate it.
The key thing to keep in mind is that you simply can't close the window in question. IMHO there are two solutions:
Adding a flag like djna mentioned or
Require that a per-session named mutex be acquired which has the unfortunate side effect of serializing writes on the session.
What is the downside of having a TooLate flag? In other words, what goes wrong if you delete the file prematurely? After all your system has to deal with the file not being present...