I have an output/ directory. When the directory to be created is
prefix := "output/2021-11-12.19.51.50.047614/2011-12"
os.MkdirAll(prefix, 0644) fails. When it is
prefix := "output/2021-11-12.19.51.50.047614"
it works.
Why can't os.MkdirAll() create nested directories? That's its purpose. Is this a bug?
The mode argument to os.MkdirAll must include some executable (111) bits.
In general, the mode argument to os.MkdirAll should be 0777, at least on Unix-like systems. In some cases, it should be 0700. (For the same reason, newly created files should be made with 0666, not 0644; sometimes 0600 is appropriate.)
The Go runtime passes the mode down to the OS-level calls. On Unix-like systems, where the permissions actually make sense,1 the bits that you provide are then modified by clearing any bits set in the current umask setting. It's this umask that should take away group and other write permissions, resulting in the eventual 0755 permissions—unless, that is, the user wants to take away more permissions (resulting in, say, 0750 or 0700) or fewer (resulting in, say, 0775).
Remember that the three groups of three bits represent read (r), write (w), and execute (x) permissions: 7 is rwx, 6 is rw-, 5 is r-x, and so on. So 0777 represents rwxrwxrwx.
Unix-like systems require execute permission to name a file within a directory, so if you don't give yourself execute permission, you can no longer work with the directory.
1On systems with ACLs, user permissions make some sense and "other" might be reasonable as the default, but "group" is ill-defined. Systems that do offer ACLs usually have a lot finer granularity than the simple rwx controls anyway, though.
Related
Which os.FileMode values should I provide to os.Mkdir as an argument to have a permission corresponding to drwxrwxr-x?
Daniel Farrell's answer covers the literal settings for modes, but if you're working on a Unix-like system, there are two other important points:
For most cases (most files), you should just use 0666.
For most of the remaining cases (most directories), you should just use 0777.
Certain Go code checkers complain about this, but they're just wrong. 😀
The reason to use 0666 (rw-rw-rw-) and 0777 (rwxrwxrwx) every time is that on Unix-like systems, newly created files are created under a umask setting. The protections that the binary program asks for, such as 0777, are always reduced.1 The reduction is based on this umask setting.
The most permissive umask setting is 0. The least is the rather unusable 0777. The common settings are 077, 007, 022, and 002.
Bits that are set in the umask get cleared in the underlying file permissions. So a umask setting of 022 means that whatever the program asks for, the file winds up being not writable by Group or Other. A program that creates a file using 0666 mode winds up with a file whose actual mode is 0644, or rw-r--r--.
Should the user wish to grant group write permissions, they can run umask 2. Now only the Other w bit will be cleared and this file will be mode rw-rw-r--.
A umask setting of 077 takes away ---rwxrwx: that is, my files and directories are now private to me only (and the super-user of course). One of 027 takes away ----w-rwx, so that newly created files are now 0640, and newly created directories (that use 0777) are now 0750, or rwxr-x--.
One not-entirely-rare exception to the above rules is that temporary files created in areas of "unknown security" (os.CreateTemp files for instance), or files that contain any sensitive data, should be created with mode 0600. They can then be written out without worrying about the data leaking (except to the super-user as usual), and then if they turn out not to be sensitive after all, the final file can be created with the correct permissions.
The really tricky case is when creating a temporary file of unknown security which you are then able to rename (rather than copying) to a file of known-to-be-lower security level, where you'd like to os.Chmod the file to the final mode that you'd get by applying the user's umask. But that's pretty rare.
1In the special case of umask = 0, no bits are removed. So the result is the original value. But that's what the user wanted: to have the reduction be a zero-valued reduction. We're still reducing the permissions, just with a degenerate case.
drwxrwxr-x
each set of rwx (r-x, etc) is represented by one digit in an octal string where 4 = r, 2 = w, and 1=x. So Read and write combine to make 6, read and execute (eg executables in /bin are 5, and read-only is 4.
For files, the most common mode is:
0644 owner gets read and write, group and other get only read
And for directories, typical is:
0755 owner gets read, write (create within) and execute (list contents) , group and other get only read.
In your case, you'd need to add 2 to the Group digit to allow any users in the directory's Group to also create files in that directory.
Reading the man pages for fchown, I find this statement:
The fchown() system call is particularly useful when used in conjunction with the file locking primitives (see flock(2)).
Here's the mystery: the man page for flock(2) makes no mention of fchown or even how ownership affects it in general.
So can anyone explain what happens when fchown and flock are used together and why it's so "useful?"
I'm developing for macOS (Darwin), but I find the same statement (and lack of an explanation) in Linux, BSD, POSIX, and virtually every other *NIX man page I've searched.
Backstory ('cause every great villain has a backstory):
I have a set-UID helper process that gets executed as root, but spends much of its time running as user. While it's running as user, the files it creates belong to user. Good so far.
However, occasionally it needs to create files while running as root. When this happens, the files belong to root and I want them to belong to user. So my plan was to create+open the file, then call fchown() to change the ownership back to user.
But a few of these files are shared and I use flock() to block concurrent access to the file and now I'm wondering what will happen to my flocks.
A lot of file permission getting changed. there is huge number of file and chmod -R taking more than 30 minutes. but most of files permission will be as needed.
Changing recursive so that if some file creates with wrong permission that gets fixed.
is there any other more perform way there?
Well, depending on the scenario there are at least three things that you can make:
Add the default ACL (and normal ACL) so some user/groups always have proper access.
Change file group and SETGID bit to group of users that must have access to the files. Also setting proper mask might be helpful.
Use audit subsystem, but with huge number of messages it is hard to maintain.
I am working on ubuntu server with 10 users at any point of time. We usually keep our code there and use the server to make builds. The build usually takes 30 to 50 minutes based on concurrency defined. The build command is make -jX where X can be anything from 1 to 24.
My problem starts when many users start giving make command with higher X value. Is there any way to block these commands or to put any limit.
For example, if someone gives make -jX (X>4), I should be able to override the command as make -j4.
I know one way is to use alias but I have no idea how to interpret the argument value through alias (like alias ll='ls -la' in .bashrc file is ok but how to interpret ll -lha through bashrc).
Also is there any way to make the alias work for all the users without editing the bashrc files of all the users?
Thanks in advance.
Although limiting the parameters to a particular command (in this case make) in a way that cannot be circumvented is generally hard, you can configure system-level process limits on each user using /etc/security/limits.conf.
If you open this file on your system, you will see in the comments that you can limits various user resources such as nproc and memory. If you play with these limits, you may be able to get something reasonable to get fair resource sharing among your developers.
I want to write a program that allows or blocks processes while openning a file depending on a policy.
I could make a control by checking the name of the program. However, it would not be enough because user can change the name to pass the policy. i.e. let's say that policy doesn't allow a.exe to access txt files whereas b.exe is allowed. If user change a.exe with b.exe, i cannot block it.
On the other hand, verifying portable executable signature is not enough for me, because i don't care whether the executable signed or not. I just want to identify the executable that is wanted to execute even its name is changed.
For this type of case, what would you propose? Any solutions are welcome.
Thanks in advance
There are many ways to identify an executable file. Here is a simple list:
Name:
The most simple and straightforward approach is to identify a file by its name. But it is one of the easiest things to change, and you already ruled that out.
Date:
Files have an access, creation, and modification date, and they are managed by the operating system. They are not foolproof, or maybe not even accurate.
Also, they are very simple to change.
Version Information:
Since we are talking about executable files, then most executable files have version information attached to it. For example, original file name, file version, product version, company, description, etc. You can check these fields if you are sure the user cannot modify them by editing your executable. It doesn't require you to keep a database of allowed files. However, it does require you to have something to compare to, like company name, or a product name. Also, If someone made an executable with the same value
they can run instead of the allowed one and bypass your protection.
Location:
If the file is located in a specific place and is protected by file system access rights, and it cannot be changed, then you can use that. You can, for example, put the allowed files in a folder where the user (without admin rights) can only read/execute them, but not rename/move. Then identify the file with its location. If it is run from this location, then allow it, else block it. It is good as it doesn't need a database of
allowed/blocked files, it just compares the location, it it is a valid one, then allow, and you can keep adding and removing files to allowed locations
without affecting your program.
Size:
If the file has a specific file size, you can quickly check its size and compare it. But it is unreliable as files can be changed/patched and without any change in size. You can avoid that by also applying a CRC check to detect if the content of the file changed.
But, both size and CRC can be changed. Also, this requires you to have a list of file names and their sizes/CRC, and keeping it up to date.
Signature:
Deanna mentioned that you can self-sign your executable files. Then check if the signature matches yours and allow/deny based on that. This seems to be a good way if
it is okay for you to sign all the executable files you want to allow. It doesn't require you to keep an updated list of allowed files.
Hash:
arx also pointed out, that you can hash the files. It is one of the slowest methods, as it requires the file to be hashed every time it is executed, then compared to a list of files. But it very reliable as it can uniquely identify each file and hard to break. But, you will need to keep an up to date database of every file hash.
Finally, and depend on your needs and options, you can mix two or more ways together to get the results you want. Like, checking file name + location, etc.
I hope I covered most of things, but I'm sure there are more ways. Anyone can freely edit my post to include anything that I have missed.
I would recommend using the signature, if it has one, or the hash otherwise. Apps such as Office that update frequently are more likely to be signed, whereas smaller apps downloaded off the Internet are unlikely to ever be updated and so should have a consistent hash.