In Hadoop, i have enabled authorization. I have set few acl for a directory.
When i execute getfacl command in hadoop bin, i can see mask value in that.
hadoop fs -getfacl /Kumar
# file: /Kumar
# owner: Kumar
# group: Hadoop
user::rwx
user:Babu:rwx
group::r-x
mask::rwx
other::r-x
If i run the same command using webhdfs, mask value not shown.
http://localhost:50070/webhdfs/v1/Kumar?op=GETACLSTATUS
{
"AclStatus": {
"entries": [
"user:Babu:rwx",
"group::r-x"
],
"group": "Hadoop",
"owner": "Kumar",
"permission": "775",
"stickyBit": false
}
}
What the reason for not showing mask value in webhdfs for GETFACL command.
Help me to figure it out.
HDFS implements the POSIX ACL model. The linked documentation explains that the mask entry is persisted into the group permission bits of the classic POSIX permission model. This is done to support the requirements of POSIX ACLs and also support backwards-compatibility with existing tools like chmod, which are unaware of the extended ACL entries. Quoting that document:
In minimal ACLs, the group class permissions are identical to the
owning group permissions. In extended ACLs, the group class may
contain entries for additional users or groups. This results in a
problem: some of these additional entries may contain permissions that
are not contained in the owning group entry, so the owning group entry
permissions may differ from the group class permissions.
This problem is solved by the virtue of the mask entry. With minimal
ACLs, the group class permissions map to the owning group entry
permissions. With extended ACLs, the group class permissions map to
the mask entry permissions, whereas the owning group entry still
defines the owning group permissions.
...
When an application changes any of the owner, group, or other class
permissions (e.g., via the chmod command), the corresponding ACL entry
changes as well. Likewise, when an application changes the permissions
of an ACL entry that maps to one of the user classes, the permissions
of the class change.
This is relevant to your question, because it means the mask is not in fact persisted as an extended ACL entry. Instead, it's in the permission bits. When querying WebHDFS, you've made a "raw" API call to retrieve information about the ACL. When running getfacl, you've run an application that layers additional display logic on top of that API call. getfacl is aware that for a file with an ACL, the group permission bits are interpreted as the mask, and so it displays accordingly.
This is not specific to WebHDFS. If an application were to call getAclStatus through the NameNode's RPC protocol, then it would see the equivalent of the WebHDFS response. Also, if you were to use the getfacl command on a webhdfs:// URI, then the command would still display the mask, because the application knows to apply that logic regardless of the FileSystem implementation.
Related
I have an output/ directory. When the directory to be created is
prefix := "output/2021-11-12.19.51.50.047614/2011-12"
os.MkdirAll(prefix, 0644) fails. When it is
prefix := "output/2021-11-12.19.51.50.047614"
it works.
Why can't os.MkdirAll() create nested directories? That's its purpose. Is this a bug?
The mode argument to os.MkdirAll must include some executable (111) bits.
In general, the mode argument to os.MkdirAll should be 0777, at least on Unix-like systems. In some cases, it should be 0700. (For the same reason, newly created files should be made with 0666, not 0644; sometimes 0600 is appropriate.)
The Go runtime passes the mode down to the OS-level calls. On Unix-like systems, where the permissions actually make sense,1 the bits that you provide are then modified by clearing any bits set in the current umask setting. It's this umask that should take away group and other write permissions, resulting in the eventual 0755 permissions—unless, that is, the user wants to take away more permissions (resulting in, say, 0750 or 0700) or fewer (resulting in, say, 0775).
Remember that the three groups of three bits represent read (r), write (w), and execute (x) permissions: 7 is rwx, 6 is rw-, 5 is r-x, and so on. So 0777 represents rwxrwxrwx.
Unix-like systems require execute permission to name a file within a directory, so if you don't give yourself execute permission, you can no longer work with the directory.
1On systems with ACLs, user permissions make some sense and "other" might be reasonable as the default, but "group" is ill-defined. Systems that do offer ACLs usually have a lot finer granularity than the simple rwx controls anyway, though.
Which os.FileMode values should I provide to os.Mkdir as an argument to have a permission corresponding to drwxrwxr-x?
Daniel Farrell's answer covers the literal settings for modes, but if you're working on a Unix-like system, there are two other important points:
For most cases (most files), you should just use 0666.
For most of the remaining cases (most directories), you should just use 0777.
Certain Go code checkers complain about this, but they're just wrong. 😀
The reason to use 0666 (rw-rw-rw-) and 0777 (rwxrwxrwx) every time is that on Unix-like systems, newly created files are created under a umask setting. The protections that the binary program asks for, such as 0777, are always reduced.1 The reduction is based on this umask setting.
The most permissive umask setting is 0. The least is the rather unusable 0777. The common settings are 077, 007, 022, and 002.
Bits that are set in the umask get cleared in the underlying file permissions. So a umask setting of 022 means that whatever the program asks for, the file winds up being not writable by Group or Other. A program that creates a file using 0666 mode winds up with a file whose actual mode is 0644, or rw-r--r--.
Should the user wish to grant group write permissions, they can run umask 2. Now only the Other w bit will be cleared and this file will be mode rw-rw-r--.
A umask setting of 077 takes away ---rwxrwx: that is, my files and directories are now private to me only (and the super-user of course). One of 027 takes away ----w-rwx, so that newly created files are now 0640, and newly created directories (that use 0777) are now 0750, or rwxr-x--.
One not-entirely-rare exception to the above rules is that temporary files created in areas of "unknown security" (os.CreateTemp files for instance), or files that contain any sensitive data, should be created with mode 0600. They can then be written out without worrying about the data leaking (except to the super-user as usual), and then if they turn out not to be sensitive after all, the final file can be created with the correct permissions.
The really tricky case is when creating a temporary file of unknown security which you are then able to rename (rather than copying) to a file of known-to-be-lower security level, where you'd like to os.Chmod the file to the final mode that you'd get by applying the user's umask. But that's pretty rare.
1In the special case of umask = 0, no bits are removed. So the result is the original value. But that's what the user wanted: to have the reduction be a zero-valued reduction. We're still reducing the permissions, just with a degenerate case.
drwxrwxr-x
each set of rwx (r-x, etc) is represented by one digit in an octal string where 4 = r, 2 = w, and 1=x. So Read and write combine to make 6, read and execute (eg executables in /bin are 5, and read-only is 4.
For files, the most common mode is:
0644 owner gets read and write, group and other get only read
And for directories, typical is:
0755 owner gets read, write (create within) and execute (list contents) , group and other get only read.
In your case, you'd need to add 2 to the Group digit to allow any users in the directory's Group to also create files in that directory.
I have the ability to create directories, but I have a problem where...
I cannot continue to access directories that might require a second or third operation after their initial creation; so, I want to create a script with admin access that can change the owner and set the correct permissions every time a directory is added.
How do I create a scriptfile and give it sufficient permissions to change the permissions and owners of other files? And how do I then set up a watch on my directory and all of its recursive children such that any time a new directory is created, its permissions and ownership are set to the desired state?
Also, for the sake of learning, if you have a better way to achieve the result, please tell me how to achieve the result in the method described above? I'm very interested to know how to monitor a folder and its children and trigger a script.
A lot of file permission getting changed. there is huge number of file and chmod -R taking more than 30 minutes. but most of files permission will be as needed.
Changing recursive so that if some file creates with wrong permission that gets fixed.
is there any other more perform way there?
Well, depending on the scenario there are at least three things that you can make:
Add the default ACL (and normal ACL) so some user/groups always have proper access.
Change file group and SETGID bit to group of users that must have access to the files. Also setting proper mask might be helpful.
Use audit subsystem, but with huge number of messages it is hard to maintain.
what will be the impact of changing the ownership of vob in clearcase? Does it impact the metatdata? Is there anything else need to be changed so that users won't face any problems? how does it effect triggertype, hyperlinks and attribute type ?
Note: the group is not going to be changed but I am not sure if the existing group will be affected after changing the ownership?
You would use the command cleartool protectvob -chown ...
It doesn't have side effect (on metadata like pvob, trtypes, attributes, ...), unless you are running it on multisite vobs.
You can use -chown by itself or in combination with -chgrp.
-cho/wn user
Specifies a new VOB owner, who becomes the owner of all the VOB's storage pools and all of the data containers in them.
Note that on multisite:
You run protectvob -chown or protectvob -chgrp on a VOB replica that preserves identities, you must follow these steps to prevent metadata divergence among replicas in the VOB family:
Stop synchronization among identities-preserving replicas in the family. Make sure that all update packets have been imported.
Run protectvob -chown or protectvob -chgrp on all identities-preserving replicas in the family. You must use the same options and arguments in each command.
Restart synchronization.
(A VOB's protection with respect to privileged access by remote users is never replicated. It must be set on each replica by a privileged user logged on to the VOB server host.)