There are couple of folders in the root dir of HDFS:
dir1
subdir1
table1
table2
subdir2
dir2
subdir1
table1
table2
dir3
They all have subfolders that contain different Parquet files that are queried with Hive.
I can't load one of the subfolders (for example table1 inside dir2) even though the permissions look ok to me, I get the EXECUTE error when trying to load it.
The code is running in a Jupyter notebook.
Users are organized in groups.
I've added rwx permissions for the directory in question to the group by using the following command:
hdfs dfs -setfacl -R -m group:user_group:rwx /dir2/subdir2
The error I'm getting looks like this:
Cannot create staging directory 'hdfs://server:8020/dir2/subdir1/table1/.hive-staging_hive_2019-08-01_13-04-22': Permission denied: user=username, access=EXECUTE, inode="/dir2":hdfs:supergroup:drwxrwx---
I've added read and execute permissions on dir2 to the user group but the error persists. It looks to me from this error that somehow the default permissions are applied and they are ---
So, to summarize;
group has read and execute privileges on the root dir, and read, write and execute privileges on the table directories, but it keeps failing with permissions for root directory.
This is how the permissions look:
# file: /dir2
# owner: hdfs
# group: supergroup
user::rwx
user:some_group1:r-x
group::---
group:some_group2:rwx
group:user_group:r-x
group:hive:rwx
group:some_group3:r-x
group:some_group4:r-x
mask::rwx
other::---
default:user::rwx
default:user:some_group1:r-x
default:group::---
default:group:some_group2:rwx
default:group:hive:rwx
default:group:some_group3:r-x
default:group:some_group4:r-x
default:mask::rwx
default:other::---
# file: /dir2/subdir1/table1
# owner: some_user
# group: supergroup
user::rwx
user:some_group1:r-x
group::---
group:some_group2:rwx
group:user_group:rwx
group:hive:rwx
group:some_group3:r-x
group:some_group4:rwx
mask::rwx
other::---
default:user::rwx
default:user:some_group1:r-x
default:group::---
default:group:some_group2:rwx
default:group:user_group:rwx
default:group:hive:rwx
default:group:some_group3:r-x
default:group:some_group4:rwx
default:mask::rwx
default:other::---
The problem was eventually solved by creating new directories that replaced the old ones. The new directories were created with the correct user and credentials.
For example, I created subdir1_new, moved the data there, renamed subdir1 to subdir1_old and renamed subdir1_new to subdir1. Not a lot of folders were affected by this issue so it didn't take a long time.
I know it's not the actual solution, but I couldn't figure out what exactly was happening and this workaround did the trick.
Related
I want to set two groups to a directory
app_group
td-agent
If use file module, there is a group attribute can do
- name: Set group to directory
file:
path: /app
group: app_group
state: directory
recurse: true
mode: '0774'
The current permissions like
$ ls -la /app/app1/
drwxrwx---. 5 root app_group 31 Oct 11 14:12 .
drwxr-xr-x. 3 root root 21 May 12 2021 ..
drwxrwx---. 3 app_user app_group 12 Apr 28 13:30 product
But I want to set Fluentd's user group td-group can also read files in /app directory. Like /app/app1/product/001.log. How to do?
If change mode to mode: o+rx, other user can read/execute the directory. Will it change current permission for app_user and app_group?
Since the underlying operating system seems to be Linux, the answer to your question
How to set permission to a directory with multiple groups?
will be you can't do that. At least not in that way. Ansible can configure only what is available.
So you may have a look into the basic concepts of how to manage Linux permissions for users, groups, and others.
In example
Linux group permissions
Granting write permissions to a group to a folder
How does group permissions work?
and maybe put the fluentd user into app_group. An other approach could be introducing Access Control Lists (ACL).
In example
How can I give permissions of files to multiple groups?
Two Groups Separate Permissions, One Directory?
I just noticed that some unknown folder has been created in my /etc/ansible directory.
Here it is ,
monitoring-server#monitoring-server:/etc/ansible$ ls
[' ansible.cfg hosts java.retry java.yml nginx.retry nginx.yml roles test.retry test.yml
This [' is the folder.
monitoring-server#monitoring-server:/etc/ansible$ ls -l
total 56
drw-r--r-- 3 root root 4096 Jul 27 07:12 ['
I need to be root to open it.
root#monitoring-server:/etc/ansible/['# tree
.
└── opt
└── rsyslog']
2 directories, 0 files
Well, the /opt/rsyslog is one of the directories I made on some remote servers using ansible.
How is this folder created and Why is it created?
Directory tree is ['/opt/rsyslog'] – seems like you mistyped string path as list sometime ago and fed it into module with local host as target.
As long as /etc/ansible also seems to be your working directory (which is really weird practice), path [' / opt / rsyslog'] has been created.
You may safely remove it.
I'm using proftpd server with below dir structure:
main www dir:
/home/www
also there are some symbolic links to that main dir:
/home/john -> linked to /home/www
/home/mark -> linked to /home/www
etc.
When one of that users upload of new file he is the owner of that file but I need to change ownership and permissions for default 0755 and www:www
Of couse all users are in the same group www but sometimes user upload file with to low permissions so no one from group can change/remove the file from ftp.
Is there any way to do it automatically?
User rok uploaded file and set the permission to 770. The file on HDFS looks like this:
-rw-rw---- 3 rok hdfs filename1
I'm using ksc user to consume the data uploaded by rok user. So first, I'd like to make sure that ksc has permission for that file filename1.
How do I find out the group name of my user ksc? Does user belong to hdfs group in Hadoop?
BTW, if I upload a file to Hadoop, the file permission looks like:
-rw-r--r-- 3 ksc ksc filename2
The local info on my Linux of ksc user is :
uid=504(ksc) gid=502(ksc) groups=502(ksc)
Use the command below:
$hdfs groups ksc
It gives all of the groups user ksc belongs to.
HDFS follows the traditional style of Linux file system permssions. To determine the group of ksc, use groups ksc if you are on Linux.
-rw-rw---- 3 rok hdfs filename1 will give you read/write permissions only if you are part of the hdfs group. Judging from your output, I'm thinking you're not.
You will need to do one of the following:
Change rok's file permissions to 664 (read permissions for all users), which is pretty insecure
Have ksc added to the hdfs group, more secure
The choice is yours...
Consult the following links for more information:
http://www.comptechdoc.org/os/linux/usersguide/linux_ugfilesp.html
https://hadoop.apache.org/docs/r1.1.1/hdfs_permissions_guide.html
The way that Hadoop maps users to groups is configurable, so HDFS groups may not be the same as the Unix groups. Also note that if your Hadoop configuration does use the Unix user-group mappings, it will use the unix mappings on the NameNode. Also note that the NameNode caches the mappings for a period of time, so any changes you make may not be available until the cache is expired/refreshed.
As for checking, in addition to what is already mentioned you can check the actual system file that contains the mappings like this if you have root access:
grep <user or group> /etc/group
More here:
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/GroupsMapping.html
I am currently running Cygwin on a target Windows Server 2003 machine to fire off a shell script that, among other things, creates a bunch of files on disc. However after the files are created I no longer have permissions to manipulate them through Windows.
When the files are created the owner is getting set to 'SYSTEM' and the permissions for Administrators/Creator Group/Creator Owner/system are set to only 'special permissions' and nothing else.
The permissions for Everyone and Users have Read & Execute, List folder contents and Read.
My problem is that I cannot delete/modify the files now through Windows. I would prefer to have something built into my scripts (either the shell script or something to call in Cygwin) that would allow Administrators full control on the folder and all contents.
My current workaround has been to either do file modifications through Cygwin but this is not preferable. I have also used setfacl -r -m default:other:rwx to add write permissions for the 'Users' group but it doesn't appear to have a recursive option and still doesn't give 'full control'
Is there a better way to use setfacl? Can I call the shell script using different/elevated permissions?
Results of getfacl on a newly created directory:
$ getfacl Directory/
# file: Directory/
# owner: SYSTEM
# group: root
user::rwx
group::r-x
group:Users:rwx
mask:rwx
other:r-x
default:user::rwx
default:group::r-x
default:group:Users:rwx
default:mask:rwx
default:other:r-x
You can try setting umask:
umask u=rwx,g=rwx,o=rwx
That should give user, group, and other read/write/execute on any newly created dirs.
If you only want the modified umask permanently, you can add it to your .bash_profile
Edit - Added example of mkdir before/after umask.
Here's the output of getfacl on a directory created before I set umask:
[/cygdrive/c/Documents and Settings/NOYB/Desktop]
==> getfacl test_wo_umask/
# file: test_wo_umask/
# owner: NOYB
# group: Domain Users
user::rwx
group::r-x
group:root:rwx
group:SYSTEM:rwx
mask:rwx
other:r-x
default:user::rwx
default:user:NOYB:rwx
default:group::r-x
default:group:root:rwx
default:group:SYSTEM:rwx
default:mask:rwx
default:other:r-x
Here's the output of getfacl on a directory created after I set umask:
[/cygdrive/c/Documents and Settings/NOYB/Desktop]
==> getfacl test_w_umask/
# file: test_w_umask/
# owner: NOYB
# group: Domain Users
user::rwx
group::rwx
group:root:rwx
group:SYSTEM:rwx
mask:rwx
other:rwx
default:user::rwx
default:user:NOYB:rwx
default:group::rwx
default:group:root:rwx
default:group:SYSTEM:rwx
default:mask:rwx
default:other:rwx