My user id and my team cannot access any of the internal tables in hive db. when we fire up the queries in HUE and 'CLI' as well, we are getting
'AccessControlException', please find the log below,
INFO : set mapreduce.job.reduces=<number> INFO : Cleaning up the staging area maprfs:/var/mapr/cluster/yarn/rm/staging/keswara/.staging/job_1494760161412_0139
ERROR : Job Submission failed with exception org.apache.hadoop.security.AccessControlException
(User keswara(user id 1802830393) does not have access to
maprfs:///user/hive/warehouse/bistore_sit.db/wt_consumer/d_partition_number=0/000114_0)'
org.apache.hadoop.security.AccessControlException: User keswara(user id 1802830393) does not have access to maprfs:///user/hive/warehouse/bistore_sit.db/wt_consumer/d_partition_number=0/000114_0
at com.mapr.fs.MapRFileSystem.getMapRFileStatus(MapRFileSystem.java:1320)
at com.mapr.fs.MapRFileSystem.getFileStatus(MapRFileSystem.java:942)
at org.apache.hadoop.fs.FileSystem.getFileBlockLocations(FileSystem.java:741)
at org.apache.hadoop.fs.FileSystem$4.next(FileSystem.java:1762)
at org.apache.hadoop.fs.FileSystem$4.next(FileSystem.java:1747) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:307) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:265) at org.apache.hadoop.hive.shims.Hadoop23Shims$1.listStatus(Hadoop23Shims.java:148) at org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat.getSplits(CombineFileInputFormat.java:218) at org.apache.hadoop.mapred.lib.CombineFileInputFormat.getSplits(CombineFileInputFormat.java:75) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getSplits(HadoopShimsSecure.java:310) at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getCombineSplits(CombineHiveInputFormat.java:472) at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:573) at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:331) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:323) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:199) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:421)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:421)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548) at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:431) at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:75)
any of the user can't access the internal tables right now,am part of the mapr group and sudo user as well!
and the table and partitions ownership belongs to the mapr group and the permissions are look good though!
[mapr#SAN2LPMR03 mapr]$ hadoop fs -ls /user/hive/warehouse/bistore.db/wt_consumer
Found 1 items
drwxrwxrwt - mapr mapr 1 2017-03-24 11:51 /user/hive/warehouse/bistore.db/wt_consumer/d_partition_number=__HIVE_DEFAULT_PARTITION__
Please help me to sort this out! Really appreciate your help!
If the tables are in parquet format then the files for that table will have write access only for the user who has created the table.
For this you can change the user permissions for that files using statement like below
hdfs dfs -chomd 777 /user/hive/warehouse/bistore_sit.db/wt_consumer/d_partition_number=0/000114_0/*
This statement will grant all users all the permissions to that particular files.
I have noticed the following while testing for some tables in both CSV and parquet formats.
When you create hive table in CSV format the table will have 777 permission for all users who have access to the group you are part of.
But when the hive table is created in parquet format only the user who has created the table will have write access. I think it has to do something with parquet format
[root#psnode44 hive-2.1]# hadoop fs -ls /user/hive/warehouse/
Found 1 items
drwxrw-rw- - mapr mapr 2 2017-06-28 12:49 /user/hive/warehouse/test
0: jdbc:hive2://10.20.30.44:10000/> select *from test;
Error: java.io.IOException: org.apache.hadoop.security.AccessControlException: User basa(user id 5005) does not have access to maprfs:/user/hive/warehouse/test (state=,code=0)
[root#psnode44 hive-2.1]# hadoop fs -ls /user/hive/warehouse/
Found 1 items
drwxrwxrwx - mapr mapr 2 2017-06-28 12:49 /user/hive/warehouse/test
Even thought, I changed the chmod on warehouse,still its getting same error.
[root#psnode44 hive-2.1]# hadoop fs -chmod -R 777 /user/hive/warehouse/
[root#psnode44 hive-2.1]# hadoop fs -ls /user/hive/warehouse/
Found 1 items
drwxrwxrwx - mapr mapr 2 2017-06-28 12:49 /user/hive/warehouse/test
0: jdbc:hive2://10.20.30.44:10000/> select *from test;
Error: java.io.IOException: org.apache.hadoop.security.AccessControlException: User basa(user id 5005) does not have access to maprfs:/user/hive/warehouse/test (state=,code=0)
Related
I have a single node Cloudera Cluster (CDH 5.16) in a Rhel 7 Remote server.
I have installed CDH using packages.
When i am running sqoop import job, i am getting the following error:
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
19/06/04 15:49:31 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.16.1
19/06/04 15:49:31 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
19/06/04 15:49:32 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
19/06/04 15:49:32 INFO tool.CodeGenTool: Beginning code generation
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
19/06/04 15:49:34 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `categories` AS t LIMIT 1
19/06/04 15:49:35 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `categories` AS t LIMIT 1
19/06/04 15:49:35 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce
Note: /tmp/sqoop-ak_bng/compile/d07f2f60a7ecbf9411c79687daa024c9/categories.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
19/06/04 15:49:37 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-ak_bng/compile/d07f2f60a7ecbf9411c79687daa024c9/categories.jar
19/06/04 15:49:38 ERROR tool.ImportTool: Import failed: org.apache.hadoop.security.AccessControlException: Permission denied by sticky bit: user=ak_bng, path="/user/hive/warehouse/sales.db/categories":hive:hive:drwxr-xr-t, parent="/user/hive/warehouse/sales.db":hive:hive:drwxr-xr-t
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkStickyBit(DefaultAuthorizationProvider.java:387)
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:159)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:152)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3885)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6861)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:4290)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInt(FSNamesystem.java:4245)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:4229)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.delete(NameNodeRpcServer.java:856)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.delete(AuthorizationProviderProxyClientProtocol.java:313)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.delete(ClientNamenodeProtocolServerSideTranslatorPB.java:626)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2278)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2274)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2272)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2106)
at org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:688)
at org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:684)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:684)
at org.apache.sqoop.tool.ImportTool.deleteTargetDir(ImportTool.java:546)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:509)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:621)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied by sticky bit: user=ak_bng, path="/user/hive/warehouse/sales.db/categories":hive:hive:drwxr-xr-t, parent="/user/hive/warehouse/sales.db":hive:hive:drwxr-xr-t
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkStickyBit(DefaultAuthorizationProvider.java:387)
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:159)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:152)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3885)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6861)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:4290)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInt(FSNamesystem.java:4245)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:4229)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.delete(NameNodeRpcServer.java:856)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.delete(AuthorizationProviderProxyClientProtocol.java:313)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.delete(ClientNamenodeProtocolServerSideTranslatorPB.java:626)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2278)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2274)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2272)
at org.apache.hadoop.ipc.Client.call(Client.java:1504)
at org.apache.hadoop.ipc.Client.call(Client.java:1441)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:231)
at com.sun.proxy.$Proxy10.delete(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:552)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
at com.sun.proxy.$Proxy11.delete(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2104)
... 13 more
Sqoop command is
sqoop import --connect jdbc:mysql://10.188.177.228:3306/sales --username vaishak --password root_123 --table categories --m 1 --delete-target-dir --target-dir /user/hive/warehouse/sales.db/categories
As per the below documentation, i tried changing the fs.defaultFS in core-site.xml.
https://community.cloudera.com/t5/CDH-Manual-Installation/Permission-denied-user-root-access-WRITE-inode-quot-user/td-p/4943
It didnt work.
I tried the below link in stackoverflow:
Permission exception for Sqoop
That too didnt work out for me.
i created a new folder for ak_bng and added to hive group as below:
sudo -u hdfs hadoop fs -mkdir /user/ak_bng
sudo -u hdfs hadoop fs -chown ak_bng:hive /user/ak_bng
i was still getting the same error.
in few links i saw suggestions of adding users(in my case ak_bng) to supergroup.
But i am not aware of how to do it.
few suggested to run sqoop script as different user. I am not aware of how to do that too.
I am very new to Unix and CDH and i am not aware of how to achieve this.
I had a similar permission issue when i tried running sqoop scripts from HUE editor
Following is the error i got at that time:
Failed to create deployment directory: AccessControlException: Permission denied: user=hive, access=WRITE, inode="/user/hue/oozie/deployments":hue:hue:drwxr-xr-x (error 500)
Before CDH, I had setup Hadoop 3.1 and sqoop separately (Both outside CDH) and i was able to successfully import data into HDFS.
But while using CDH i am getting these errors.
Can someone please shed some light on what is the issue here and how to resolve this issue.
Output of hadoop fs -ls /user
drwx------ - hdfs supergroup 0 2019-06-04 12:47 /user/hdfs
drwxr-xr-x - mapred hadoop 0 2019-05-27 20:06 /user/history
drwxr-xr-t - hive hive 0 2019-06-03 18:01 /user/hive
drwxr-xr-x - hue hue 0 2019-06-03 18:01 /user/hue
drwxr-xr-x - impala impala 0 2019-05-27 20:08 /user/impala
drwxr-xr-x - oozie oozie 0 2019-05-27 20:12 /user/oozie
drwxr-xr-x - spark spark 0 2019-05-27 20:06 /user/spark
Group details:
root:x:0:
bin:x:1:
daemon:x:2:
sys:x:3:
adm:x:4:
tty:x:5:
disk:x:6:
lp:x:7:
mem:x:8:
kmem:x:9:
wheel:x:10:ak_bng
cdrom:x:11:
mail:x:12:postfix
man:x:15:
dialout:x:18:
floppy:x:19:
games:x:20:
tape:x:33:
video:x:39:
ftp:x:50:
lock:x:54:
audio:x:63:
nobody:x:99:
users:x:100:
utmp:x:22:
utempter:x:35:
input:x:999:
systemd-journal:x:190:
systemd-network:x:192:
dbus:x:81:
polkitd:x:998:
ssh_keys:x:997:
sshd:x:74:
postdrop:x:90:
postfix:x:89:
printadmin:x:996:
dip:x:40:
cgred:x:995:
rpc:x:32:
libstoragemgmt:x:994:
unbound:x:993:
kvm:x:36:qemu
qemu:x:107:
chrony:x:992:
gluster:x:991:
rtkit:x:172:
radvd:x:75:
tss:x:59:
usbmuxd:x:113:
colord:x:990:
abrt:x:173:
geoclue:x:989:
saslauth:x:76:
libvirt:x:988:
pulse-access:x:987:
pulse-rt:x:986:
pulse:x:171:
gdm:x:42:
setroubleshoot:x:985:
gnome-initial-setup:x:984:
stapusr:x:156:
stapsys:x:157:
stapdev:x:158:
tcpdump:x:72:
avahi:x:70:
slocate:x:21:
ntp:x:38:
ak_bng:x:1000:
localadmin:x:1001:
am_bng:x:1002:
localuser:x:1003:
apache:x:48:
cassandra:x:983:
mysql:x:27:
cloudera-scm:x:982:
hadoop:x:1011:yarn,hdfs,mapred
postgres:x:26:
zookeeper:x:981:
yarn:x:980:
hdfs:x:979:
mapred:x:978:
kms:x:977:kms
httpfs:x:976:httpfs
hbase:x:975:
hive:x:974:impala
sentry:x:973:
solr:x:972:
sqoop:x:971:
flume:x:970:
spark:x:969:
oozie:x:968:
hue:x:967:
impala:x:966:
llama:x:965:
kudu:x:964:
I need to run sqoop scripts from command line as user ak_bng
As the vaishak user, sqoop wants to write to /user/hive/warehouse/sales.db. However, vaishak does not have permission to write to that directory, so sqoop throws
Permission denied by sticky bit: user=ak_bng, path="/user/hive/warehouse/sales.db/categories":hive:hive:drwxr-xr-t, parent="/user/hive/warehouse/sales.db":hive:hive:drwxr-xr-t
Try to specify a target directory that is owned by vaishak and rerun. For example: /user/vaishak/sales.db
I created a Parquet table(orders_parquet) from existing table(orders) with CTAS as below :
CREATE TABLE orders_parquet
ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
AS SELECT * FROM orders;
It is loading with below
Query ID = jonnavithulasivakrishna_20171105234912_e608ac1f-a10b-435e-8307-92747fb5c37d
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Can't load log handler "java.util.logging.FileHandler"
java.io.FileNotFoundException: /tmp/parquet-3.log (Permission denied)
java.io.FileNotFoundException: /tmp/parquet-3.log (Permission denied)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
at java.util.logging.FileHandler.open(FileHandler.java:210)
at java.util.logging.FileHandler.rotate(FileHandler.java:661)
at java.util.logging.FileHandler.openFiles(FileHandler.java:538)
at java.util.logging.FileHandler.<init>(FileHandler.java:263)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
But the table is loaded with data.Could you please help why the error is coming?
Hive need write permission in /tmp location to log information. In your case hive don't have write permission to /tmp/parquet-3.log. Give write permission to given file as chmod 777 /tmp/parquet-3.log
I'm getting the following error when executing a select count(*) from tablename query when connected to beeline.
ERROR : Job Submission failed with exception 'org.apache.hadoop.security.AccessControlException(Permission denied
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkOwner(FSPermissionChecker.java:201)
I can execute showtables; successfully but get this error anytime I execute a query. I am logged in as the hadoop user that has access to both hadoop and hive.
I've granted the folder where the tables resides full permissions:
drwxr-xr-x - hadoop supergroup 0 2015-06-03 15:44 /data1
drwxrwxrwx - hadoop hadoop 0 2015-06-05 15:23 /tmp
drwxrwxrwx - hadoop supergroup 0 2015-06-05 15:24 /user
The table is in the user directory.
Environment details:
OS: CentOS
Hadoop: HW 2.6.0
Hive: 1.2
Any help would be greatly appreciated.
Is this a hive managed table in that case could you print what you get when you do
hadoop fs -ls /user
hadoop fs -ls /user/hive
hadoop fs -ls /user/hive/warehouse
the error suggests that you are accessing a table from a user who is not the owner and seems like user does not have read and execute access
hduser#Connected:~$ hive
Logging initialized using configuration in jar:file:/usr/local/hive/lib/hive-common-0.14.0.jar!/hive-log4j.properties
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwx--x--x
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:444)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:672)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
Caused by: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwx--x--x
at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:529)
at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:478)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:430)
I have created the hive directories in hadoop as:
hadoop fs -mkdir /usr/hive/warehouse
and
set permissions for the table:
hadoop fs -chmod g+w /usr/hive/warehouse
but its still not working? what should I do?
Looks like the HDFS directory /tmp/hive is missing or doesn't have enough permission to write files inside. Execute the following command for assigning proper permission.
Switch to HDFS admin user first (sudo -su hdfs command can be used), then execute the following commands.
hadoop fs -chmod 777 /tmp;
hadoop fs -mkdir /tmp/hive;
hadoop fs -chmod -R 777 /tmp/hive;
Check the value for the below tag on hive-site.xml, then change the permission for the folder mentioned
<property>
<name>hive.exec.local.scratchdir</name>
<value>/tmp/mydir</value>
<description>Local scratch space for Hive jobs</description>
</property>
hadoop fs -rmr /tmp/mydir;
hadoop fs -mkdir /tmp/mydir;
hadoop fs -chmod 777 /tmp/mydir;
hadoop fs -chmod -R 777 /tmp/mydir;
I am having some difficulties running "standard" pig tutorial - pig script1-hadoop.pig
However, because of cluster set up (users), I had to modify an example a bit. Standard tutorial expects all files on / of HDFS, which I cannot use in my case, so I created /pig dir for that purpose
drwxrwxrwx - hdfs hdfs 0 2014-03-31 11:15 /pig
with the uploaded content
-rw-r--r-- 3 jakub hdfs 10408717 2014-03-31 10:41 /pig/excite.log.bz2
I also modified the pig script script1-hadoop.pig as well, to respect those changes as follows (mainly just for load and store commands):
raw = LOAD '/pig/excite.log.bz2' USING PigStorage('\t') AS (user, time, query);
...
STORE ordered_uniq_frequency INTO '/pig/script1-hadoop-results' USING PigStorage();
I run the pig script:
[jakub#hadooptools pigtmp]$ pig script1-hadoop.pig
but with no luck and getting error:
2014-03-31 10:15:11,896 [main] ERROR org.apache.pig.tools.grunt.Grunt - You don't have permission to perform the operation. Error from the server: Permission denied: user=jakub, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:234)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:214)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:158)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5202)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5184)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5158)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3405)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3375)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3349)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:724)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:502)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59598)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2053)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2047)
I am not quite sure why PIG script is trying to write into / on HDFS. I know that PIG can store some immediate results on HDFS so I modified pig.temp.dir property (/etc/pig/conf/pig.properties) and created location on HDFS /pig/tmp
drwxrwxrwx - jakub hdfs 0 2014-03-31 11:15 /pig/tmp
Any idea what might be wrong? Pig in local mode is ok.
Sorted.
User running Pig script has to have permissions to write to tmp directory created and /user/pig_user_running has to be present on the cluster as well with permissions allowing him to write there.
Super-user on HDFS is the user under which namenode process is running, which is typycally HDFS.