I want Hive to automatically acquire kerberos ticket whenever hive(More specifically hive-shell not hive-server) is executed and also renew it automatically in between if job run more then timeout of ticket.
I found similar functionality in Pig. See This. I tested and it is working it acquires ticket automatically from keytab I don't have to acquire it manually using kinit and then start job. It renews tickets also whenever needed as mentioned in doc.
On some research I came across user-name-handling-in-hadoop. I found out similar log statement of dumping configuration parameters of class UserGroupInformation when starting hive.
As i wanted it every time hive is executed I tried it putting in HADOOP_OPTS which looks like this
export HADOOP_OPTS="$HADOOP_OPTS -Djava.security.krb5.conf=/etc/krb5.conf -Dhadoop.security.krb5.principal=root#MSI.COM -Dhadoop.security.krb5.keytab=/etc/security/keytab/user.service.keytab"
but whenever I execute it. It dumps following parameters which means it is not considering principle and keytab may be property name can be wrong since I have used names I found in Pig. It is observed that krb5.conf property is taken into consider as changing name of conf file shows default realm can't found as it is not able to read correct conf file.
23/01/23 23:33:28 DEBUG security.UserGroupInformation: hadoop login commit
23/01/23 23:33:28 DEBUG security.UserGroupInformation: using kerberos user:null
23/01/23 23:33:28 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: root
23/01/23 23:33:28 DEBUG security.UserGroupInformation: Using user: "UnixPrincipal: root" with name root
23/01/23 23:33:28 DEBUG security.UserGroupInformation: User entry: "root"
23/01/23 23:33:28 DEBUG security.UserGroupInformation: Assuming keytab is managed externally since logged in from subject.
23/01/23 23:33:28 DEBUG security.UserGroupInformation: UGI loginUser:root (auth:KERBEROS)
For any guidance Thanks in Advance
I ultimately want whenever hive-shell or hive-cli is invoked it automatically request for Kerberos ticket and renew it if needed.
Related
I am trying to do a simple data move from a mapr fs origin to a mapr fs destination (this is not my use case, just doing this simple movement for testing purposes). When trying to validate this pipeline, the error message I see in the staging area is:
HADOOPFS_11 - Cannot connect to the filesystem. Check if the Hadoop FS location: 'maprfs:///mapr/mycluster.cluster.local' is valid or not: 'java.io.IOException: Provided Subject must contain a KerberosPrincipal
Tyring different variations of the hadoop fs URI field (eg. mfs:///mapr/mycluster.cluster.local, maprfs:///mycluster.cluster.local) does not seem to help. Looking at the logs after trying to validate, I see
2018-01-04 10:28:56,686 mfs2mfs/mapr2sqlserver850bfbf0-6dc0-4002-8d44-b73e33fcf9b3 INFO Created source of type: com.streamsets.pipeline.stage.origin.maprfs.ClusterMapRFSSource#16978460 DClusterSourceOffsetCommitter *admin preview-pool-1-thread-3
2018-01-04 10:28:56,697 mfs2mfs/mapr2sqlserver850bfbf0-6dc0-4002-8d44-b73e33fcf9b3 INFO Error connecting to FileSystem: java.io.IOException: Provided Subject must contain a KerberosPrincipal ClusterHdfsSource *admin preview-pool-1-thread-3
java.io.IOException: Provided Subject must contain a KerberosPrincipal
....
2018-01-04 10:20:39,159 mfs2mfs/mapr2mapr850bfbf0-6dc0-4002-8d44-b73e33fcf9b3 INFO Authentication Config: ClusterHdfsSource *admin preview-pool-1-thread-3
2018-01-04 10:20:39,159 mfs2mfs/mapr2mapr850bfbf0-6dc0-4002-8d44-b73e33fcf9b3 ERROR Issues: Issue[instance='MapRFS_01' service='null' group='HADOOP_FS' config='null' message='HADOOPFS_11 - Cannot connect to the filesystem. Check if the Hadoop FS location: 'maprfs:///mapr/mycluster.cluster.local' is valid or not: 'java.io.IOException: Provided Subject must contain a KerberosPrincipal''] ClusterHdfsSource *admin preview-pool-1-thread-3
2018-01-04 10:20:39,169 mfs2mfs/mapr2mapr850bfbf0-6dc0-4002-8d44-b73e33fcf9b3 INFO Validation Error: Failed to configure or connect to the 'maprfs:///mapr/mycluster.cluster.local' Hadoop file system: java.io.IOException: Provided Subject must contain a KerberosPrincipal HdfsTargetConfigBean *admin 0 preview-pool-1-thread-3
java.io.IOException: Provided Subject must contain a KerberosPrincipal
....
However, to my knowledge, the system is not running Keberos, so this error message is a bit confusing for me. Uncommenting #export SDC_JAVA_OPTS="-Dmaprlogin.password.enabled=true ${SDC_JAVA_OPTS}" in the sdc environment variable file for native mapr authentication did not seem to help the problem (even when reinstalling and commenting this line before running the streamsets mapr setup script).
Does anyone have any idea what is happening and how to fix it? Thanks.
This answer was provided on the mapr community forums and worked for me (using mapr v6.0). Note that the instruction here differ from those currently provided by the streamsets documentation. Throughout these instructions, I was logged in as user root.
After installing streamsets (and the mapr prerequisites) as per the documentation...
Change the owner of the the streamsets $SDC_DIST or $SDC_HOME location to the mapr user (or whatever other user you plan to use for the hadoop impersonation): $chown -R mapr:mapr $SDC_DIST (for me this was the /opt/streamsets-datacollector dir.). Do the same for $SDC_CONF (/etc/sdc for me) as well as /var/lib/sdc and var/log/sdc.
In $SDC_DIST/libexec/sdcd-env.sh, set the user and group name (near the top of the file) to mapr user "mapr" and enable mapr password login. The file should end up looking like:
# user that will run the data collector, it must exist in the system
#
export SDC_USER=mapr
# group of the user that will run the data collector, it must exist in the system
#
export SDC_GROUP=mapr
....
# Indicate that MapR Username/Password security is enabled
export SDC_JAVA_OPTS="-Dmaprlogin.password.enabled=true ${SDC_JAVA_OPTS}
Edit the file /usr/lib/systemd/system/sdc.service to look like:
[Service]
User=mapr
Group=mapr
$cd into /etc/systemd/system/ and create a directory called sdc.service.d. Within that directory, create a file (with any name) and add the contents (without spaces):
Environment=SDC_JAVA_OPTS=-Dmaprlogin.passowrd.enabled=true
If you are using mapr's sasl ticket auth. system (or something similar), generate a ticket for the this user on the node that is running streamsets. In this case, with the $maprlogin password command.
Then finally, restart the sdc service: $systemctl deamon-reload then $systemctl retart sdc.
Run something like $ps -aux | grep sdc | grep maprlogin to check that the sdc process is ownned by mapr and that the -Dmaprlogin.passowrd.enabled=true parameter has been successfully set. Once this is done, should be able to validate/run maprFS to maprFS operations in streamsets pipeline builder in batch processing mode.
** NOTE: If using Hadoop Configuration Directory param. instead of Hadoop FS URI, remember to have the files in your $HADOOP_HOME/conf directory (eg.hadoop-site.xml, yarn-site.xml, etc.) (in the case of mapr, something like /opt/mapr/hadoop/hadoop-<version>/etc/hadoop/) either soft-linked or hard-copied to a directory $SDC_DIST/resource/<some hadoop config dir. you made need to create> (I just copy eberything in the directory) and add this path to the Hadoop Configuration Directory param. for your MaprFS (or HadoopFS). In the sdc web UI Hadoop Configuration Directory box, it would look like Hadoop Configuration Directory: <the directory within $SDC_DIST/resources/ that holds the hadoop files>.
** NOTE: If you are still logging errors of the form
2018-01-16 14:26:10,883
ingest2sa_demodata_batch/ingest2sademodatabatchadca8442-cb00-4a0e-929b-df2babe4fd41
ERROR Error in Slave Runner: ClusterRunner *admin
runner-pool-2-thread-29
com.streamsets.datacollector.runner.PipelineRuntimeException:
CONTAINER_0800 - Pipeline
'ingest2sademodatabatchadca8442-cb00-4a0e-929b-df2babe4fd41'
validation error : HADOOPFS_11 - Cannot connect to the filesystem.
Check if the Hadoop FS location: 'maprfs:///' is valid or not:
'java.io.IOException: Provided Subject must contain a
KerberosPrincipal'
you may also need to add -Dmaprlogin.password.enabled=true to the pipeline's /cluster/Worker Java Options tab for the origin and destination hadoop FS stages.
** The video linked to in the mapr community link also says to generate a mapr ticket for the sdc user (the default user that sdc process runs as when running as a service), but I did not do this and the solution still worked for me (so if anyone has any idea why it should be done regardless, please let me know in the comments).
My Hive shell hangs at logging initialization at configuration
[cloudera#quickstart hive]$ hive
2017-03-01 08:23:50,909 WARN [main] mapreduce.TableMapReduceUtil: The hbase-prefix-tree module jar containing PrefixTreeCodec is not present. Continuing without it.
Logging initialized using configuration in file:/etc/hive/conf.dist/hive-log4j.properties
This is the log file description.
2017-02-28 08:56:34,685 WARN [main]: hive.metastore
(HiveMetaStoreClient.java:open(448)) - set_ugi() not successful,
Likely cause: new client talking to old server. Continuing without it.
org.apache.thrift.transport.TTransportException:
java.net.SocketTimeoutException: Read timed out at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
I also had this problem when I started hive cli. I tried to find some ways to solve the problem, but it didn't work. When I changed the metastore's version (for example, mysql8 to mysql5.6, hive version is 1.10 and don't change) , the problem has been solved!!!!
the log file has already said new client talking to old server. So you should change your metastore's version to a new one. It should be kept in mind that your metastore's version match hive's version.
I have an RDD that I wish to write to HDFS.
data.saveAsTextFile("hdfs://path/vertices")
This returns:
WARN RetryInvocationHandler: Exception while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over null. Not retrying because try once and fail.
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]
I have checked KERBEROS and it is properly authenticated.
How do I solve this?
Well,
You need to check your path /etc/security/keytabs and check if your spark keytab is there.
This path is the recommended to the Kerberos configuration. Maybe it can be in other path.
But the most important, this keytab should be in all workers machines in the same path.
Other thing that you can check is the configuration file of Spark that should be installed in:
SPARK_HOME/conf
This folder should have the spark conf file spark-defaults.conf this conf file need to have this stuffs:
spark.history.kerberos.enabled true
spark.history.kerberos.keytab /etc/security/keytabs/spark.keytab
spark.history.kerberos.principal user#DOMAIN.LOCAL
The issue was actually related to how you reference a file in HDFS when using kerberos.
Rather than hdfs://<HOST>:<HTTP_PORT>
It is webhdfs://<HOST>:<HTTP_PORT>
Configured oozie to work with hadoop-2.6.0 and enabled kerberos security.
I didn't get ticket using kinit command but when i submit job using below command,
oozie job -oozie http://hostname:11000/oozie -config job.properties -run
it throws the following exception,
Error: E0501 : E0501: Could not perform authorization operation, User: oozie/hostname#EXAMPLE.COM is not allowed to impersonate Kumar
I know how to solve the above error but my question is
Kumar is my local account username. As i configured kerberos, it should check my user ticket. But it didn't show me any error like "No credential found"
If i get ticket using kinit for any other user then also oozie shows the same exception with my local user account name.
Is there anything to configure? I don't understand the concept. I am following this to configure oozie with kerberos on secured cluster.
I just found the answer in Oozie Authentication
Once authentication is performed successfully the received authentication token is cached in the user home directory in the .oozie-auth-token file with owner-only permissions. Subsequent requests reuse the cached token while valid.
This is the reason for using invalid user even getting the ticket for any other user using kinit command.
I just resolved as below
The use of the cache file can be disabled by invoking the oozie CLI with the -Doozie.auth.token.cache false= option.
Try this.
I was trying to enable Kerberos for my CDH 4.3 (via Cloudera Manager) test bed. So after changing authentication from Simple to Kerberos in the WebUI, I'm unable to do any hadoop operations as shown below. Is there anyway to specify the keytab explicitly?
[root#host-dn15 ~]# su - hdfs
-bash-4.1$ hdfs dfs -ls /
13/09/10 08:15:35 ERROR security.UserGroupInformation: PriviledgedActionException as:hdfs (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
13/09/10 08:15:35 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
13/09/10 08:15:35 ERROR security.UserGroupInformation: PriviledgedActionException as:hdfs (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "host-dn15.hadoop.com/192.168.10.227"; destination host is: "host-dn15.hadoop.com":8020;
-bash-4.1$ kdestroy
-bash-4.1$ kinit
Password for hdfs#HADOOP.COM:
-bash-4.1$ klist
Ticket cache: FILE:/tmp/krb5cc_494
Default principal: hdfs#HADOOP.COM
Valid starting Expires Service principal
09/10/13 08:20:31 09/11/13 08:20:31 krbtgt/HADOOP.COM#HADOOP.COM
renew until 09/10/13 08:20:31
-bash-4.1$ klist -e
Ticket cache: FILE:/tmp/krb5cc_494
Default principal: hdfs#HADOOP.COM
Valid starting Expires Service principal
09/10/13 08:20:31 09/11/13 08:20:31 krbtgt/HADOOP.COM#HADOOP.COM
renew until 09/10/13 08:20:31, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
-bash-4.1$
So I took a good look at the namenode log,
2013-09-10 10:02:06,085 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8022: readAndProcess threw exception javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Encryption type AES256 CTS mode with HMAC SHA1-96 is not supported/enabled)] from client 10.132.100.228. Count of bytes read: 0
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Encryption type AES256 CTS mode with HMAC SHA1-96 is not supported/enabled)]
JCE policy files are Installed already on all nodes.
[root#host-dn15 security]# sha256sum ./local_policy.jar
4a5c8f64107c349c662ea688563e5cd07d675255289ab25246a3a46fc4f85767 ./local_policy.jar
[root#host-dn15 security]# sha256sum ./US_export_policy.jar
b800fef6edc0f74560608cecf3775f7a91eb08d6c3417aed81a87c6371726115 ./US_export_policy.jar
[root#host-dn15 security]# sha256sum ./local_policy.jar.bak
7b26d0e16722e5d84062240489dea16acef3ea2053c6ae279933499feae541ab ./local_policy.jar.bak
[root#host-dn15 security]# sha256sum ./US_export_policy.jar.bak
832133c52ed517df991d69770f97c416d2e9afd874cb4f233a751b23087829a3 ./US_export_policy.jar.bak
[root#host-dn15 security]#
And the list of principals in the realm.
kadmin: listprincs
HTTP/host-dn15.hadoop.com#HADOOP.COM
HTTP/host-dn16.hadoop.com#HADOOP.COM
HTTP/host-dn17.hadoop.com#HADOOP.COM
K/M#HADOOP.COM
cloudera-scm/admin#HADOOP.COM
hbase/host-dn15.hadoop.com#HADOOP.COM
hbase/host-dn16.hadoop.com#HADOOP.COM
hbase/host-dn17.hadoop.com#HADOOP.COM
hdfs/host-dn15.hadoop.com#HADOOP.COM
hdfs/host-dn16.hadoop.com#HADOOP.COM
hdfs/host-dn17.hadoop.com#HADOOP.COM
hdfs#HADOOP.COM
hue/host-dn15.hadoop.com#HADOOP.COM
host-dn16/hadoop.com#HADOOP.COM
kadmin/admin#HADOOP.COM
kadmin/changepw#HADOOP.COM
kadmin/host-dn15.hadoop.com#HADOOP.COM
krbtgt/HADOOP.COM#HADOOP.COM
mapred/host-dn15.hadoop.com#HADOOP.COM
mapred/host-dn16.hadoop.com#HADOOP.COM
mapred/host-dn17.hadoop.com#HADOOP.COM
root/admin#HADOOP.COM
root#HADOOP.COM
zookeeper/host-dn15.hadoop.com#HADOOP.COM
kadmin: exit
[root#host-dn15 ~]#
exported the keytab for hdfs and used to kinit.
-bash-4.1$ kinit -kt ./hdfs.keytab hdfs
-bash-4.1$ klist
Ticket cache: FILE:/tmp/krb5cc_494
Default principal: hdfs#HADOOP.COM
Valid starting Expires Service principal
09/10/13 09:49:42 09/11/13 09:49:42 krbtgt/HADOOP.COM#HADOOP.COM
renew until 09/10/13 09:49:42
Everything went futile. Any idea??
Thanks ahead,
I ran into a problem in which I had a Kerberized CDH cluster and even with a valid Kerberos ticket, I couldn't run any hadoop commands from the command line.
NOTE: After writing this answer I wrote it up as a blog post at http://sarastreeter.com/2016/09/26/resolving-hadoop-problems-on-kerberized-cdh-5-x/ . Please share!
So even with a valid ticket, this would fail:
$ hadoop fs -ls /
WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
Here is what I learned and how I ended up resolving the problem. I have linked to Cloudera doc for the current version where possible, but some of the doc seems to be present only for older versions.
Please note that the problem comes down to a configuration issue but that Kerberos itself and Cloudera Manager were both installed correctly. Many of the problems I ran across while searching for answers came down to Kerberos or Hadoop being installed incorrectly. The problem I had occurred even though both Hadoop and Kerberos were functional, but they were not configured to work together properly.
TL;DR
MAKE SURE YOU HAVE A TICKET
Do a klist from the user you are trying to execute the hadoop command.
$ sudo su - myuser
$ klist
If you don't have a ticket, it will print:
klist: Credentials cache file '/tmp/krb5cc_0' not found
If you try to do a hadoop command without a ticket you will get the GSS INITIATE FAILED error by design:
WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
In other words, that is not an install problem. If this is your situation, take a look at:
http://www.roguelynn.com/words/explain-like-im-5-kerberos/
For other troubleshooting of Kerberos in general, check out https://steveloughran.gitbooks.io/kerberos_and_hadoop/content/sections/errors.html
CDH DEFAULT HDFS USER AND GROUP RESTRICTIONS
A default install of Cloudera has user and group restrictions on execution of hadoop commands, including a specific ban on certain users ( more on page 57 of http://www.cloudera.com/documentation/enterprise/5-6-x/PDF/cloudera-security.pdf ).
There are several properties that deal with this, including the supergroup for hdfs being set to the string supergroup instead of hdfs, dfs_permissions enabled property being set to false by default (hadoop user file permissions), users with uid over 1000 being banned.
Any of these could be a factor, for me it was HDFS being listed in the banned.users property.
Specifically for user HDFS, make sure you have removed hdfs from the banned.users configuration property in hdfs-site.xml configuration if you are trying to use it to execute hadoop commands.
1) UNPRIVILEGED USER AND WRITE PERMISSIONS
The Cloudera-recommended way to execute Hadoop commands is to create an unprivileged user and matching principal, instead of using the hdfs user. A gotcha is that this user also needs its own /user directory and can run into write permissions errors with the /user directory. If your unprivileged user does not have a directory in /user, it may result in the WRITE permissions denied error.
Cloudera Knowledge Article
http://community.cloudera.com/t5/CDH-Manual-Installation/How-to-resolve-quot-Permission-denied-quot-errors-in-CDH/ta-p/36141
2) DATANODE PORTS AND DATA DIR PERMISSIONS
Another related issue is that Cloudera sets dfs.datanode.data.dir to 750 on a non-kerberized cluster, but requires 700 on a kerberized cluster. With the wrong dir permissions set, the Kerberos install will fail. The ports for the datanodes must also be set to values below 1024, which are recommended as 1006 for the HTTP port and 1004 for the Datanode port.
Datanode Directory
http://www.cloudera.com/documentation/enterprise/5-6-x/topics/cdh_ig_hdfs_cluster_deploy.html
Datanode Ports
http://www.cloudera.com/documentation/archive/manager/4-x/4-7-2/Configuring-Hadoop-Security-with-Cloudera-Manager/cmchs_enable_security_s9.html
3) SERVICE-SPECIFIC CONFIGURATION TASKS
On page 60 of the security doc, there are steps to kerberize Hadoop services. Make sure you did these!
MapReduce
$ sudo -u hdfs hadoop fs -chown mapred:hadoop ${mapred.system.dir}
HBase
$ sudo -u hdfs hadoop fs -chown -R hbase ${hbase.rootdir}
Hive
$ sudo -u hdfs hadoop fs -chown hive /user/hive
YARN
$ rm -rf ${yarn.nodemanager.local-dirs}/usercache/*
All of these steps EXCEPT for the YARN one can happen at any time. The step for YARN must happen after Kerberos installation because what it is doing is removing the user cache for non-kerberized YARN data. When you run mapreduce after the Kerberos install it should populate this with the Kerberized user cache data.
YARN User Cache
YARN Application exited with exitCode: -1000 Not able to initialize user directories
KERBEROS PRINCIPAL ISSUES
1) SHORT NAME RULES MAPPING
Kerberos principals are "mapped" to the OS-level services users. For example, hdfs/WHATEVER#REALM maps to the service user 'hdfs' in your operating system only because of a name mapping rule set in the core-site of Hadoop. Without name mapping, Hadoop wouldn't know which user is authenticated by which principal.
If you are using a principal that should map to hdfs, make sure the principal name resolves correctly to hdfs according to these Hadoop rules.
Good
(has a name mapping rule by default)
hdfs#REALM
hdfs/_HOST#REALM
Bad
(no name mapping rule by default)
hdfs-TAG#REALM
The "bad" example will not work unless you add a rule to accommodate it
Name Rules Mapping
http://www.cloudera.com/documentation/archive/cdh/4-x/4-5-0/CDH4-Security-Guide/cdh4sg_topic_19.html )
2) KEYTAB AND PRINCIPAL KEY VERSION NUMBERS MUST MATCH
The Key Version Number (KVNO) is the version of the key that is actively being used (as if you had a house key but then changed the lock on the door so it used a new key, the old one is no longer any good). Both the keytab and principal have a KVNO and the version number must match.
By default, when you use ktadd or xst to export the principal to a keytab, it changes the keytab version number, but does not change the KVNO of the principal. So you can end up accidentally creating a mismatch.
Use -norandkey with kadmin or kadmin.local when exporting a principal to a keytab to avoid updating the keytab number and creating a KVNO mismatch.
In general, whenever having principal issues authentication issues, make sure to check that the KVNO of the principal and keytab match:
Principal
$ kadmin.local -q 'getprinc myprincipalname'
Keytab
$ klist -kte mykeytab
Creating Principals
http://www.cloudera.com/documentation/archive/cdh/4-x/4-3-0/CDH4-Security-Guide/cdh4sg_topic_3_4.html
SECURITY JARS AND JAVA HOME
1) JAVA VERSION MISMATCH WITH JCE JARS
Hadoop needs the Java security JCE Unlimited Strength jars installed in order to use AES-256 encryption with Kerberos. Both Hadoop and Kerberos need to have access to these jars. This is an install issue but it is easy to miss because you can think you have the security jars installed when you really don't.
JCE Configurations to Check:
the jars are the right version - the correct security jars are bundled with Java, but if you install them after the fact you have to make sure the version of the jars corresponds to the version of Java or you will continue to get errors. To troubleshoot, check the md5sum hash from a brand new download of the JDK that you're using in against the md5sum hash of the ones on the Kerberos server.
the jars are in the right location $JAVA_HOME/jre/lib/security
Hadoop is configured to look for them in the right place. Check if there is an export statement for $JAVA_HOME to the correct Java install location in /etc/hadoop/conf/hadoop-env.sh
If Hadoop has JAVA_HOME set incorrectly it will fail with "GSS INITIATE FAILED". If the jars are not in the right location, Kerberos won't find them and will give an error that it doesn't support the AES-256 encryption type (UNSUPPORTED ENCTYPE).
Cloudera with JCE Jars
http://www.cloudera.com/documentation/enterprise/5-5-x/topics/cm_sg_s2_jce_policy.html
Troubleshooting JCE Jars
https://community.cloudera.com/t5/Cloudera-Manager-Installation/Problem-with-Kerberos-amp-user-hdfs/td-p/6809
TICKET RENEWAL WITH JDK 6 AND MIT KERBEROS 1.8.1 AND HIGHER
Cloudera has an issue documented at http://www.cloudera.com/documentation/archive/cdh/3-x/3u6/CDH3-Security-Guide/cdh3sg_topic_14_2.html in which tickets must be renewed before hadoop commands can be issued. This only happens with Oracle JDK 6 Update 26 or earlier and package version 1.8.1 or higher of the MIT Kerberos distribution.
To check the package, do an rpm -qa | grep krb5 on CentOS/RHEL or aptitude search krb5 -F "%c %p %d %V" on Debian/Ubuntu.
So do a regular kinit as you would, then do a kinit -R to force the ticket to be renewed.
$ kinit -kt mykeytab myprincipal
$ kinit -R
And finally, the issue I actually had which I could not find documented anywhere ...
CONFIGURATION FILES AND TICKET CACHING
There are two important configuration files for Kerberos, the krb5.conf and the kdc.conf. These are configurations for the krb5kdc service and the KDC database. My problem was the krb5.conf file had a property:
default_ccache_name = KEYRING:persistent:%{uid}.
This set my cache name to KEYRING:persistent and user uid (explained https://web.mit.edu/kerberos/krb5-1.13/doc/basic/ccache_def.html). When I did a kinit, it created the ticket in /tmp because the cache name was being set elsewhere as /tmp. Cloudera services obtain authentication with files generated at runtime in /var/run/cloudera-scm-agent/process , and these all export the cache name environment variable (KRB5CCNAME) before doing their kinit. That's why Cloudera could obtain tickets but my hadoop user couldn't.
The solution was to remove the line from krb5.conf that set default_ccache_name and allow kinit to store credentials in /tmp, which is the MIT Kerberos default value DEFCCNAME (documented at https://web.mit.edu/kerberos/krb5-1.13/doc/mitK5defaults.html#paths).
Cloudera and Kerberos installation guides:
Step-by-Step
https://www.cloudera.com/documentation/enterprise/5-6-x/topics/cm_sg_intro_kerb.html .
Advanced troubleshooting
http://www.cloudera.com/documentation/enterprise/5-6-x/PDF/cloudera-security.pdf, starting on page 48 .