Facing issue in setting up oozie with secure MapR cluster - hadoop

Question

We are facing an issue with setting up an oozie service with secure mapr cluster.
We are using the MapR installer to setup the MapR Cluster. Below are the configuration and steps that we followed.
MapR version - 6.1
Os - Ubuntu 16.04
Authentication - Kerberos
Nodes - Single node
We have enabled the Mapr security by using the Enable Secure Cluster option in the installer.
Reference doc - https://docs.datafabric.hpe.com/61/AdvancedInstallation/using_enable_secure_cluster_option.html
We have installed the kerberos in the machine.
Reference doc - https://linuxconfig.org/how-to-install-kerberos-kdc-server-and-client-on-ubuntu-18-04
Below are the commands we executed to setup kerberos authentication for the MapR cluster
Reference docs -
https://docs.datafabric.hpe.com/61/SecurityGuide/Configuring-Kerberos-User-Authentication.html
https://docs.datafabric.hpe.com/61/SecurityGuide/ConfiguringSPNEGOonMapR.html
sudo kadmin.local
addprinc -randkey mapr/my.cluster.com
ktadd -k /opt/mapr/conf/mapr.keytab mapr/my.cluster.com
addprinc -randkey HTTP/<instance-name>#<realm-name>
ktadd -k /opt/mapr/conf/http.keytab HTTP/<instance-name>#<realm-name>
addprinc -randkey mapr/<instance-name>#<realm-name>
ktadd -k /opt/mapr/conf/mapr2.keytab mapr/<instance-name>#<realm-name>
sudo chown mapr:mapr /opt/mapr/conf/mapr.keytab /opt/mapr/conf/http.keytab /opt/mapr/conf/mapr2.keytab
sudo chmod 777 /opt/mapr/conf/mapr.keytab /opt/mapr/conf/http.keytab /opt/mapr/conf/mapr2.keytab
ktutil
rkt /opt/mapr/conf/mapr.keytab
rkt /opt/mapr/conf/http.keytab
rkt /opt/mapr/conf/mapr2.keytab
wkt /opt/mapr/conf/mapr.keytab
sudo /opt/mapr/server/configure.sh -N my.cluster.com -C <CLDB Node>:7222 -Z <ZookeeperNode>:5181 -K -P "mapr/my.cluster.com#<realm-name>"
Note:
The command which is mentioned in the doc (configure.sh -K -P "<cldbPrincipal>") throws error , but the above command works.
kinit
maprlogin kerberos
hadoop fs -ls
3.1 ) We are able to access the mapr file system.
3.2) We are using the below command to run a simple mapreduce job and it works fine.
hadoop jar /opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.0-mapr-1808.jar pi 16 1000
Oozie configuration with kerberos authentication
Reference doc - https://docs.datafabric.hpe.com/61/Oozie/ConfiguringOozieonaSecureCluster.html
We have added below properties in the oozie-site.xml
<property>
<name>oozie.authentication.type</name>
<value>kerberos</value>
<description>
Defines authentication used for Oozie HTTP endpoint.
Supported values are: simple | kerberos | #AUTHENTICATION_HANDLER_CLASSNAME#
</description>
</property>
<property>
<name>oozie.service.HadoopAccessorService.keytab.file</name>
<value>/opt/mapr/conf/mapr.keytab</value>
<description>
Location of the Oozie user keytab file.
</description>
</property>
<property>
<name>local.realm</name>
<value>{local.realm}</value>
<description>
Kerberos Realm used by Oozie and Hadoop. Using 'local.realm' aligns with Hadoop configuration
</description>
</property>
<property>
<name>oozie.service.HadoopAccessorService.kerberos.principal</name>
<value>mapr/<hostname>#${local.realm}</value>
<description>
Kerberos principal for Oozie service.
</description>
</property>
<property>
<name>oozie.authentication.kerberos.principal</name>
<value>HTTP/<hostname>#${local.realm}</value>
<description>
Indicates the Kerberos principal to be used for the HTTP endpoint. The principal MUST start with 'HTTP/' per the Kerberos HTTP SPNEGO specification.
</description>
</property>
We are checking the oozie status by using bin/oozie admin -status -auth KERBEROS command , we are getting below error.
java.io.IOException: Error while connecting Oozie server. No of retries = 1. Exception = Could not authenticate, Authentication failed, status: 302
Kindly help us to resolve this issue

Answer 1

Oozie is a frigging nightmare in general. Adding Kerberos won't make it easier. Just saying.
The issue that you are describing appears to be that some component isn't getting the memo about the Kerberos identity that you are using or doesn't have access/permissions to validate an access. This is a common problem and typically requires step-by-step interaction to work through what is known and what is not yet known (but often is assumed). I am definitely not an expert on these kinds of issues, however.
You have a really excellent problem report here which is exactly the sort of thing that the support team can use.
Do you have an active support or partner in place?