user authenticate issue In kerberos with keytab - hadoop

Question

I am trying to integrate Kerberos Hadoop with Pinot.and using below configurations.
Executables:
export HADOOP_HOME=/usr/hdp/2.6.3.0-235/hadoop
export HADOOP_VERSION=2.7.3.2.6.3.0-235
export HADOOP_GUAVA_VERSION=11.0.2
export HADOOP_GSON_VERSION=2.2.4
export GC_LOG_LOCATION=/home/hdfs/Pinot/pinotGcLog
export PINOT_VERSION=0.7.1
export PINOT_DISTRIBUTION_DIR=/home/hdfs/Pinot_IMP_FOLDER/apache-pinot-incubating-0.7.1-bin
export HADOOP_CLIENT_OPTS="-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins -Dlog4j2.configurationFile=${PINOT_DISTRIBUTION_DIR}/conf/pinot-ingestion-job-log4j2.xml"
export SERVER_CONF_DIR=/home/hdfs/Pinot_IMP_FOLDER/apache-pinot-incubating-0.7.1-bin/bin
export ZOOKEEPER_ADDRESS=<ZOOKEEPER_ADDRESS>
export CLASSPATH_PREFIX="${HADOOP_HOME}/hadoop-hdfs/hadoop-hdfs-${HADOOP_VERSION}.jar:${HADOOP_HOME}/hadoop-annotations-${HADOOP_VERSION}.jar:${HADOOP_HOME}/hadoop-auth-${HADOOP_VERSION}.jar:${HADOOP_HOME}/hadoop-common-${HADOOP_VERSION}.jar:${HADOOP_HOME}/lib/guava-${HADOOP_GUAVA_VERSION}.jar:${HADOOP_HOME}/lib/gson-${HADOOP_GSON_VERSION}.jar"
export JAVA_OPTS="-Xms4G -Xmx16G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:${GC_LOG_LOCATION}/gc-pinot-server.log"
controller.conf
controller.data.dir=<fs.defaultFS>/user/hdfs/controller_segment
controller.local.temp.dir=/home/hdfs/Pinot/pinot_tmp/
controller.zk.str=<ZOOKEEPER_ADDRESS>
controller.enable.split.commit=true
controller.access.protocols.http.port=9000
controller.helix.cluster.name=PinotCluster
pinot.controller.storage.factory.class.hdfs=org.apache.pinot.plugin.filesystem.HadoopPinotFS
pinot.controller.storage.factory.hdfs.hadoop.conf.path=/usr/hdp/2.6.3.0-235/hadoop/conf
pinot.controller.segment.fetcher.protocols=file,http,hdfs
pinot.controller.segment.fetcher.hdfs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.controller.segment.fetcher.hdfs.hadoop.kerberos.principle='hdfs#HDFSSITHDP.COM'
pinot.controller.segment.fetcher.hdfs.hadoop.kerberos.keytab='/home/hdfs/hdfs.keytab'
pinot.controller.storage.factory.hdfs.hadoop.kerberos.principle='hdfs#HDFSSITHDP.COM'
pinot.controller.storage.factory.hdfs.hadoop.kerberos.keytab='/home/hdfs/hdfs.keytab'
controller.vip.port=9000
controller.port=9000
pinot.set.instance.id.to.hostname=true
pinot.server.grpc.enable=true
Kerbeous Information:
kinit -V -k -t /home/hdfs/hdfs.keytab hdfs#HDFSSITHDP.COM
Using default cache: /tmp/krb5cc_57372
Using principal: hdfs#HDFSSITHDP.COM
Using keytab: /home/hdfs/hdfs.keytab
Authenticated to Kerberos v5
ERROR MESSAGE:
END: Invoking TASK controller pipeline for event ResourceConfigChange::15fc3764_TASK for cluster PinotCluster, took 278 ms
START AsyncProcess: TASK::TaskGarbageCollectionStage
END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms
Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Trying to authenticate user 'hdfs#HDFSSITHDP.COM' with keytab '/home/hdfs/hdfs.keytab'..
Could not instantiate file system for class org.apache.pinot.plugin.filesystem.HadoopPinotFS with scheme hdfs
java.lang.RuntimeException: Failed to authenticate user principal ['hdfs#HDFSSITHDP.COM'] with keytab ['/home/hdfs/hdfs.keytab']
at org.apache.pinot.plugin.filesystem.HadoopPinotFS.authenticate(HadoopPinotFS.java:258) ~[pinot-hdfs-0.7.1-shaded.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
Caused by: java.io.IOException: Login failure for 'hdfs#HDFSSITHDP.COM' from keytab '/home/hdfs/hdfs.keytab': javax.security.auth.login.LoginException: Unable to obtain password from user
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:962) ~[pinot-orc-0.7.1-shaded.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
at org.apache.pinot.plugin.filesystem.HadoopPinotFS.authenticate(HadoopPinotFS.java:254) ~[pinot-hdfs-0.7.1-shaded.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
... 15 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:901) ~[?:1.8.0_241]
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:764) ~[?:1.8.0_241]
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617) ~[?:1.8.0_241]
at org.apache.pinot.plugin.filesystem.HadoopPinotFS.authenticate(HadoopPinotFS.java:254) ~[pinot-hdfs-0.7.1-shaded.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
... 15 more
Failed to start a Pinot [CONTROLLER] at 21.954 since launch
java.lang.RuntimeException: java.lang.RuntimeException: Failed to authenticate user principal ['hdfs#HDFSSITHDP.COM'] with keytab ['/home/hdfs/hdfs.keytab']
at org.apache.pinot.spi.filesystem.PinotFSFactory.register(PinotFSFactory.java:58) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
P.s. I am executing this hdfs user and for keytab file also user is hdfs .i have also given 777 access to hdfs.keytab file.
someone Kindly suggest What is the issue here.I have read multiple blocks and everywhere found that it is because of wrong prinicpal/keytab file combination/user don't have access/give 777 access to file/try with different user. tried all the the options but nothing worked as of now.

Answer 1

It Worked now.I just Removed ' from keytab and principle name. it was unable to read keytab with '.
Old Configuration:
pinot.controller.segment.fetcher.hdfs.hadoop.kerberos.principle='hdfs#HDFSSITHDP.COM'
pinot.controller.segment.fetcher.hdfs.hadoop.kerberos.keytab='/home/hdfs/hdfs.keytab'
pinot.controller.storage.factory.hdfs.hadoop.kerberos.principle='hdfs#HDFSSITHDP.COM'
pinot.controller.storage.factory.hdfs.hadoop.kerberos.keytab='/home/hdfs/hdfs.keytab'
New Configuration:
pinot.controller.segment.fetcher.hdfs.hadoop.kerberos.principle=hdfs#HDFSSITHDP.COM
pinot.controller.segment.fetcher.hdfs.hadoop.kerberos.keytab=/home/hdfs/hdfs.keytab
pinot.controller.storage.factory.hdfs.hadoop.kerberos.principle=hdfs#HDFSSITHDP.COM
pinot.controller.storage.factory.hdfs.hadoop.kerberos.keytab=/home/hdfs/hdfs.keytab