Apache Phoenix query server thin client with Kerberos - hadoop

Question

I'm trying to setup Apache Phoenix QueryServer in secure HBase environment.
My hbase-site.xml is:
<configuration>
<property>
<name>hbase.regionserver.wal.codec</name>
<value>org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>zk1,zk2,zk3</value>
</property>
<property>
<name>hbase.security.authentication</name>
<value>kerberos</value>
</property>
<property>
<name>phoenix.queryserver.keytab.file</name>
<value>/usr/lib/apache-phoenix-queryserver/bin/phoenix.keytab</value>
</property>
<property>
<name>phoenix.queryserver.kerberos.principal</name>
<value>rwqueryserver/_HOST#FOO.BAR</value>
</property>
<property>
<name>phoenix.queryserver.http.keytab.file</name>
<value>/usr/lib/apache-phoenix-queryserver/bin/phoenix.keytab</value>
</property>
<property>
<name>phoenix.queryserver.http.kerberos.principal</name>
<value>rwqueryserver/_HOST#FOO.BAR</value>
</property>
</configuration>
Query server is starting without any problems.
2018-12-12 09:13:07,353 INFO org.apache.phoenix.queryserver.server.QueryServer: Login successful.
I checked KDC side and I can see logins from the principal used for server/client connections. No errors on KDC side as well.
Thin client command is:
./sqlline-thin.py 'http://dns-of-query-server:8765;principal="rwqueryserver/dns-of-query-server#DATASYS.CF.WTF";keytab="/usr/lib/apache-phoenix-queryserver/bin/phoenix.keytab"'
I tried to use user's principal and server's principal - situation is the same.
java.lang.RuntimeException: Failed to execute HTTP Request, got HTTP/404
From the queryserver log:
2018-12-12 09:15:30,987 WARN org.apache.phoenix.shaded.org.eclipse.jetty.security.SpnegoLoginService:
GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)
at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:856)
at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
at sun.security.jgss.spnego.SpNegoContext.GSS_acceptSecContext(SpNegoContext.java:906)
at sun.security.jgss.spnego.SpNegoContext.acceptSecContext(SpNegoContext.java:556)
at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
at org.apache.phoenix.shaded.org.eclipse.jetty.security.SpnegoLoginService.login(SpnegoLoginService.java:137)
at org.apache.phoenix.shaded.org.eclipse.jetty.security.authentication.LoginAuthenticator.login(LoginAuthenticator.java:61)
at org.apache.phoenix.shaded.org.eclipse.jetty.security.authentication.SpnegoAuthenticator.validateRequest(SpnegoAuthenticator.java:99)
at org.apache.phoenix.shaded.org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:512)
at org.apache.phoenix.shaded.org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
at org.apache.phoenix.shaded.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.apache.phoenix.shaded.org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.apache.phoenix.shaded.org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
at org.apache.phoenix.shaded.org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.apache.phoenix.shaded.org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: KrbException: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96
at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:278)
at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:149)
at sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:108)
at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:829)
... 19 more
When I'm trying to use non-existing principal I'm getting different errors.
I checked JCE - it's installed.
jrunscript -e 'print (javax.crypto.Cipher.getMaxAllowedKeyLength("AES") >= 256);'
true
Can you advice anything ?

Answer 1

This looks like a match to a known Java bug.
Please refer; https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6907425
Have a look at the above documentation and try again with an up to date JDK version.