Secured Nifi Cluster Setup - apache-nifi

I am trying to configure the 3node secured Nifi cluster setup by followinng the below Link .
But between nodes the connection not happened after enabled SSL/LDAP and i am getting the below error.
2017-04-01 09:05:47,494 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'HEARTBEAT' protocol message due to: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2017-04-01 09:05:47,494 ERROR [Process Cluster Protocol Request-7] o.a.nifi.security.util.CertificateUtils The incoming request did not contain client certificates and thus the DN cannot be extracted. Check that the other endpoint is providing a complete client certificate chain
2017-04-01 09:05:47,494 WARN [Process Cluster Protocol Request-7] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from HKLPATHAS02.hk.example.com due to org.apache.nifi.cluster.protocol.ProtocolException: java.security.cert.CertificateException: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
org.apache.nifi.cluster.protocol.ProtocolException: java.security.cert.CertificateException: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.getRequestorDN(SocketProtocolListener.java:221) ~[nifi-framework-cluster-protocol-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
at org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:133) ~[nifi-framework-cluster-protocol-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
at org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:136) [nifi-socket-utils-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_102]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_102]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
Caused by: java.security.cert.CertificateException: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at org.apache.nifi.security.util.CertificateUtils.extractPeerDNFromClientSSLSocket(CertificateUtils.java:306) ~[nifi-security-utils-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
at org.apache.nifi.security.util.CertificateUtils.extractPeerDNFromSSLSocket(CertificateUtils.java:261) ~[nifi-security-utils-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
at org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.getRequestorDN(SocketProtocolListener.java:219) ~[nifi-framework-cluster-protocol-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
... 5 common frames omitted
Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:431) ~[na:1.8.0_102]
at org.apache.nifi.security.util.CertificateUtils.extractPeerDNFromClientSSLSocket(CertificateUtils.java:291) ~[nifi-security-utils-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2]
... 7 common frames omitted
Please guide me to resolve this thread.

The error is saying that one node in the cluster is attempting to make a heartbeat connection to the other node, but it is not providing a valid client certificate in order to authenticate itself during the TLS handshake negotiation. There are a few possible reasons for this error:
The node is not sending the client certificate. Ensure that nifi.security.needClientAuth=true and nifi.cluster.protocol.is.secure=true are present in your nifi.properties file.
The truststore on the receiving node does not contain the public key certificate of the connecting node. When you followed the instructions from that link, how did you generate the respective certificates? Using the Apache NiFi TLS Toolkit as described by Pierre should ensure that all node certificates are signed by the same CA and that the CA is imported into the common truststore. If you manually generated your certificates, ensure that they are trusted on each node (you can do this with OpenSSL's s_client tool).

Related

kRB Ticket issues

Namenode and zk service went down and try to restart getting below error.
Open source hadoop cluster with KRB security enabled
ERROR TSaslTransport: SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
Getting m

Kerberos HBase Zookeeper fails

I'm trying to kerberise my HBase Cluster and I get some problems with Zookeeper. When I start Hbase I get this error on the Master log :
ERROR [main-SendThread(X.X.X.X:2181)] client.ZooKeeperSaslClient: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state.
ERROR [main-SendThread(X.X.X.X:2181)] zookeeper.ClientCnxn: SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state.
DEBUG [main-EventThread] zookeeper.ZKWatcher: master:16000-0x16c236187be0000, quorum=Y.Y.Y.Y:2181,X.X.X.X:2181, baseZNode=/hbase Received ZooKeeper Event, type=None, state=AuthFailed, path=null
DEBUG [main] zookeeper.ZooKeeper: Close called on already closed client
On the Zookeeper log, I get :
WARN [QuorumPeer[myid=0]/0:0:0:0:0:0:0:0:2181] quorum.Learner: Unexpected exception, tries=0, connecting to /X.X.X.X:2888
java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:229)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:71)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:937)
I verified my firewall, the ports are open
For the configuration, I followed the HBase Reference Guide :
http://hbase.apache.org/book.html#zk.sasl.auth
At first I thought it was a problem with my keytab but Hadoop is working fine with it.
I run HBase 2.0.5, Hadoop 3.1.2 and the Zookeeper is the one provided by HBase.
Following #SamsonScharfrichter 's comment, I've tried a few things :
I've created and specified in /etc/hosts the FQDN of my servers and modified my configurations to reflect this change.
Changed the hostname of my servers for the FQDN
tried to nslookup my hostnames, didn't work since they are specified in /etc/hosts
It didn't do anything, I'm still getting the error. My guess is that Kerberos tries to search for a DNS on my public NIC and not my private. I do not know why it struggles so hard to find my servers, since hadoop has absolutely no problem with it.
EDIT - I set up a private DNS on my network. DNS working great, still getting the error. I'm about to give up
EDIT 2 - I installed tshark on the node with the error. Apparently I get a frame with the message :
Error: KRB5KDC_ERR_C_PRINCIPAL_UNKNOWN
which is weird, I verified my keytab and the principals listed in kadmin. Maybe there defaults principals that I don't use ?

MQ light not responding to SSL requests after enabling SSL

I installed the MQ Light server on Windows 7 and followed the steps at https://developer.ibm.com/messaging/docs/mq-light-api/tutorials/security-tutorial/ to enable SSL using a self signed certificate generated by OpenSSL 1.1.0g.
At that point, once I started the MQ light server, my browser timed out trying to do the TLS handshake to the server and any attempt to connect to the server immediately using the Java MQ libraries aborted because the server closed the connection.
I couldn't figure out where the MQ server logs are to check them. Disabling SSL results in the connection working again.
I'm using Oracle Java 8u161 (64-bit).
Any idea what's going on here and how to get SSL working?
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2397;AMQ9204: Connection to host 'localhost(5671)' rejected. [1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2397;AMQ9771: SSL handshake failed. [1=javax.net.ssl.SSLHandshakeException[Remote host closed connection during handshake],3=localhost/127.0.0.1:5671 (localhost),4=SSLSocket.startHandshake,5=default]],3=localhost(5671),5=RemoteTCPConnection.protocolConnect]
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:2282)
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1294)
at com.ibm.mq.ese.jmqi.InterceptedJmqiImpl.jmqiConnect(InterceptedJmqiImpl.java:376)
at com.ibm.mq.ese.jmqi.ESEJMQI.jmqiConnect(ESEJMQI.java:560)
at com.ibm.mq.MQSESSION.MQCONNX_j(MQSESSION.java:916)
at com.ibm.mq.MQManagedConnectionJ11.<init>(MQManagedConnectionJ11.java:235)
... 16 more
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2397;AMQ9771: SSL handshake failed. [1=javax.net.ssl.SSLHandshakeException[Remote host closed connection during handshake],3=localhost/127.0.0.1:5671 (localhost),4=SSLSocket.startHandshake,5=default]
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.protocolConnect(RemoteTCPConnection.java:1329)
at com.ibm.mq.jmqi.remote.impl.RemoteConnection.connect(RemoteConnection.java:863)
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSessionFromNewConnection(RemoteConnectionSpecification.java:409)
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSession(RemoteConnectionSpecification.java:305)
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionPool.getSession(RemoteConnectionPool.java:146)
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1730)
... 21 more
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1002)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection$6.run(RemoteTCPConnection.java:1298)
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection$6.run(RemoteTCPConnection.java:1290)
at java.security.AccessController.doPrivileged(Native Method)
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.protocolConnect(RemoteTCPConnection.java:1290)
... 26 more
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:505)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
... 33 more

WSO2 API Manager "unable to find valid certification path to requested target {org .wso2.carbon.apimgt.hostobjects.APIProviderHostObject}"

We have deployed a WSO2 API Manager to expose the APIs of the system. When we published an API with a https target endpoint (Eg: https://abc.d.e:), the requests get failed with the following exception.
TID: [-1234] [] [2017-06-26 06:46:43,226] ERROR {org.wso2.carbon.apimgt.hostobje
cts.APIProviderHostObject} - Error occurred while connecting to backend : https
://list-micro.aws.na.sysco.net:9090, reason : sun.security.validator.ValidatorEx
ception: PKIX path building failed: sun.security.provider.certpath.SunCertPathBu
ilderException: unable to find valid certification path to requested target {org
.wso2.carbon.apimgt.hostobjects.APIProviderHostObject}
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException:
PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderExce
ption: unable to find valid certification path to requested target
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.
java:1514)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.jav
a:216)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:961)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062)
Wso2/Carbon uses Java 8 .. Keep in mind there is an issue where the apparently SSL client doesn't pares SNI very well. So if you are using multiple certificates on 1 ip address, this could be the issue:
http://javabreaks.blogspot.com/2015/12/java-ssl-handshake-with-server-name.html

Pivotal Cloud Foundry - Spring Cloud Services

I am trying to push the company app into Pivotal Cloud Foundry. The app should be bound with service registry. I didn't do any changes in application.yml file. I had have service-registry service in my Pivotal apps manager console. After I pushed my company app into PCF and binded with service-registry service. I did restage my company app. The app is not registered in the service registry console.
2016-06-06T11:25:36.72+0530 [APP/0] OUT Caused by: org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://p-spring-cloud-services.uaa.******.com/oauth/token":sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to fin
d valid certification path to requested target; nested exception is javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.securit
y.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
and
2016-06-06T05:49:35.000+00:00 [APP] OUT 2016-06-06 05:49:35.668 ERROR 22 --- [pool-5-thread-1] com.netflix.discovery.DiscoveryClient : DiscoveryClient_COMPANY/https://p-spring-cloud-services.uaa.******.com/oauth/token:****** - was unable to refresh its cache! status = Error requesting access token.
How to solve these issues?
The cause of problem is ssl certification for PCF. I think you are not uploaded ssl certificate to your environment.
Your PCF instance should have valid root certificates of your apps installed at their side and vice versa....

Resources