UI not opening in the nifi 1.0.0 secure cluster - hortonworks-data-platform

Tried to form a secure cluster in nifi 1.0.0 by following the instructions from the below link
http://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy
I have generated keystore and truststore file in the target folder, from nifi-cert.pem and nifi-key.key I have generated the p12 file and loaded it in browser
After starting the nif instances, nodes has been connected and exchanging the heartbeat signals between them and the port specified are listening to their numbers but when we launch the UI, It's not viewing in the browser
Update:
Update:
Update:
Update:
due to org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling protocol message in response to message type: CONNECTION_REQUEST due to java.net.SocketException: Software caused connection abort: socket write error
org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling protocol message in response to message type: CONNECTION_REQUEST due to java.net.SocketException: Software caused connection abort: socket write error
at org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:176) ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
at org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:136) [nifi-socket-utils-1.0.0.jar:1.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]

There are a couple steps you can take to continue debugging:
Run NiFi with Java TLS (SSL) debugging enabled. In conf/bootstrap.conf add the line java.arg.15=-Djavax.net.debug=ssl,handshake (ensuring that 15 is a number that does not conflict with your existing list of arguments). This will add a substantial amount of log output, but will explicitly document any TLS handshake negotiation attempts.
Use the OpenSSL s_client tool to debug the connection. Running the command below will attempt a handshake negotiation with additional log output: $ openssl s_client -connect <host:port> -debug -state -cert <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile <path_to_your_CA_cert.pem>
Substitute your server for <host:port>
Substitute your public key certificate for <path_to_your_cert.pem>
Substitute your private key for <path_to_your_key.pem>
Substitute your server's public key certificate or the CA public key certificate for <path_to_your_CA_cert.pem>
You can extract the public key certificate and private key from your PKCS12 keystore by using the following commands
Extract the private key: $ openssl pkcs12 -in CN\=Andy_LoPresto_OU\=Apache_NiFi.p12 -nocerts -out client.key
Extract the public key: $ openssl pkcs12 -in CN\=Andy_LoPresto_OU\=Apache_NiFi.p12 -clcerts -nokeys -out client.pem
As #bryan-bende pointed out above, the error message in the browser screenshot you shared does seem to indicate that the TLS cipher suites cannot be negotiated due to a protocol version incompatibility. The commands above will output all available cipher suites for the connection. You can also use a tool like CipherScan to enumerate these explicitly.
One possible issue is that Java 7 defaults to TLS 1.0 and Java 8 to TLS 1.2. What OS and JRE are you using to host NiFi?
In some rare cases, a user deploys NiFi with a keystore that does not actually contain an RSA key, and TLS handshake negotiation fails because "no cipher suites are available", when the issue is really that all of the available cipher suites require an RSA key (if not to encrypt the actual channel data, at least to sign the ephemeral keys). Can you verify that the keystore you provided NiFi has a valid (check dates as well) private key available?
Hopefully these steps help you diagnose the issue. If you can provide more information, we're happy to work with you to investigate further.

Related

Making POST request on HTTPS using Jira API in a Jira instance

My Jira instance is running on an HTTPS port. I have written a script that creates sub-tasks of a task in a project using a Jira API POST request on the Jira instance. It was working fine on my HTTP version. When I shifted to the HTTPS version, it started giving errors.
Error 1:
It gave the certificate error that was not present in Java.
Resolution. I resolved the issue using the below commands in my Jira instance.
openssl x509 -in <(openssl s_client -connect 192.168.2.214:8083 -prexit 2>/dev/null) -out ~/server-name.crt
keytool -importcert -file ~/server-name.crt -alias server-name -keystore $JAVA_HOME/jre/lib/security/cacerts -storepass changeit
My Instance is running on AWS and has different internal and external IPs to access it.
The problem I am facing after resolving the above issue is :
javax.net.ssl.SSLHandshakeException: No subject alternative names matching IP address 192.168.2.214 found
at Script2.run(Script2.groovy:11)
Caused by: java.security.cert.CertificateException: No subject alternative names matching IP address 192.168.2.214 found
... 1 more
My internal IP is 192.168.2.214 and my external IP is different from it.
My external IP is mapped to a URL and the certificate is purchased against that URL.
How do I get rid of this error as I only can access my HTTPS port using the internal IP when making an API call and there is no HTTP port defined. If I define an HTTP port, I would need to reconstruct my whole instance which is not possible at the moment.

Test IMAP connection to Outlook using OpenSSL

I am trying to access the Outlook mail server via IMAP using PHP. As this is not working as intended and the error messages are not helping, I looked up ways to test the basic connectivity and came across the following line of code:
openssl s_client -connect outlook.office365.com:993
This should, supposedly, let me connect to the mail server using an encrypted connection, so that I can then issue some commands to actually log in etc.
However, the command only generates the following output:
CONNECTED(00000003)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 313 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
I have tried researching the error message (write:errno=104), which lead to some suggestions like enforcing TLSv1.2 using the -tls1_2 parameter, neither of which made any difference.
Can someone point me in the right direction? I already ran this command on Ubuntu and Windows 10 using the latest version of OpenSSL.
I can connect using your command, but I needed to force \r\n line ending characters using the -crlf option when connecting in order to be able to type commands and have the server recognise them.
You can also add -quiet to reduce the amount of output:
openssl s_client -connect outlook.office365.com:993 -crlf -quiet
However, it seems the connection is getting reset in your case. This means there's some sort of network issue. There might be a firewall blocking your access to port 993 on outlook.office365.com. e.g. this might be the case if you normally have to connect via a proxy server.
Recent versions of openssl s_client have a -proxy option, but don't seem to allow specifying a proxy username and password. Also it might not work with your proxy even if you don't need to authenticate to the proxy server. One workaround might be to use http://ntlmaps.sourceforge.net/. I have tried it in the past, but it was over 15 years ago.
You could also try using cURL's IMAP support instead of openssl s_client, since it has better proxy support.
Another possibility is to connect to port 143 and use STARTTLS instead of connecting to port 993. Obviously if port 993 is firewalled then port 143 might also be, but in your case it seems like it is allowed:
openssl s_client -connect outlook.office365.com:143 -starttls imap -crlf -quiet
If you are behind a proxy and can't connect to the IMAP server directly, then you can also use something like EmailEngine that handles connections itself and allows access mailbox contents via a REST API.

Securing minifi c++ agent against remote NIFI

I have a remote 3 node secure NIFI server to which I want to send some data via MINIFI C++ agent. I am trying to understand the mechanics of generating and signing certificates for MINIFI (client) but I am not able to find detailed documentation.
I see the below configs:
#nifi.security.need.ClientAuth=
#nifi.security.client.certificate=
#nifi.security.client.private.key=
#nifi.security.client.pass.phrase=
#nifi.security.client.ca.certificate=
but how do I generate a client.pem and sign it?
EDIT:
This is what I tried to do (self-sign), but this fails with:
[2020-08-14 07:19:08.872] [org::apache::nifi::minifi::utils::HTTPClient] [error] curl_easy_perform() failed SSL connect error
[2020-08-14 07:19:08.872] [org::apache::nifi::minifi::RemoteProcessorGroupPort] [error] ProcessGroup::refreshRemoteSite2SiteInfo -- curl_easy_perform() failed
cd $HOME
openssl req -new -newkey rsa:4096 -nodes -keyout machine.key -out machine.csr
openssl x509 -req -sha256 -days 365 -in machine.csr -signkey machine.key -out machine.pem
###
Downloaded the public certificate from the server into ---> $HOME/server.crt
nifi.security.need.ClientAuth=true
nifi.security.client.certificate=$HOME/machine.pem
nifi.security.client.private.key=$HOME/machine.key
nifi.security.client.pass.phrase=password
nifi.security.client.ca.certificate=$HOME/server.crt
There are many ways to generate an X.509 certificate and sign it (openssl, TinyCert, Let's Encrypt, NiFi TLS Toolkit, etc.). The important requirement is that the certificate is either explicitly trusted (the public certificate is imported into the NiFi truststore) or implicitly trusted (any of the public certificates in the signing chain are present in the NiFi truststore).
One approach is to follow the walkthrough for deploying a secure NiFi cluster and use the TLS Toolkit to generate a client keystore, then export the certificate and key from that keystore to PEM format using these commands.

Connecting Mulesoft to Kafka hosted at Heroku

community!
I'm trying to connect my Mulesoft application to a Kafka Server that is hosted at Heroku. Anypoint Studio 7 (Mule 4) have a Kafka connector that has a few options to connect (Basic, Kerberos, Kerberos SSL and SSL):
According to Heroku's docs it "support" SSL but does not mention that is required. Can anyone confirm?
Heroku documentation that I saw [https://devcenter.heroku.com/articles/kafka-on-heroku]
When I set up an app at Heroku, add the Kafka AddOn, create the topic, I get the bootstrap server(s) from running the command heroku config:get KAFKA_URL If I try to to abasic test I get the following error:
The error states org.mule.runtime.api.connection.ConnectionException: invalid connection!
org.mule.runtime.api.connection.ConnectionException: invalid connection!
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
what I'm trying to build is a PoC with 2 flows... one that produces a message to a Topic and the other flow to consume from it.
Any help on how to set up the connector and the Heroku environment will be most welcome
UPDATE:
I found that when you create the app at Heroku and include the KAFKA addon, you get the following variables:
KAFKA_URL: A comma-separated list of SSL URLs to the Kafka brokers making up the cluster. Example:
kafka+ssl://ec2-3-*****-100.compute-1.amazonaws.com:9096,kafka+ssl://ec2-3-******-127.compute-1.amazonaws.com:9096
KAFKA_TRUSTED_CERT: The brokers’ SSL certificate (in PEM format), to check that you’re connecting to the right servers. Example:
-----BEGIN CERTIFICATE----- MIIDfzCCAmegAwIBAgIBADANBgkqhkiG9w0BAQsFADAyMTAwLgYDVQQDDCdjYS1l
ZjQwOWUzNy00NjhhLTRiMGEtOGVkOC0wZWYxMmRhYjkyZWEwHhcNMTkxMjEyMTUx
NzU5WhcNMjkxMjEyMTUxNzU5WjAyMTAwLgYDVQQDDCdjYS1lZjQwOWUzNy00Njhh
KAFKA_CLIENT_CERT: The required client certificate (in PEM format) to authenticate clients against the broker. Example:
-----BEGIN CERTIFICATE----- MIIDQzCCAiugAwIBAgIBADANBgkqhkiG9w0BAQsFADAyMTAwLgYDVQQDDCdjYS1l
ZjQwOWUzNy00NjhhLTRiMGEtOGVkOC0wZWYxMmRhYjkyZWEwHhcNMjAwMTE1MTU1
MjU2WhcNMzAwMTE1MTU1MjU2WjAZMRcwFQYDVQQDDA51NnZtYWVzM2cwZnMyZjCC
KAFKA_CLIENT_CERT_KEY: The required client certificate key (in PEM format) to authenticate clients against the broker.
Kafka clusters require authenticating using the provided client certificate. Any requests not using the client certificate will be denied. Example:
-----BEGIN RSA PRIVATE KEY----- MIIEpAIBAAKCAQEAmmu+j9DulVnqwIOt02++6Ehw9Mg7KaocdIQjODVtVipC5AyK
iaHsdNVh9bgJQAJSfstIV/6O1mCLmjUS/YmyIEEgmBvATFxcldPGBGcpPVSV2R3Q
I assume that I need to "transform" those into the KeyStore and TrustStore for the Connector SSL config.. can anyone confirm? since for what I see from the docs, SSL is required
Another Update:
I downloaded the certificates I describe above and using the tool "Key explorer" I created a keystore.jks and imported the KAFKA_CLIENT_CERT and KAFKA_CLIENT_CERT_KEY into it and then I created a truststore.jsk and imported the file KAFKA_TRUSTED_CERT there. on both cases I set a basic password ... was looking good but I get an error:
org.mule.runtime.api.connection.ConnectionException: invalid connection!
org.mule.runtime.api.connection.ConnectionException: invalid connection!
Caused by: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1521)
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:528)
at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1197)
at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1165)
at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
at org.apache.kafka.common.network.SslTransportLayer.handshakeWrap(SslTransportLayer.java:448)
at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:313)
at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:265)
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:170)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:547)
at org.apache.kafka.common.network.Selector.poll(Selector.java:483)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:540)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1196)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1709)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:318)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:970)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:967)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459)
at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:402)
at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:484)
at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:340)
... 7 more
Caused by: java.security.cert.CertificateException: No name matching ec2-3-220-121-33.compute-1.amazonaws.com found
at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:231)
at sun.security.util.HostnameChecker.match(HostnameChecker.java:96)
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455)
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:436)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:252)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1626)
... 16 more
Could be related to the line Caused by:
java.security.cert.CertificateException: No name matching
ec2-3-220-121-33.compute-1.amazonaws.com found
I got it to work.
create the JKS files using these commands (require HEROKU CLI)
client_key=`heroku config:get KAFKA_CLIENT_CERT_KEY --app <SET_HEROKU_APP_NAME_HERE>`
client_cert=`heroku config:get KAFKA_CLIENT_CERT --app <SET_HEROKU_APP_NAME_HERE>`
trusted_cert=`heroku config:get KAFKA_TRUSTED_CERT --app <SET_HEROKU_APP_NAME_HERE>`
# Write config vars to files.
echo "$client_key" >> keystore.pem
echo -n "$client_cert" >> keystore.pem
echo -n "$trusted_cert" > truststore.pem
# Set passwords
TRUSTSTORE_PASSWORD=<SET_PASSWORD_HERE>
KEYSTORE_PASSWORD=<SET_PASSWORD_HERE>
echo $TRUSTSTORE_PASSWORD
echo $KEYSTORE_PASSWORD
# Import cert.
keytool -importcert -file truststore.pem -keystore kafka.client.truststore.jks -deststorepass $TRUSTSTORE_PASSWORD -noprompt
# Create PKCS12 file.
openssl pkcs12 -export -in keystore.pem -out keystore.pkcs12 -password pass:$KEYSTORE_PASSWORD
# Create jks files.
keytool -importkeystore -srcstoretype PKCS12 \
-destkeystore kafka.client.keystore.jks -deststorepass $KEYSTORE_PASSWORD \
-srckeystore keystore.pkcs12 -srcstorepass $KEYSTORE_PASSWORD
Then config the connector with the SSL option and add the following INLINE:

Connecting to Apple Push Notification Server

I'm trying to connect to Apple's push notification server using my key and certificate
openssl s_client -connect gateway.sandbox.push.apple.com:2195 -cert MyApp.pem -key MyApp.pem
I keep getting the following error. Does anyone know what might be wrong?
7495:error:20074002:BIO routines:FILE_CTRL:system lib:/SourceCache/OpenSSL098/OpenSSL098-47/src/crypto/bio/bss_file.c:358:
unable to load client certificate private key file
You might be entering the wrong pass phrase.

Resources