Nifi - "Connection refused" in gui after installation - apache-nifi

Question

I'm installing a three node non-secure nifi cluster (1.5) with embedded zookeeper.
The installation has completed, the cluster startup up and election have finished, without any obvious issues.
However when I hit the gui I see the following:
javax.ws.rs.ProcessingException: java.net.ConnectException: Connection refused (Connection refused)
In nifi-apps.log there is:
2018-03-07 14:41:37,857 WARN [Replicate Request Thread-7] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET /nifi-api/flow/current-user to test-nifi01:8080 due to javax.ws.rs.ProcessingException: java.net.ConnectException: Connection refused (Connection refused)
2018-03-07 14:41:37,858 WARN [Replicate Request Thread-7] o.a.n.c.c.h.r.ThreadPoolRequestReplicator
javax.ws.rs.ProcessingException: java.net.ConnectException: Connection refused (Connection refused)
at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:284)
at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:278)
at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:753)
....
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
....
in nifi-user.log:
2018-03-07 15:18:49,475 INFO [NiFi Web Server-21] org.apache.nifi.web.filter.RequestLogger Attempting request for (<no user found>) POST http://test-nifi01:8080/nifi-api/access/kerberos (source ip: 172.100.1.11)
2018-03-07 15:18:49,476 INFO [NiFi Web Server-21] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: Access tokens are only issued over HTTPS.. Returning Conflict response.
2018-03-07 15:18:49,572 INFO [NiFi Web Server-17] org.apache.nifi.web.filter.RequestLogger Attempting request for (<no user found>) POST http://test-nifi01:8080/nifi-api/access/oidc/exchange (source ip: 172.100.1.11)
2018-03-07 15:18:49,573 INFO [NiFi Web Server-17] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: User authentication/authorization is only supported when running over HTTPS.. Returning Conflict response.
2018-03-07 15:18:49,672 INFO [NiFi Web Server-23] org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous) GET http://test-nifi01:8080/nifi-api/flow/current-user (source ip: 172.100.1.11)
It's the same results regardless of what host I connect to.
I'm not sure what I need to be checking here.
EDIT: Updated with more detail.
Node1:
https://pastebin.com/5nMzWfXw
# web properties #
nifi.web.war.directory=./lib
nifi.web.http.host=test-nifi01
nifi.web.http.port=8080
nifi.web.http.network.interface.default=eth0
nifi.web.https.host=
nifi.web.https.port=
nifi.web.https.network.interface.default=
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
nifi.web.max.header.size=16 KB
nifi.web.proxy.context.path=
# cluster node properties (only configure for cluster nodes) #
nifi.cluster.is.node=true
nifi.cluster.node.address=test-nifi01
nifi.cluster.node.protocol.port=11003
nifi.cluster.node.protocol.threads=10
nifi.cluster.node.protocol.max.threads=50
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=5 sec
nifi.cluster.node.read.timeout=5 sec
nifi.cluster.node.max.concurrent.requests=100
nifi.cluster.firewall.file=
nifi.cluster.flow.election.max.wait.time=5 mins
nifi.cluster.flow.election.max.candidates=
Node 2: https://pastebin.com/n5GMy1nA
# web properties #
nifi.web.war.directory=./lib
nifi.web.http.host=test-nifi02
nifi.web.http.port=8080
nifi.web.http.network.interface.default=eth0
nifi.web.https.host=
nifi.web.https.port=
nifi.web.https.network.interface.default=
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
nifi.web.max.header.size=16 KB
nifi.web.proxy.context.path=
# cluster node properties (only configure for cluster nodes) #
nifi.cluster.is.node=true
nifi.cluster.node.address=test-nifi02
nifi.cluster.node.protocol.port=11003
nifi.cluster.node.protocol.threads=10
nifi.cluster.node.protocol.max.threads=50
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=5 sec
nifi.cluster.node.read.timeout=5 sec
nifi.cluster.node.max.concurrent.requests=100
nifi.cluster.firewall.file=
nifi.cluster.flow.election.max.wait.time=5 mins
nifi.cluster.flow.election.max.candidates=
Node 3: https://pastebin.com/EqztLfnD
# web properties #
nifi.web.war.directory=./lib
nifi.web.http.host=test-nifi03
nifi.web.http.port=8080
nifi.web.http.network.interface.default=eth0
nifi.web.https.host=
nifi.web.https.port=
nifi.web.https.network.interface.default=
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
nifi.web.max.header.size=16 KB
nifi.web.proxy.context.path=
# cluster node properties (only configure for cluster nodes) #
nifi.cluster.is.node=true
nifi.cluster.node.address=test-nifi03
nifi.cluster.node.protocol.port=11003
nifi.cluster.node.protocol.threads=10
nifi.cluster.node.protocol.max.threads=50
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=5 sec
nifi.cluster.node.read.timeout=5 sec
nifi.cluster.node.max.concurrent.requests=100
nifi.cluster.firewall.file=
nifi.cluster.flow.election.max.wait.time=5 mins
nifi.cluster.flow.election.max.candidates=
Anyone have any thoughts on what I should be looking at?
EDIT 2:
It's looks like each node is refusing to connect to itself. E.g on node 2 we have
2018-03-07 20:31:08,099 WARN [Replicate Request Thread-4] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET /nifi-api/flow/current-user to test-nifi02:8080 due to javax.ws.rs.ProcessingException: java.net.ConnectException: Connection refused (Connection refused)
and a curl also fails:
[taas#test-nifi02 logs]$ curl http://test-nifi02:8080
curl: (7) Failed connect to test-nifi02:8080; Connection refused
Curls to the other nodes are fine. The same behavior is occurring on each node, e,g node1 can't curl to node1 and node3 can't curl to node3.

Answer 1

Make sure you have dns enabled for network/subnet . if you dont have luxury of dns try updating /etc/hosts files with the exact ip address and hostnames
ex:
192.168.0.1 test-nifi01
192.168.0.2 test-nifi02
in all the hosts of the nifi nodes then try connecting it. and also make sure to check if you can able to connect from one node to another node by using shell via ssh

Answer 2

In a NiFi cluster, a request comes in to one node, and then that node replicates the request to all other nodes.
This issue shown nifi-app.log is indicating that the connection was refused between the current node test-nifi01:8080.
I assume you have hosts like test-nifi01, test-nifi02, and test-nifi03, so each of these nodes would need to be able to resolve the other hostnames. You can start by issuing a ping from one node to the other two, and do that on each node.