Apache Ignite Static IP node discover - amazon-ec2

Question

I am building an Apache Ignite cluster that uses Static IP Discovery rather than Multicast IP Discovery (its in AWS EC2).
I have configured like so (https://apacheignite.readme.io/docs/cluster-config#static-ip-based-discovery):
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
<value>host1.example.com</value>
<value>host2.example.com</value>
<value>host3.example.com</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
When I start the nodes, I can see the following error:
[15:04:36,385][WARNING][main][TcpDiscoverySpi] Failed to connect to any address from IP finder (make sure IP finder addresses are correct and firewalls are disabled on all host machines): [host1.example.com/10.0.21.223:47501, host2.example.com/10.0.21.207:47501, host3.example.com/10.0.21.191:47501]
From this I can see that Ignite has correctly resolved the hostnames to ip addresses, but it can't connect to form the cluster. These hosts are all in the same subnet in AWS EC2, and I have verified that port 47500 is open between them (I am not sure what the 47501 reference is).
I have also tried used just the ip addresses, rather than the hostnames, but the result is the same.
Am I missing something here?
Please note that Ignite is running as a Docker container on each server. The necessary TCP ports for Ignite are exposed and functional.
Full Log:
[15:55:18,231][INFO][main][IgniteKernal] Config URL: file:/apache-ignite-fabric-1.6.0-bin/config/default-config.xml
[15:55:18,231][INFO][main][IgniteKernal] Daemon mode: off
[15:55:18,231][INFO][main][IgniteKernal] OS: Linux 4.4.19-29.55.amzn1.x86_64 amd64
[15:55:18,231][INFO][main][IgniteKernal] OS user: root
[15:55:18,231][INFO][main][IgniteKernal] Language runtime: Java Platform API Specification ver. 1.8
[15:55:18,232][INFO][main][IgniteKernal] VM information: Java(TM) SE Runtime Environment 1.8.0_131-b11 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.131-b11
[15:55:18,233][INFO][main][IgniteKernal] VM total memory: 3.0GB
[15:55:18,233][INFO][main][IgniteKernal] Remote Management [restart: on, REST: on, JMX (remote: on, port: 49112, auth: off, ssl: off)]
[15:55:18,233][INFO][main][IgniteKernal] IGNITE_HOME=/apache-ignite-fabric-1.6.0-bin
[15:55:18,233][INFO][main][IgniteKernal] VM arguments: [-Xms1g, -Xmx1g, -XX:+AggressiveOpts, -XX:MaxMetaspaceSize=256m, -DIGNITE_QUIET=true, -DIGNITE_SUCCESS_FILE=/apache-ignite-fabric-1.6.0-bin/work/ignite_success_a8723f31-1954-4ea4-b190-762a36088140, -Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.port=49112, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.ssl=false, -DIGNITE_HOME=/apache-ignite-fabric-1.6.0-bin, -DIGNITE_PROG_NAME=/apache-ignite-fabric-1.6.0-bin/bin/ignite.sh, -Xmx3g, -XX:+UseG1GC]
[15:55:18,234][INFO][main][IgniteKernal] Configured caches ['ignite-marshaller-sys-cache', 'ignite-sys-cache', 'ignite-atomics-sys-cache']
[15:55:18,234][WARNING][main][IgniteKernal] Peer class loading is enabled (disable it in production for performance and deployment consistency reasons)
[15:55:18,238][INFO][main][IgniteKernal] 3-rd party licenses can be found at: /apache-ignite-fabric-1.6.0-bin/libs/licenses
[15:55:18,284][INFO][main][IgniteKernal] Non-loopback local IPs: 172.17.0.2, fe80:0:0:0:42:acff:fe11:2%eth0
[15:55:18,284][INFO][main][IgniteKernal] Enabled local MACs: 0242AC110002
[15:55:18,294][INFO][main][IgnitePluginProcessor] Configured plugins:
[15:55:18,295][INFO][main][IgnitePluginProcessor] ^-- None
[15:55:18,295][INFO][main][IgnitePluginProcessor]
[15:55:18,347][INFO][main][TcpCommunicationSpi] IPC shared memory server endpoint started [port=48100, tokDir=/apache-ignite-fabric-1.6.0-bin/work/ipc/shmem/cb41f1a5-d2f5-453d-a1c5-e008c1e12d2a-62]
[15:55:18,348][INFO][main][TcpCommunicationSpi] Successfully bound shared memory communication to TCP port [port=48100, locHost=0.0.0.0/0.0.0.0]
[15:55:18,362][INFO][main][TcpCommunicationSpi] Successfully bound to TCP port [port=47100, locHost=0.0.0.0/0.0.0.0]
[15:55:18,383][WARNING][main][NoopCheckpointSpi] Checkpoints are disabled (to enable configure any GridCheckpointSpi implementation)
[15:55:18,405][WARNING][main][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival).
[15:55:18,408][WARNING][main][NoopSwapSpaceSpi] Swap space is disabled. To enable use FileSwapSpaceSpi.
[15:55:18,409][INFO][main][IgniteKernal] Security status [authentication=off, tls/ssl=off]
[15:55:18,652][INFO][main][GridTcpRestProtocol] Command protocol successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11211]
[15:55:18,719][INFO][main][TcpDiscoverySpi] Successfully bound to TCP port [port=47500, localHost=0.0.0.0/0.0.0.0]
[15:55:21,746][WARNING][main][TcpDiscoverySpi] Failed to connect to any address from IP finder (make sure IP finder addresses are correct and firewalls are disabled on all host machines): [/10.0.21.191:47500, /10.0.21.207:47500, /10.0.21.223:47500]

Answer 1

Default port for ignite is 47500, but in logs as you can see it tries to connect to 47501:
try to specify a port in your address list:
<value>host1.example.com:47500</value>
also checkout this doc: https://apacheignite.readme.io/v2.0/docs/aws-deployment it can be useful in your case.

Answer 2

First of all I would recommend to update Ignite from 1.6 to 1.9.
Second, specify port, as was suggested here.
Run your docker with --net=host, so application inside container will see network interfaces from host. This should help.
Also, it's possible to configure special S3 IpFinder

Answer 3

Inspite of settings in my previous answer if it is not working then it might be because your IP may not be stable.
As per Ignite docs
Node discovery on AWS cloud is usually proven to be more challenging. Amazon EC2, just like most of the other virtual environments, has the following limitations:
Multicast is disabled.
TCP addresses change every time a new image is started.
Although you can use TCP-based discovery in the absence of the Multicast, you still have to deal with constantly changing IP addresses and constantly updating the configuration. This creates a major inconvenience and makes configurations based on static IPs virtually unusable in such environments.
Please refer to this link for Amazon S3 Based Discovery
https://apacheignite.readme.io/v1.1/docs/aws-config#section-amazon-s3-based-discovery

Answer 4

setLocalAddress and setLocalPort on the DiscoverySPI
<!-- Initial local port to listen to. -->
<property name="localPort" value="yourport"/>
<!-- Initial local address for discovery . -->
<property name="localAddress" value="yourhost"/>
<!-- Changing local port range. This is an optional action. -->
<property name="localPortRange" value="20"/>
and use the same port in the Addresslist that you specify on IPFinder.
<bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
<value>yourhost:yourport</value>
<value>otherhost:otherhostport</value>
</list>
</property>
</bean>
Ok all this being done should work with TCPVmIPfinder.