Hazelcast cluster not available on Eureka - spring-boot

Question

I had a problem configuring simple Hazelcast cluster for existing Eureka client.
My application is simple Java SpringBoot application with Eureka discovery enabled, it is properly available for Eureka, however Hazelcast cluster is not.
This is my configuration:
hazelcast.xml
<hazelcast
xsi:schemaLocation="http://www.hazelcast.com/schema/config http://www.hazelcast.com/schema/config/hazelcast-config-3.9.xsd"
xmlns="http://www.hazelcast.com/schema/config"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<properties>
<property name="hazelcast.discovery.enabled">true</property>
<property name="hazelcast.jmx">true</property>
<property name="hazelcast.http.healthcheck.enabled">true</property>
<property name="hazelcast.rest.enabled">true</property>
<property name="hazelcast.logging.type">slf4j</property>
<property name="hazelcast.diagnostics.enabled">true</property>
<property name="hazelcast.diagnostics.metric.level">info</property>
<property name="hazelcast.name">hazelcast-cluster</property>
</properties>
<group>
<name>xxxx</name>
<password>xx</password>
</group>
<instance-name>hazelcast-cluster</instance-name>
<network>
<join>
<multicast enabled="false"/>
<tcp-ip enabled="false"/>
<aws enabled="false"/>
<discovery-strategies>
<discovery-strategy class="com.hazelcast.eureka.one.EurekaOneDiscoveryStrategy" enabled="true">
<properties>
<property name="self-registration">true</property>
<property name="namespace">hazelcast</property>
</properties>
</discovery-strategy>
</discovery-strategies>
</join>
</network>
maps....
</hazelcast>
Spring boot application:
#SpringBootApplication
#EnableAutoConfiguration ( exclude = { WebMvcAutoConfiguration.class } )
#EnableDiscoveryClient
#EnableEurekaClient
public class HazelcastClusterApplication {
public static void main( String[] args ) {
SpringApplication.run( HazelcastClusterApplication.class, args );
}
}
Configuration class:
#Configuration
public class HazelcastInstanceConfiguration {
#Bean
public HazelcastInstance hazelcastInstance(Config config, EurekaClient eurekaClient) {
EurekaOneDiscoveryStrategyFactory.setEurekaClient( eurekaClient );
return Hazelcast.newHazelcastInstance(config);
}
#Bean
public Config config() {
Config config = new ClasspathXmlConfig( "hazelcast.xml" );
return config;
}
}
bootstrap.yml
eureka:
client:
serviceUrl:
defaultZone: http://localhost:8011/eureka/
register-with-eureka: true
spring:
mvc:
favicon:
enabled: false
application:
name: cache-server
cloud:
config:
discovery:
enabled: true
serviceId: config-server
failFast: false
retry:
initialInterval: 10000
maxInterval: 60000
maxAttempts: 12
multiplier: 10.1
config:
name: application.*, ${spring.application.name}*.*
server:
port: 7990
Maven dependencies:
<dependency>
<groupId>com.hazelcast</groupId>
<artifactId>hazelcast</artifactId>
<version>3.11</version>
</dependency>
<dependency>
<groupId>com.hazelcast</groupId>
<artifactId>hazelcast-spring</artifactId>
<version>3.11</version>
</dependency>
<dependency>
<groupId>com.hazelcast</groupId>
<artifactId>hazelcast-eureka-one</artifactId>
<version>1.0.2</version>
</dependency>
After running, 'cache-server' application is available on Eureka, however I guess also 'hazelcast-cluster' should be available, but it is not. Or my assumption is wrong?
Startup logs:
INFO 17628 --- [ main] c.h.s.d.integration.DiscoveryService : [10.230.115.128]:5701 [xxxx] [3.11] Waiting for registration with Eureka...
INFO 17628 --- [ main] c.h.s.d.integration.DiscoveryService : [10.230.115.128]:5701 [xxxx] [3.11] Waiting for registration with Eureka...
INFO 17628 --- [nfoReplicator-0] com.netflix.discovery.DiscoveryClient : DiscoveryClient_CACHE-SERVER/pc_data:cache-server:7990: registering service...
INFO 17628 --- [nfoReplicator-0] com.netflix.discovery.DiscoveryClient : DiscoveryClient_CACHE-SERVER/pc_data:cache-server:7990 - registration status: 204
INFO 17628 --- [ main] c.h.s.d.integration.DiscoveryService : [10.230.115.128]:5701 [xxxx] [3.11] Waiting for registration with Eureka...
DEBUG 17628 --- [ main] c.h.i.cluster.impl.DiscoveryJoiner : [10.230.115.128]:5701 [xxxx] [3.11] This node will assume master role since no possible member where connected to.
DEBUG 17628 --- [ main] c.h.internal.cluster.ClusterService : [10.230.115.128]:5701 [xxxx] [3.11] Setting master address to [10.230.115.128]:5701
DEBUG 17628 --- [ main] c.h.i.cluster.impl.MembershipManager : [10.230.115.128]:5701 [xxxx] [3.11] Local member list join version is set to 1
DEBUG 17628 --- [ main] c.h.i.cluster.impl.DiscoveryJoiner : [10.230.115.128]:5701 [xxxx] [3.11] PostJoin master: [10.230.115.128]:5701, isMaster: true
INFO 17628 --- [ main] c.h.internal.cluster.ClusterService : [10.230.115.128]:5701 [xxxx] [3.11]
Members {size:1, ver:1} [
Member [10.230.115.128]:5701 - f6ed031e-5740-4daa-b583-5b91d98816c2 this
]
And then my cache-server application is available on Eureka, but not hazelcast-cluster as I guess it should.
Hazelcast client service logs (cutted):
2018-12-11 09:48:01.673 INFO [security-service,,,] 21496 --- [ main] com.hazelcast.client.HazelcastClient : hz.client_0 [xxxx] [3.11] A non-empty group password is configured for the Hazelcast client. Starting with Hazelcast version 3.11, clients with the same group name, but with different group passwords (that do not use authentication) will be accepted to a cluster. The group password configuration will be removed completely in a future release.
2018-12-11 09:48:01.700 INFO [security-service,,,] 21496 --- [ main] com.hazelcast.core.LifecycleService : hz.client_0 [xxxx] [3.11] HazelcastClient 3.11 (20181023 - 1500bbb) is STARTING
2018-12-11 09:48:03.233 INFO [security-service,,,] 21496 --- [ main] o.s.c.n.eureka.InstanceInfoFactory : Setting initial instance status as: STARTING
2018-12-11 09:48:03.249 INFO [security-service,,,] 21496 --- [ main] com.netflix.discovery.DiscoveryClient : Initializing Eureka in region us-east-1
2018-12-11 09:48:03.264 INFO [security-service,,,] 21496 --- [ main] c.n.d.provider.DiscoveryJerseyProvider : Using JSON encoding codec LegacyJacksonJson
2018-12-11 09:48:03.266 INFO [security-service,,,] 21496 --- [ main] c.n.d.provider.DiscoveryJerseyProvider : Using JSON decoding codec LegacyJacksonJson
2018-12-11 09:48:03.267 INFO [security-service,,,] 21496 --- [ main] c.n.d.provider.DiscoveryJerseyProvider : Using XML encoding codec XStreamXml
2018-12-11 09:48:03.267 INFO [security-service,,,] 21496 --- [ main] c.n.d.provider.DiscoveryJerseyProvider : Using XML decoding codec XStreamXml
2018-12-11 09:48:03.417 INFO [security-service,,,] 21496 --- [ main] c.n.d.s.r.aws.ConfigClusterResolver : Resolving eureka endpoints via configuration
2018-12-11 09:48:03.421 INFO [security-service,,,] 21496 --- [ main] com.netflix.discovery.DiscoveryClient : Disable delta property : false
2018-12-11 09:48:03.421 INFO [security-service,,,] 21496 --- [ main] com.netflix.discovery.DiscoveryClient : Single vip registry refresh property : null
2018-12-11 09:48:03.422 INFO [security-service,,,] 21496 --- [ main] com.netflix.discovery.DiscoveryClient : Force full registry fetch : false
2018-12-11 09:48:03.422 INFO [security-service,,,] 21496 --- [ main] com.netflix.discovery.DiscoveryClient : Application is null : false
2018-12-11 09:48:03.423 INFO [security-service,,,] 21496 --- [ main] com.netflix.discovery.DiscoveryClient : Registered Applications size is zero : true
2018-12-11 09:48:03.424 INFO [security-service,,,] 21496 --- [ main] com.netflix.discovery.DiscoveryClient : Application version is -1: true
2018-12-11 09:48:03.424 INFO [security-service,,,] 21496 --- [ main] com.netflix.discovery.DiscoveryClient : Getting all instance registry info from the eureka server
2018-12-11 09:48:03.426 INFO [security-service,,,] 21496 --- [ main] com.netflix.discovery.DiscoveryClient : The response status is 200
2018-12-11 09:48:03.426 INFO [security-service,,,] 21496 --- [ main] com.netflix.discovery.DiscoveryClient : Starting heartbeat executor: renew interval is: 5
2018-12-11 09:48:03.426 INFO [security-service,,,] 21496 --- [ main] c.n.discovery.InstanceInfoReplicator : InstanceInfoReplicator onDemand update allowed rate per min is 4
2018-12-11 09:48:03.442 INFO [security-service,,,] 21496 --- [ main] com.netflix.discovery.DiscoveryClient : Discovery Client initialized at timestamp 1544518083442 with initial instances count: 3
2018-12-11 09:48:03.488 INFO [security-service,,,] 21496 --- [ main] c.h.s.d.integration.DiscoveryService : hz.client_0 [xxxx] [3.11] Waiting for registration with Eureka...
2018-12-11 09:48:08.468 INFO [security-service,,,] 21496 --- [tbeatExecutor-0] com.netflix.discovery.DiscoveryClient : DiscoveryClient_SECURITY-SERVICE/GPLPLW2002.gft.com:security-service:8035 - Re-registering apps/SECURITY-SERVICE
2018-12-11 09:48:08.468 INFO [security-service,,,] 21496 --- [tbeatExecutor-0] com.netflix.discovery.DiscoveryClient : DiscoveryClient_SECURITY-SERVICE/GPLPLW2002.gft.com:security-service:8035: registering service...
2018-12-11 09:48:08.499 INFO [security-service,,,] 21496 --- [ main] c.h.s.d.integration.DiscoveryService : hz.client_0 [xxxx] [3.11] Waiting for registration with Eureka...
2018-12-11 09:48:08.530 INFO [security-service,,,] 21496 --- [tbeatExecutor-0] com.netflix.discovery.DiscoveryClient : DiscoveryClient_SECURITY-SERVICE/GPLPLW2002.gft.com:security-service:8035 - registration status: 204
2018-12-11 09:48:13.500 INFO [security-service,,,] 21496 --- [ main] c.h.s.d.integration.DiscoveryService : hz.client_0 [xxxx] [3.11] Waiting for registration with Eureka...
2018-12-11 09:48:18.503 INFO [security-service,,,] 21496 --- [ main] c.h.s.d.integration.DiscoveryService : hz.client_0 [xxxx] [3.11] Waiting for registration with Eureka...
2018-12-11 09:48:23.512 INFO [security-service,,,] 21496 --- [ main] c.h.s.d.integration.DiscoveryService : hz.client_0 [xxxx] [3.11] Waiting for registration with Eureka...
2018-12-11 09:48:28.513 INFO [security-service,,,] 21496 --- [ main] c.h.s.d.integration.DiscoveryService : hz.client_0 [xxxx] [3.11] Waiting for registration with Eureka...
2018-12-11 09:48:33.515 INFO [security-service,,,] 21496 --- [ main] c.h.s.d.integration.DiscoveryService : hz.client_0 [xxxx] [3.11] Waiting for registration with Eureka...
2018-12-11 09:48:33.557 INFO [security-service,,,] 21496 --- [ main] c.h.client.spi.ClientInvocationService : hz.client_0 [xxxx] [3.11] Running with 2 response threads
2018-12-11 09:48:33.742 INFO [security-service,,,] 21496 --- [ main] com.hazelcast.core.LifecycleService : hz.client_0 [xxxx] [3.11] HazelcastClient 3.11 (20181023 - 1500bbb) is STARTED
2018-12-11 09:48:33.772 DEBUG [security-service,,,] 21496 --- [ main] c.h.i.networking.nio.NioNetworking : hz.client_0 [xxxx] [3.11] TcpIpConnectionManager configured with Non Blocking IO-threading model: 1 input threads and 1 output threads
2018-12-11 09:48:33.774 DEBUG [security-service,,,] 21496 --- [ main] c.h.i.networking.nio.NioNetworking : hz.client_0 [xxxx] [3.11] IO threads selector mode is SELECT
2018-12-11 09:48:33.818 WARN [security-service,,,] 21496 --- [ient_0.cluster-] c.h.c.c.ClientConnectionManager : hz.client_0 [xxxx] [3.11] Unable to get alive cluster connection, try in 3000 ms later, attempt 1 of 2.
2018-12-11 09:48:36.822 WARN [security-service,,,] 21496 --- [ient_0.cluster-] c.h.c.c.ClientConnectionManager : hz.client_0 [xxxx] [3.11] Unable to get alive cluster connection, attempt 2 of 2.
2018-12-11 09:48:36.824 WARN [security-service,,,] 21496 --- [ient_0.cluster-] c.h.c.c.ClientConnectionManager : hz.client_0 [xxxx] [3.11] Could not connect to cluster, shutting down the client. Unable to connect to any address! The following addresses were tried: []
2018-12-11 09:48:36.828 INFO [security-service,,,] 21496 --- [clientShutdown-] com.hazelcast.core.LifecycleService : hz.client_0 [xxxx] [3.11] HazelcastClient 3.11 (20181023 - 1500bbb) is SHUTTING_DOWN
2018-12-11 09:48:36.839 INFO [security-service,,,] 21496 --- [clientShutdown-] com.netflix.discovery.DiscoveryClient : Shutting down DiscoveryClient ...
2018-12-11 09:48:36.843 INFO [security-service,,,] 21496 --- [clientShutdown-] com.netflix.discovery.DiscoveryClient : Unregistering ...
2018-12-11 09:48:36.844 WARN [security-service,,,] 21496 --- [ main] ationConfigEmbeddedWebApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: o.....org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'cacheManager' defined in class path resource
.......
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hazelcastInstance' defined in class path resource [xxxx/hazelcast/HazelcastClientConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'hazelcastInstance' threw exception; nested exception is java.lang.IllegalStateException: Unable to connect to any address! The following addresses were tried: []
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:599)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1173)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1067)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066)
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
... 150 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'hazelcastInstance' threw exception; nested exception is java.lang.IllegalStateException: Unable to connect to any address! The following addresses were tried: []
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
... 163 common frames omitted
Caused by: java.lang.IllegalStateException: Unable to connect to any address! The following addresses were tried: []
at com.hazelcast.client.connection.nio.ClusterConnector.connectToClusterInternal(ClusterConnector.java:206)
at com.hazelcast.client.connection.nio.ClusterConnector.access$400(ClusterConnector.java:56)
at com.hazelcast.client.connection.nio.ClusterConnector$2.call(ClusterConnector.java:215)
at com.hazelcast.client.connection.nio.ClusterConnector$2.call(ClusterConnector.java:211)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)
What is wrong with my configuration, is this a server problem or client?
Client configuration:
#Bean
#Autowired
public HazelcastInstance hazelcastInstance( EurekaClient eurekaClient, GroupConfig groupConfig ) {
EurekaOneDiscoveryStrategyFactory.setEurekaClient( eurekaClient );
ClientConfig config = new ClientConfig();
config.setGroupConfig( groupConfig );
config.setProperty( "hazelcast.discovery.enabled", "true" );
DiscoveryStrategyConfig discoveryStrategyConfig = new DiscoveryStrategyConfig( new EurekaOneDiscoveryStrategyFactory() );
discoveryStrategyConfig.addProperty( "use-classpath-eureka-client-props", "false" );
discoveryStrategyConfig.addProperty( "self-registration", "false" );
DiscoveryConfig discoveryConfig = new DiscoveryConfig();
discoveryConfig.addDiscoveryStrategyConfig( discoveryStrategyConfig );
config.getNetworkConfig().setDiscoveryConfig( discoveryConfig );
return HazelcastClient.newHazelcastClient( config );
}

Answer 1

The problem is that you use the same EurekaClient for you app discovery (port 7990) and your Hazelcast discovery (port 5701). For more information, read the related GH issue.
You can solve it in two ways:
Use the property use-metadata-for-host-and-port as described here
Use separate Eureka Client and ports for your app and Hazelcast
EDIT: The use-metadata-for-host-and-port will be released in hazelcast-eureka:1.0.3 soon. I've also prepared Hazelcast Eureka Code Sample.

Answer 2

Have you seen the example configs in Eureka plugin repo and also the code sample with Eureka? I see there are some differences in the configs with yours. Please check and let me know if you still have problems.