Slow startup of hazelcast - spring-boot

Question

I have a spring-boot application, and i would like to use hazelcast as a cache provider with spring-boot caching.
I have the following configuration in a hazelcast.yaml file:
hazelcast:
cluster-name: message-handler-cluster
network:
join:
auto-detection:
enabled: false
multicast:
enabled: false
tcp-ip:
enabled: true
member-list:
- 127.0.0.1
integrity-checker:
enabled: false
When i use hazelcast-spring 5.1.1 maven dependency hazelcast is starting painfully slow. Here is the startup log:
2022-04-19 09:59:30 WARN (main) [] [StandardLoggerFactory$StandardLogger] [] [] Hazelcast is starting in a Java modular environment (Java 9 and newer) but without proper access to required Java packages. Use additional Java arguments to provide Hazelcast access to Java internal API. The internal API access is used to get the best performance results. Arguments to be used:
--add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED
2022-04-19 09:59:30 INFO (main) [] [StandardLoggerFactory$StandardLogger] [] [] [LOCAL] [message-handler-cluster] [5.1.1] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [127.0.0.1]
2022-04-19 09:59:30 INFO (main) [] [StandardLoggerFactory$StandardLogger] [] [] [127.0.0.1]:5701 [message-handler-cluster] [5.1.1]
+ + o o o o---o o----o o o---o o o----o o--o--o
+ + + + | | / \ / | | / / \ | |
+ + + + + o----o o o o o----o | o o o o----o |
+ + + + | | / \ / | | \ / \ | |
+ + o o o o o---o o----o o----o o---o o o o----o o
2022-04-19 09:59:30 INFO (main) [] [StandardLoggerFactory$StandardLogger] [] [] [127.0.0.1]:5701 [message-handler-cluster] [5.1.1] Copyright (c) 2008-2022, Hazelcast, Inc. All Rights Reserved.
2022-04-19 09:59:30 INFO (main) [] [StandardLoggerFactory$StandardLogger] [] [] [127.0.0.1]:5701 [message-handler-cluster] [5.1.1] Hazelcast Platform 5.1.1 (20220317 - 5b5fa10) starting at [127.0.0.1]:5701
2022-04-19 09:59:30 INFO (main) [] [StandardLoggerFactory$StandardLogger] [] [] [127.0.0.1]:5701 [message-handler-cluster] [5.1.1] Cluster name: message-handler-cluster
2022-04-19 09:59:30 INFO (main) [] [StandardLoggerFactory$StandardLogger] [] [] [127.0.0.1]:5701 [message-handler-cluster] [5.1.1] Integrity Checker is disabled. Fail-fast on corrupted executables will not be performed.
To enable integrity checker do one of the following:
- Change member config using Java API: config.setIntegrityCheckerEnabled(true);
- Change XML/YAML configuration property: Set hazelcast.integrity-checker.enabled to true
- Add system property: -Dhz.integritychecker.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
- Add environment variable: HZ_INTEGRITYCHECKER_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)
2022-04-19 09:59:30 INFO (main) [] [StandardLoggerFactory$StandardLogger] [] [] [127.0.0.1]:5701 [message-handler-cluster] [5.1.1] The Jet engine is disabled.
To enable the Jet engine on the members, do one of the following:
- Change member config using Java API: config.getJetConfig().setEnabled(true)
- Change XML/YAML configuration property: Set hazelcast.jet.enabled to true
- Add system property: -Dhz.jet.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
- Add environment variable: HZ_JET_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)
2022-04-19 10:00:22 INFO (main) [] [StandardLoggerFactory$StandardLogger] [] [] [127.0.0.1]:5701 [message-handler-cluster] [5.1.1] Enable DEBUG/FINE log level for log category com.hazelcast.system.security or use -Dhazelcast.security.recommendations system property to see security recommendations and the status of current config.
2022-04-19 10:00:23 INFO (main) [] [StandardLoggerFactory$StandardLogger] [] [] [127.0.0.1]:5701 [message-handler-cluster] [5.1.1] Using TCP/IP discovery
2022-04-19 10:00:23 WARN (main) [] [StandardLoggerFactory$StandardLogger] [] [] [127.0.0.1]:5701 [message-handler-cluster] [5.1.1] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.
2022-04-19 10:00:23 INFO (main) [] [StandardLoggerFactory$StandardLogger] [] [] [127.0.0.1]:5701 [message-handler-cluster] [5.1.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2022-04-19 10:00:23 INFO (main) [] [StandardLoggerFactory$StandardLogger] [] [] [127.0.0.1]:5701 [message-handler-cluster] [5.1.1] [127.0.0.1]:5701 is STARTING
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.hazelcast.internal.networking.nio.SelectorOptimizer (file:/C:/Users/ZC15PL/.m2/repository/com/hazelcast/hazelcast/5.1.1/hazelcast-5.1.1.jar) to field sun.nio.ch.SelectorImpl.selectedKeys
WARNING: Please consider reporting this to the maintainers of com.hazelcast.internal.networking.nio.SelectorOptimizer
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2022-04-19 10:00:25 INFO (hz.upbeat_engelbart.cached.thread-2) [] [StandardLoggerFactory$StandardLogger] [] [] [127.0.0.1]:5701 [message-handler-cluster] [5.1.1] [127.0.0.1]:5703 is added to the blacklist.
2022-04-19 10:00:25 INFO (hz.upbeat_engelbart.cached.thread-1) [] [StandardLoggerFactory$StandardLogger] [] [] [127.0.0.1]:5701 [message-handler-cluster] [5.1.1] [127.0.0.1]:5702 is added to the blacklist.
2022-04-19 10:00:26 INFO (main) [] [StandardLoggerFactory$StandardLogger] [] [] [127.0.0.1]:5701 [message-handler-cluster] [5.1.1]
Members {size:1, ver:1} [
Member [127.0.0.1]:5701 - 01db03fc-2dc4-45a6-813c-80b843d1e1b4 this
]
2022-04-19 10:00:26 INFO (main) [] [StandardLoggerFactory$StandardLogger] [] [] [127.0.0.1]:5701 [message-handler-cluster] [5.1.1] [127.0.0.1]:5701 is STARTED
Startup time is about 30-40 seconds.
When i use hazelcast-spring 4.2.4 maven dependency hazelcast is starting very quick. Here is the startup log:
2022-04-19 10:04:41 WARN (main) [] [StandardLoggerFactory$StandardLogger] [] [] Hazelcast is starting in a Java modular environment (Java 9 and newer) but without proper access to required Java packages. Use additional Java arguments to provide Hazelcast access to Java internal API. The internal API access is used to get the best performance results. Arguments to be used:
--add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED
2022-04-19 10:04:41 INFO (main) [] [StandardLoggerFactory$StandardLogger] [] [] [LOCAL] [message-handler-cluster] [4.2.4] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [127.0.0.1]
2022-04-19 10:04:41 INFO (main) [] [StandardLoggerFactory$StandardLogger] [] [] [127.0.0.1]:5701 [message-handler-cluster] [4.2.4] Hazelcast 4.2.4 (20211220 - 25f0049) starting at [127.0.0.1]:5701
2022-04-19 10:04:42 INFO (main) [] [StandardLoggerFactory$StandardLogger] [] [] [127.0.0.1]:5701 [message-handler-cluster] [4.2.4] Using TCP/IP discovery
2022-04-19 10:04:42 WARN (main) [] [StandardLoggerFactory$StandardLogger] [] [] [127.0.0.1]:5701 [message-handler-cluster] [4.2.4] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.
2022-04-19 10:04:43 INFO (main) [] [StandardLoggerFactory$StandardLogger] [] [] [127.0.0.1]:5701 [message-handler-cluster] [4.2.4] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2022-04-19 10:04:43 INFO (main) [] [StandardLoggerFactory$StandardLogger] [] [] [127.0.0.1]:5701 [message-handler-cluster] [4.2.4] [127.0.0.1]:5701 is STARTING
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.hazelcast.internal.networking.nio.SelectorOptimizer (file:/C:/Users/ZC15PL/.m2/repository/com/hazelcast/hazelcast/4.2.4/hazelcast-4.2.4.jar) to field sun.nio.ch.SelectorImpl.selectedKeys
WARNING: Please consider reporting this to the maintainers of com.hazelcast.internal.networking.nio.SelectorOptimizer
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2022-04-19 10:04:45 INFO (hz.vibrant_ganguly.cached.thread-3) [] [StandardLoggerFactory$StandardLogger] [] [] [127.0.0.1]:5701 [message-handler-cluster] [4.2.4] [127.0.0.1]:5703 is added to the blacklist.
2022-04-19 10:04:45 INFO (hz.vibrant_ganguly.cached.thread-2) [] [StandardLoggerFactory$StandardLogger] [] [] [127.0.0.1]:5701 [message-handler-cluster] [4.2.4] [127.0.0.1]:5702 is added to the blacklist.
2022-04-19 10:04:46 INFO (main) [] [StandardLoggerFactory$StandardLogger] [] [] [127.0.0.1]:5701 [message-handler-cluster] [4.2.4]
Members {size:1, ver:1} [
Member [127.0.0.1]:5701 - c810dae2-5697-4d3d-9c60-0a7ed89f66d7 this
]
2022-04-19 10:04:46 INFO (main) [] [StandardLoggerFactory$StandardLogger] [] [] [127.0.0.1]:5701 [message-handler-cluster] [4.2.4] [127.0.0.1]:5701 is STARTED
Startup time is 3-4 seconds.
How should i configure hazelcast 5.1.1 to achieve the above, very quick startup time, or what am i missing in the configuration?
Update:
More detailed (DEBUG) log with 5.1.1 (demo github project):
2022-04-19 16:44:42.425 DEBUG 14620 --- [ main] c.h.s.i.o.impl.BackpressureRegulator : [127.0.0.1]:5701 [demo-cluster] [5.1.1] Backpressure is disabled
2022-04-19 16:44:42.494 DEBUG 14620 --- [ main] h.s.i.o.i.InboundResponseHandlerSupplier : [127.0.0.1]:5701 [demo-cluster] [5.1.1] Running with 2 response threads
2022-04-19 16:45:35.617 DEBUG 14620 --- [ main] c.h.i.server.tcp.LocalAddressRegistry : [127.0.0.1]:5701 [demo-cluster] [5.1.1] LinkedAddresses{primaryAddress=[127.0.0.1]:5701, allLinkedAddresses=[[fe80:0:0:0:5582:4f40:ed57:9ab0%eth3]:5701, [fe80:0:0:0:e154:4b5e:dd8b:23aa%wlan4]:5701, [172.31.64.1]:5701, [fe80:0:0:0:f161:66b7:46bd:3d9c%eth26]:5701, [172.25.112.1]:5701, [fe80:0:0:0:e0b7:e6fc:65d7:29c9]:5701, [fe80:0:0:0:f161:66b7:46bd:3d9c]:5701, [fe80:0:0:0:e154:4b5e:dd8b:23aa]:5701, [fe80:0:0:0:150:ba24:fd1c:f8d6]:5701, [192.168.96.1]:5701, [fe80:0:0:0:f1d8:d5be:d257:80f4]:5701, [fe80:0:0:0:7d64:92ab:6a06:9db2%eth8]:5701, [fe80:0:0:0:b45f:145f:a925:afd%eth15]:5701, [fe80:0:0:0:81c6:d272:c19d:e398%eth51]:5701, [fe80:0:0:0:5918:ecac:57d2:bbc1%net0]:5701, [fe80:0:0:0:4cd7:74e4:1e3c:e686]:5701, [127.0.0.1]:5701, [fe80:0:0:0:4cd7:74e4:1e3c:e686%eth23]:5701, [fe80:0:0:0:e0b7:e6fc:65d7:29c9%eth2]:5701, [fe80:0:0:0:81c6:d272:c19d:e398]:5701, [fe80:0:0:0:1c97:db6c:48a5:dee5]:5701, [fe80:0:0:0:1c97:db6c:48a5:dee5%eth10]:5701, [192.168.160.1]:5701, [fe80:0:0:0:5582:4f40:ed57:9ab0]:5701, [fe80:0:0:0:f1d8:d5be:d257:80f4%eth31]:5701, [fe80:0:0:0:150:ba24:fd1c:f8d6%eth7]:5701, [fe80:0:0:0:20ca:afcd:2c09:c89a]:5701, [fe80:0:0:0:f145:d1f6:dc9b:a5a8]:5701, [fe80:0:0:0:b45f:145f:a925:afd]:5701, [fe80:0:0:0:e927:f676:20d4:509d%wlan3]:5701, [172.16.128.213]:5701, [fe80:0:0:0:20ca:afcd:2c09:c89a%eth21]:5701, [fe80:0:0:0:f145:d1f6:dc9b:a5a8%wlan0]:5701, [10.83.179.202]:5701, [fe80:0:0:0:e927:f676:20d4:509d]:5701, [192.168.224.1]:5701, [172.31.112.1]:5701, [0:0:0:0:0:0:0:1]:5701, [fe80:0:0:0:7d64:92ab:6a06:9db2]:5701, [fe80:0:0:0:5918:ecac:57d2:bbc1]:5701, [172.27.16.1]:5701]} are registered for the local member with local uuid=e291a410-c08f-498d-9cc1-445bf3999ace
2022-04-19 16:45:35.885 DEBUG 14620 --- [ main] com.hazelcast.system.security : [127.0.0.1]:5701 [demo-cluster] [5.1.1]
Cheers,
Zsolt

Answer 1

I'm the one who added this change in 5.1. I didn't think that it would take so long before. With 5.1, we started to register all the addresses of the server sockets in a registry so that the Hazelcast member knows that these addresses are its own addresses. See: https://github.com/hazelcast/hazelcast/blob/bbcb69ae732a0717cbc5a28339c94dfd49a73493/hazelcast/src/main/java/com/hazelcast/internal/server/tcp/LocalAddressRegistry.java#L246-L261. This results in that if a server binds a socket to any local address, it iterates all network interfaces and records these addresses in the registry as the member's own address and this seems to take too long in your environment. Could you try to use -Dhazelcast.socket.bind.any=false property to avoid binding any interface for the member's server sockets, otherwise it binds 0.0.0.0 and we iterate over all the interfaces .