hazelcast with spring-boot keeps restarting - spring-boot

Question

I am deploying a spring-boot application that is using hazelcast to a kubernetes cluster for test purpose. In this deployment (using kind) there is only one instance.
Hazelcast is configured like this:
hazelcast:
network:
join:
multicast:
enabled: false
kubernetes:
enabled: true
service-dns: my-app-hs
Which is the same configuration applied in the real deployment, the only difference is that in the real deployment there are 3 instances at least.
The issue I now see is that the spring-boot application is going down and a new instance is starting up again.
Here the full logs (debug level for hazelcast):
{"thread":"main","logger":"org.example.app.MyApplicationKt","message":"Starting MyApplicationKt using Java 11.0.15 on my-app-74975f549-jg9d6 with PID 1 (/app/classes started by root in /)","context":"default","severity":"INFO","time":"2022-06-07T11:40:34.449"}
{"thread":"main","logger":"org.example.app.MyApplicationKt","message":"The following 1 profile is active: \"production\"","context":"default","severity":"INFO","time":"2022-06-07T11:40:34.525"}
{"thread":"main","logger":"org.springframework.cloud.context.scope.GenericScope","message":"BeanFactory id=40671f22-f42d-3e93-8223-2e3e4f329f31","context":"default","severity":"INFO","time":"2022-06-07T11:40:38.584"}
{"thread":"main","logger":"com.hazelcast.internal.config.AbstractConfigLocator","message":"Loading 'hazelcast.yaml' from the classpath.","context":"default","severity":"INFO","time":"2022-06-07T11:40:39.8"}
{"thread":"main","logger":"com.hazelcast.internal.util.JavaVersion","message":"Detected runtime version: Java 11","context":"default","severity":"DEBUG","time":"2022-06-07T11:40:40.648"}
{"thread":"main","logger":"com.hazelcast.instance.impl.HazelcastInstanceFactory","message":"Hazelcast is starting in a Java modular environment (Java 9 and newer) but without proper access to required Java packages. Use additional Java arguments to provide Hazelcast access to Java internal API. The internal API access is used to get the best performance results. Arguments to be used:\n --add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED","context":"default","severity":"WARN","time":"2022-06-07T11:40:40.651"}
{"thread":"main","logger":"com.hazelcast.instance.AddressPicker","message":"[LOCAL] [dev] [5.1.1] Prefer IPv4 stack is true, prefer IPv6 addresses is false","context":"default","severity":"DEBUG","time":"2022-06-07T11:40:40.727"}
{"thread":"main","logger":"com.hazelcast.instance.AddressPicker","message":"[LOCAL] [dev] [5.1.1] Trying to bind inet socket address: 0.0.0.0/0.0.0.0:5701","context":"default","severity":"DEBUG","time":"2022-06-07T11:40:40.758"}
{"thread":"main","logger":"com.hazelcast.instance.AddressPicker","message":"[LOCAL] [dev] [5.1.1] Bind successful to inet socket address: /0:0:0:0:0:0:0:0:5701","context":"default","severity":"DEBUG","time":"2022-06-07T11:40:40.76"}
{"thread":"main","logger":"com.hazelcast.instance.AddressPicker","message":"[LOCAL] [dev] [5.1.1] Picked [10.244.0.44]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5701], bind any local is true","context":"default","severity":"DEBUG","time":"2022-06-07T11:40:40.762"}
{"thread":"main","logger":"com.hazelcast.system.logo","message":"[10.244.0.44]:5701 [dev] [5.1.1] \n\t+ + o o o o---o o----o o o---o o o----o o--o--o\n\t+ + + + | | / \\ / | | / / \\ | | \n\t+ + + + + o----o o o o o----o | o o o o----o | \n\t+ + + + | | / \\ / | | \\ / \\ | | \n\t+ + o o o o o---o o----o o----o o---o o o o----o o ","context":"default","severity":"INFO","time":"2022-06-07T11:40:40.812"}
{"thread":"main","logger":"com.hazelcast.system","message":"[10.244.0.44]:5701 [dev] [5.1.1] Copyright (c) 2008-2022, Hazelcast, Inc. All Rights Reserved.","context":"default","severity":"INFO","time":"2022-06-07T11:40:40.813"}
{"thread":"main","logger":"com.hazelcast.system","message":"[10.244.0.44]:5701 [dev] [5.1.1] Hazelcast Platform 5.1.1 (20220317 - 5b5fa10) starting at [10.244.0.44]:5701","context":"default","severity":"INFO","time":"2022-06-07T11:40:40.813"}
{"thread":"main","logger":"com.hazelcast.system","message":"[10.244.0.44]:5701 [dev] [5.1.1] Cluster name: dev","context":"default","severity":"INFO","time":"2022-06-07T11:40:40.814"}
{"thread":"main","logger":"com.hazelcast.system","message":"[10.244.0.44]:5701 [dev] [5.1.1] Configured Hazelcast Serialization version: 1","context":"default","severity":"DEBUG","time":"2022-06-07T11:40:40.814"}
{"thread":"main","logger":"com.hazelcast.system","message":"[10.244.0.44]:5701 [dev] [5.1.1] Integrity Checker is disabled. Fail-fast on corrupted executables will not be performed.\nTo enable integrity checker do one of the following: \n - Change member config using Java API: config.setIntegrityCheckerEnabled(true);\n - Change XML/YAML configuration property: Set hazelcast.integrity-checker.enabled to true\n - Add system property: -Dhz.integritychecker.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)\n - Add environment variable: HZ_INTEGRITYCHECKER_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)","context":"default","severity":"INFO","time":"2022-06-07T11:40:40.815"}
{"thread":"main","logger":"com.hazelcast.system","message":"[10.244.0.44]:5701 [dev] [5.1.1] The Jet engine is disabled.\nTo enable the Jet engine on the members, do one of the following:\n - Change member config using Java API: config.getJetConfig().setEnabled(true)\n - Change XML/YAML configuration property: Set hazelcast.jet.enabled to true\n - Add system property: -Dhz.jet.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)\n - Add environment variable: HZ_JET_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)","context":"default","severity":"INFO","time":"2022-06-07T11:40:40.822"}
{"thread":"main","logger":"com.hazelcast.internal.util.ServiceLoader","message":"The class com.hazelcast.jet.impl.metrics.JetMetricsDataSerializerHook does not implement the expected interface com.hazelcast.nio.serialization.SerializerHook","context":"default","severity":"DEBUG","time":"2022-06-07T11:40:41.106"}
{"thread":"main","logger":"com.hazelcast.internal.util.ServiceLoader","message":"The class com.hazelcast.jet.impl.observer.JetObserverDataSerializerHook does not implement the expected interface com.hazelcast.nio.serialization.SerializerHook","context":"default","severity":"DEBUG","time":"2022-06-07T11:40:41.119"}
{"thread":"main","logger":"com.hazelcast.internal.util.ServiceLoader","message":"The class com.hazelcast.jet.impl.metrics.JetMetricsDataSerializerHook does not implement the expected interface com.hazelcast.nio.serialization.SerializerHook","context":"default","severity":"DEBUG","time":"2022-06-07T11:40:41.136"}
{"thread":"main","logger":"com.hazelcast.internal.util.ServiceLoader","message":"The class com.hazelcast.jet.impl.observer.JetObserverDataSerializerHook does not implement the expected interface com.hazelcast.nio.serialization.SerializerHook","context":"default","severity":"DEBUG","time":"2022-06-07T11:40:41.136"}
{"thread":"main","logger":"com.hazelcast.internal.metrics.impl.MetricsConfigHelper","message":"[10.244.0.44]:5701 [dev] [5.1.1] Collecting debug metrics and sending to diagnostics is disabled","context":"default","severity":"DEBUG","time":"2022-06-07T11:40:41.15"}
{"thread":"main","logger":"com.hazelcast.spi.impl.operationservice.impl.BackpressureRegulator","message":"[10.244.0.44]:5701 [dev] [5.1.1] Backpressure is disabled","context":"default","severity":"DEBUG","time":"2022-06-07T11:40:41.198"}
{"thread":"main","logger":"com.hazelcast.spi.impl.operationservice.impl.InboundResponseHandlerSupplier","message":"[10.244.0.44]:5701 [dev] [5.1.1] Running with 2 response threads","context":"default","severity":"DEBUG","time":"2022-06-07T11:40:41.232"}
{"thread":"main","logger":"com.hazelcast.internal.server.tcp.LocalAddressRegistry","message":"[10.244.0.44]:5701 [dev] [5.1.1] LinkedAddresses{primaryAddress=[10.244.0.44]:5701, allLinkedAddresses=[[fe80:0:0:0:7049:95ff:fe69:3b4d%eth0]:5701, [10.244.0.44]:5701, [fe80:0:0:0:7049:95ff:fe69:3b4d]:5701, [0:0:0:0:0:0:0:1]:5701, [0:0:0:0:0:0:0:1%lo]:5701, [127.0.0.1]:5701]} are registered for the local member with local uuid=dad735b3-e933-4af2-9756-50bcb47a3491","context":"default","severity":"DEBUG","time":"2022-06-07T11:40:41.326"}
{"thread":"main","logger":"com.hazelcast.spi.discovery.integration.DiscoveryService","message":"[10.244.0.44]:5701 [dev] [5.1.1] Kubernetes Discovery properties: { service-dns: my-app-hs, service-dns-timeout: 5, service-name: null, service-port: 0, service-label: null, service-label-value: true, namespace: null, pod-label: null, pod-label-value: null, resolve-not-ready-addresses: true, expose-externally-mode: AUTO, use-node-name-as-external-address: false, service-per-pod-label: null, service-per-pod-label-value: null, kubernetes-api-retries: 3, kubernetes-master: https://kubernetes.default.svc}","context":"default","severity":"INFO","time":"2022-06-07T11:40:41.54"}
{"thread":"main","logger":"com.hazelcast.spi.utils.RetryUtils","message":"Couldn't connect to the service, [1] retrying in 1 seconds...","context":"default","severity":"WARN","time":"2022-06-07T11:40:42.088"}
{"thread":"main","logger":"com.hazelcast.spi.utils.RetryUtils","message":"Couldn't connect to the service, [2] retrying in 2 seconds...","context":"default","severity":"WARN","time":"2022-06-07T11:40:43.621"}
{"thread":"main","logger":"com.hazelcast.spi.utils.RetryUtils","message":"Couldn't connect to the service, [3] retrying in 3 seconds...","context":"default","severity":"WARN","time":"2022-06-07T11:40:45.895"}
{"thread":"main","logger":"com.hazelcast.spi.discovery.integration.DiscoveryService","message":"[10.244.0.44]:5701 [dev] [5.1.1] Kubernetes Discovery activated with mode: DNS_LOOKUP","context":"default","severity":"INFO","time":"2022-06-07T11:40:49.284"}
{"thread":"main","logger":"com.hazelcast.system.security","message":"[10.244.0.44]:5701 [dev] [5.1.1] \n🔒 Security recommendations and their status:\n ⚠️ Use a custom cluster name\n ✅ Disable member multicast discovery/join method\n ⚠️ Use advanced networking, separate client and member sockets\n ⚠️ Bind Server sockets to a single network interface (disable hazelcast.socket.server.bind.any)\n ✅ Disable scripting in the Management Center\n ✅ Disable console in the Management Center\n ⚠️ Enable Security (Enterprise)\n ⚠️ Use TLS communication protection (Enterprise)\n ⚠️ Enable auditlog (Enterprise)\nCheck the hazelcast-security-hardened.xml/yaml example config file to find why and how to configure these security related settings.\n","context":"default","severity":"DEBUG","time":"2022-06-07T11:40:49.286"}
{"thread":"main","logger":"com.hazelcast.instance.impl.Node","message":"[10.244.0.44]:5701 [dev] [5.1.1] Using Discovery SPI","context":"default","severity":"INFO","time":"2022-06-07T11:40:49.366"}
{"thread":"main","logger":"com.hazelcast.cp.CPSubsystem","message":"[10.244.0.44]:5701 [dev] [5.1.1] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.","context":"default","severity":"WARN","time":"2022-06-07T11:40:49.372"}
{"thread":"main","logger":"com.hazelcast.internal.metrics.impl.MetricsService","message":"[10.244.0.44]:5701 [dev] [5.1.1] Configuring metrics collection, collection interval=5 seconds, retention=5 seconds, publishers=[Management Center Publisher, JMX Publisher]","context":"default","severity":"DEBUG","time":"2022-06-07T11:40:49.713"}
{"thread":"main","logger":"com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl","message":"[10.244.0.44]:5701 [dev] [5.1.1] Starting 8 partition threads and 5 generic threads (1 dedicated for priority tasks)","context":"default","severity":"DEBUG","time":"2022-06-07T11:40:49.734"}
{"thread":"main","logger":"com.hazelcast.sql.impl.SqlServiceImpl","message":"[10.244.0.44]:5701 [dev] [5.1.1] Optimizer class \"com.hazelcast.jet.sql.impl.CalciteSqlOptimizer\" not found, falling back to com.hazelcast.sql.impl.optimizer.DisabledSqlOptimizer","context":"default","severity":"DEBUG","time":"2022-06-07T11:40:49.741"}
{"thread":"main","logger":"com.hazelcast.internal.diagnostics.Diagnostics","message":"[10.244.0.44]:5701 [dev] [5.1.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.","context":"default","severity":"INFO","time":"2022-06-07T11:40:49.747"}
{"thread":"main","logger":"com.hazelcast.core.LifecycleService","message":"[10.244.0.44]:5701 [dev] [5.1.1] [10.244.0.44]:5701 is STARTING","context":"default","severity":"INFO","time":"2022-06-07T11:40:49.756"}
{"thread":"main","logger":"com.hazelcast.internal.partition.InternalPartitionService","message":"[10.244.0.44]:5701 [dev] [5.1.1] Adding Member [10.244.0.44]:5701 - dad735b3-e933-4af2-9756-50bcb47a3491 this","context":"default","severity":"DEBUG","time":"2022-06-07T11:40:49.757"}
{"thread":"main","logger":"com.hazelcast.internal.networking.nio.NioNetworking","message":"[10.244.0.44]:5701 [dev] [5.1.1] TcpIpConnectionManager configured with Non Blocking IO-threading model: 3 input threads and 3 output threads","context":"default","severity":"DEBUG","time":"2022-06-07T11:40:49.777"}
{"thread":"main","logger":"com.hazelcast.internal.networking.nio.NioNetworking","message":"[10.244.0.44]:5701 [dev] [5.1.1] write through enabled:true","context":"default","severity":"DEBUG","time":"2022-06-07T11:40:49.778"}
{"thread":"main","logger":"com.hazelcast.internal.networking.nio.NioNetworking","message":"[10.244.0.44]:5701 [dev] [5.1.1] IO threads selector mode is SELECT","context":"default","severity":"DEBUG","time":"2022-06-07T11:40:49.778"}
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.hazelcast.internal.networking.nio.SelectorOptimizer (file:/app/libs/hazelcast-5.1.1.jar) to field sun.nio.ch.SelectorImpl.selectedKeys
WARNING: Please consider reporting this to the maintainers of com.hazelcast.internal.networking.nio.SelectorOptimizer
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
{"thread":"main","logger":"com.hazelcast.internal.cluster.ClusterService","message":"[10.244.0.44]:5701 [dev] [5.1.1] Setting master address to null","context":"default","severity":"DEBUG","time":"2022-06-07T11:40:49.821"}
{"thread":"main","logger":"com.hazelcast.spi.discovery.integration.DiscoveryService","message":"[10.244.0.44]:5701 [dev] [5.1.1] DNS lookup for serviceDns 'my-app-hs' failed: unknown host","context":"default","severity":"WARN","time":"2022-06-07T11:40:49.833"}
{"thread":"main","logger":"com.hazelcast.spi.discovery.integration.DiscoveryService","message":"[10.244.0.44]:5701 [dev] [5.1.1] DNS lookup for serviceDns 'my-app-hs' failed: unknown host","context":"default","severity":"WARN","time":"2022-06-07T11:40:49.845"}
{"thread":"main","logger":"com.hazelcast.spi.discovery.integration.DiscoveryService","message":"[10.244.0.44]:5701 [dev] [5.1.1] DNS lookup for serviceDns 'my-app-hs' failed: unknown host","context":"default","severity":"WARN","time":"2022-06-07T11:40:49.865"}
{"thread":"main","logger":"com.hazelcast.spi.discovery.integration.DiscoveryService","message":"[10.244.0.44]:5701 [dev] [5.1.1] DNS lookup for serviceDns 'my-app-hs' failed: unknown host","context":"default","severity":"WARN","time":"2022-06-07T11:40:49.906"}
{"thread":"main","logger":"com.hazelcast.spi.discovery.integration.DiscoveryService","message":"[10.244.0.44]:5701 [dev] [5.1.1] DNS lookup for serviceDns 'my-app-hs' failed: unknown host","context":"default","severity":"WARN","time":"2022-06-07T11:40:49.987"}
{"thread":"main","logger":"com.hazelcast.spi.discovery.integration.DiscoveryService","message":"[10.244.0.44]:5701 [dev] [5.1.1] DNS lookup for serviceDns 'my-app-hs' failed: unknown host","context":"default","severity":"WARN","time":"2022-06-07T11:40:50.148"}
{"thread":"main","logger":"com.hazelcast.spi.discovery.integration.DiscoveryService","message":"[10.244.0.44]:5701 [dev] [5.1.1] DNS lookup for serviceDns 'my-app-hs' failed: unknown host","context":"default","severity":"WARN","time":"2022-06-07T11:40:50.469"}
{"thread":"main","logger":"com.hazelcast.spi.discovery.integration.DiscoveryService","message":"[10.244.0.44]:5701 [dev] [5.1.1] DNS lookup for serviceDns 'my-app-hs' failed: unknown host","context":"default","severity":"WARN","time":"2022-06-07T11:40:50.97"}
{"thread":"main","logger":"com.hazelcast.spi.discovery.integration.DiscoveryService","message":"[10.244.0.44]:5701 [dev] [5.1.1] DNS lookup for serviceDns 'my-app-hs' failed: unknown host","context":"default","severity":"WARN","time":"2022-06-07T11:40:51.471"}
{"thread":"main","logger":"com.hazelcast.spi.discovery.integration.DiscoveryService","message":"[10.244.0.44]:5701 [dev] [5.1.1] DNS lookup for serviceDns 'my-app-hs' failed: unknown host","context":"default","severity":"WARN","time":"2022-06-07T11:40:51.971"}
{"thread":"main","logger":"com.hazelcast.spi.discovery.integration.DiscoveryService","message":"[10.244.0.44]:5701 [dev] [5.1.1] DNS lookup for serviceDns 'my-app-hs' failed: unknown host","context":"default","severity":"WARN","time":"2022-06-07T11:40:52.472"}
{"thread":"main","logger":"com.hazelcast.spi.discovery.integration.DiscoveryService","message":"[10.244.0.44]:5701 [dev] [5.1.1] DNS lookup for serviceDns 'my-app-hs' failed: unknown host","context":"default","severity":"WARN","time":"2022-06-07T11:40:52.973"}
{"thread":"main","logger":"com.hazelcast.spi.discovery.integration.DiscoveryService","message":"[10.244.0.44]:5701 [dev] [5.1.1] DNS lookup for serviceDns 'my-app-hs' failed: unknown host","context":"default","severity":"WARN","time":"2022-06-07T11:40:53.474"}
{"thread":"main","logger":"com.hazelcast.spi.discovery.integration.DiscoveryService","message":"[10.244.0.44]:5701 [dev] [5.1.1] DNS lookup for serviceDns 'my-app-hs' failed: unknown host","context":"default","severity":"WARN","time":"2022-06-07T11:40:53.976"}
{"thread":"main","logger":"com.hazelcast.spi.discovery.integration.DiscoveryService","message":"[10.244.0.44]:5701 [dev] [5.1.1] DNS lookup for serviceDns 'my-app-hs' failed: unknown host","context":"default","severity":"WARN","time":"2022-06-07T11:40:54.477"}
{"thread":"main","logger":"com.hazelcast.internal.cluster.impl.DiscoveryJoiner","message":"[10.244.0.44]:5701 [dev] [5.1.1] This node will assume master role since none of the possible members accepted join request.","context":"default","severity":"DEBUG","time":"2022-06-07T11:40:54.979"}
{"thread":"main","logger":"com.hazelcast.internal.cluster.ClusterService","message":"[10.244.0.44]:5701 [dev] [5.1.1] Setting master address to [10.244.0.44]:5701","context":"default","severity":"DEBUG","time":"2022-06-07T11:40:54.979"}
{"thread":"main","logger":"com.hazelcast.internal.cluster.impl.MembershipManager","message":"[10.244.0.44]:5701 [dev] [5.1.1] Local member list join version is set to 1","context":"default","severity":"DEBUG","time":"2022-06-07T11:40:54.979"}
{"thread":"main","logger":"com.hazelcast.internal.cluster.impl.DiscoveryJoiner","message":"[10.244.0.44]:5701 [dev] [5.1.1] PostJoin master: [10.244.0.44]:5701, isMaster: true","context":"default","severity":"DEBUG","time":"2022-06-07T11:40:54.98"}
{"thread":"main","logger":"com.hazelcast.internal.cluster.ClusterService","message":"[10.244.0.44]:5701 [dev] [5.1.1] \n\nMembers {size:1, ver:1} [\n\tMember [10.244.0.44]:5701 - dad735b3-e933-4af2-9756-50bcb47a3491 this\n]\n","context":"default","severity":"INFO","time":"2022-06-07T11:40:54.98"}
{"thread":"main","logger":"com.hazelcast.core.LifecycleService","message":"[10.244.0.44]:5701 [dev] [5.1.1] [10.244.0.44]:5701 is STARTED","context":"default","severity":"INFO","time":"2022-06-07T11:40:54.996"}
{"thread":"main","logger":"org.example.app.proxy.GatewayConfiguration","message":"Adding routes for */gateway with backend http://collaboration-server/co-unblu and identity provider 'microsoft'","context":"default","severity":"INFO","time":"2022-06-07T11:40:55.218"}
{"thread":"main","logger":"org.example.app.proxy.GatewayConfiguration","message":"Adding public (unprotected) route '/gateway/rest/product/all'","context":"default","severity":"INFO","time":"2022-06-07T11:40:55.225"}
{"thread":"main","logger":"org.springframework.cloud.gateway.route.RouteDefinitionRouteLocator","message":"Loaded RoutePredicateFactory [After]","context":"default","severity":"INFO","time":"2022-06-07T11:40:56.433"}
{"thread":"main","logger":"org.springframework.cloud.gateway.route.RouteDefinitionRouteLocator","message":"Loaded RoutePredicateFactory [Before]","context":"default","severity":"INFO","time":"2022-06-07T11:40:56.434"}
{"thread":"main","logger":"org.springframework.cloud.gateway.route.RouteDefinitionRouteLocator","message":"Loaded RoutePredicateFactory [Between]","context":"default","severity":"INFO","time":"2022-06-07T11:40:56.434"}
{"thread":"main","logger":"org.springframework.cloud.gateway.route.RouteDefinitionRouteLocator","message":"Loaded RoutePredicateFactory [Cookie]","context":"default","severity":"INFO","time":"2022-06-07T11:40:56.434"}
{"thread":"main","logger":"org.springframework.cloud.gateway.route.RouteDefinitionRouteLocator","message":"Loaded RoutePredicateFactory [Header]","context":"default","severity":"INFO","time":"2022-06-07T11:40:56.434"}
{"thread":"main","logger":"org.springframework.cloud.gateway.route.RouteDefinitionRouteLocator","message":"Loaded RoutePredicateFactory [Host]","context":"default","severity":"INFO","time":"2022-06-07T11:40:56.435"}
{"thread":"main","logger":"org.springframework.cloud.gateway.route.RouteDefinitionRouteLocator","message":"Loaded RoutePredicateFactory [Method]","context":"default","severity":"INFO","time":"2022-06-07T11:40:56.435"}
{"thread":"main","logger":"org.springframework.cloud.gateway.route.RouteDefinitionRouteLocator","message":"Loaded RoutePredicateFactory [Path]","context":"default","severity":"INFO","time":"2022-06-07T11:40:56.435"}
{"thread":"main","logger":"org.springframework.cloud.gateway.route.RouteDefinitionRouteLocator","message":"Loaded RoutePredicateFactory [Query]","context":"default","severity":"INFO","time":"2022-06-07T11:40:56.435"}
{"thread":"main","logger":"org.springframework.cloud.gateway.route.RouteDefinitionRouteLocator","message":"Loaded RoutePredicateFactory [ReadBody]","context":"default","severity":"INFO","time":"2022-06-07T11:40:56.435"}
{"thread":"main","logger":"org.springframework.cloud.gateway.route.RouteDefinitionRouteLocator","message":"Loaded RoutePredicateFactory [RemoteAddr]","context":"default","severity":"INFO","time":"2022-06-07T11:40:56.436"}
{"thread":"main","logger":"org.springframework.cloud.gateway.route.RouteDefinitionRouteLocator","message":"Loaded RoutePredicateFactory [Weight]","context":"default","severity":"INFO","time":"2022-06-07T11:40:56.436"}
{"thread":"main","logger":"org.springframework.cloud.gateway.route.RouteDefinitionRouteLocator","message":"Loaded RoutePredicateFactory [CloudFoundryRouteService]","context":"default","severity":"INFO","time":"2022-06-07T11:40:56.436"}
{"thread":"main","logger":"org.springframework.boot.web.embedded.netty.NettyWebServer","message":"Netty started on port 80","context":"default","severity":"INFO","time":"2022-06-07T11:40:57.513"}
{"thread":"main","logger":"org.springframework.boot.actuate.endpoint.web.EndpointLinksResolver","message":"Exposing 6 endpoint(s) beneath base path '/actuator'","context":"default","severity":"INFO","time":"2022-06-07T11:40:57.654"}
{"thread":"main","logger":"org.springframework.boot.web.embedded.netty.NettyWebServer","message":"Netty started on port 8081","context":"default","severity":"INFO","time":"2022-06-07T11:40:57.728"}
{"thread":"main","logger":"org.example.app.MyApplicationKt","message":"Started MyApplicationKt in 25.429 seconds (JVM running for 26.302)","context":"default","severity":"INFO","time":"2022-06-07T11:40:57.817"}
{"thread":"hz.unruffled_matsumoto.cached.thread-2","logger":"com.hazelcast.internal.cluster.impl.MembershipManager","message":"[10.244.0.44]:5701 [dev] [5.1.1] Sending member list to the non-master nodes: \n\nMembers {size:1, ver:1} [\n\tMember [10.244.0.44]:5701 - dad735b3-e933-4af2-9756-50bcb47a3491 this\n]\n","context":"default","severity":"DEBUG","time":"2022-06-07T11:41:49.728"}
{"thread":"hz.ShutdownThread","logger":"com.hazelcast.instance.impl.Node","message":"[10.244.0.44]:5701 [dev] [5.1.1] Running shutdown hook... Current state: ACTIVE","context":"default","severity":"INFO","time":"2022-06-07T11:41:58.095"}
{"thread":"hz.ShutdownThread","logger":"com.hazelcast.core.LifecycleService","message":"[10.244.0.44]:5701 [dev] [5.1.1] [10.244.0.44]:5701 is SHUTTING_DOWN","context":"default","severity":"INFO","time":"2022-06-07T11:41:58.095"}
{"thread":"hz.ShutdownThread","logger":"com.hazelcast.instance.impl.Node","message":"[10.244.0.44]:5701 [dev] [5.1.1] Terminating forcefully...","context":"default","severity":"WARN","time":"2022-06-07T11:41:58.099"}
{"thread":"hz.ShutdownThread","logger":"com.hazelcast.internal.cluster.ClusterService","message":"[10.244.0.44]:5701 [dev] [5.1.1] Setting master address to null","context":"default","severity":"DEBUG","time":"2022-06-07T11:41:58.1"}
{"thread":"hz.ShutdownThread","logger":"com.hazelcast.instance.impl.Node","message":"[10.244.0.44]:5701 [dev] [5.1.1] Shutting down connection manager...","context":"default","severity":"INFO","time":"2022-06-07T11:41:58.1"}
{"thread":"hz.ShutdownThread","logger":"com.hazelcast.instance.impl.Node","message":"[10.244.0.44]:5701 [dev] [5.1.1] Shutting down node engine...","context":"default","severity":"INFO","time":"2022-06-07T11:41:58.102"}
{"thread":"hz.ShutdownThread","logger":"com.hazelcast.internal.cluster.ClusterService","message":"[10.244.0.44]:5701 [dev] [5.1.1] Setting master address to null","context":"default","severity":"DEBUG","time":"2022-06-07T11:41:58.108"}
{"thread":"hz.ShutdownThread","logger":"com.hazelcast.instance.impl.NodeExtension","message":"[10.244.0.44]:5701 [dev] [5.1.1] Destroying node NodeExtension.","context":"default","severity":"INFO","time":"2022-06-07T11:41:58.11"}
{"thread":"hz.ShutdownThread","logger":"com.hazelcast.instance.impl.Node","message":"[10.244.0.44]:5701 [dev] [5.1.1] Hazelcast Shutdown is completed in 12 ms.","context":"default","severity":"INFO","time":"2022-06-07T11:41:58.111"}
{"thread":"hz.ShutdownThread","logger":"com.hazelcast.core.LifecycleService","message":"[10.244.0.44]:5701 [dev] [5.1.1] [10.244.0.44]:5701 is SHUTDOWN","context":"default","severity":"INFO","time":"2022-06-07T11:41:58.111"}
{"thread":"SpringApplicationShutdownHook","logger":"com.hazelcast.core.LifecycleService","message":"[10.244.0.44]:5701 [dev] [5.1.1] [10.244.0.44]:5701 is SHUTTING_DOWN","context":"default","severity":"INFO","time":"2022-06-07T11:41:58.111"}
{"thread":"SpringApplicationShutdownHook","logger":"com.hazelcast.instance.impl.Node","message":"[10.244.0.44]:5701 [dev] [5.1.1] Node is already shutting down... Waiting for shutdown process to complete...","context":"default","severity":"INFO","time":"2022-06-07T11:41:58.111"}
{"thread":"SpringApplicationShutdownHook","logger":"com.hazelcast.core.LifecycleService","message":"[10.244.0.44]:5701 [dev] [5.1.1] [10.244.0.44]:5701 is SHUTDOWN","context":"default","severity":"INFO","time":"2022-06-07T11:41:58.111"}
Stream closed EOF for env-easing-grove/my-app-74975f549-jg9d6 (my-app)
Why is the application going down all the time? Because there is only one instance?
Update:
I verified with k9s with :svc the services. The services my-app and my-app-hs refer to the same pot.
But when starting with 2 replicas they will not find each other. So really the dns lookup fails in this kind cluster.

Answer 1

In my case the issue was cause by an issue with the configuration of the livenessProbe. The wrong port was configured for the liveness probe. Fixing the port to be the one of the spring actuator fixed the problem.
livenessProbe {
httpGet {
path = "/actuator/health/liveness"
this.port = IntOrString(managamentPort)
}
initialDelaySeconds = 60
failureThreshold = 6
}
General suggestion if you come across the same problem: Check your liveness and readiness probe.