Using spring-session-hazelcast on Kubernetes with service-dns causing SplitBrainMergeValidationOp ERROR - spring-boot

Question

We are deploying a spring-boot application using spring-session-hazelcast + hazelcast-kubernetes on an OpenShift/Kubernetes cluster.
Due to the nature of our platform, we can only use service-dns configuration. We expose a service on port 5701 for multicasting and set service-dns property to the multicast service name.
Below is a snippet for creation of the Hazelcast instance.
#Bean
public HazelcastInstance hazelcastInstance() {
var config = new Config();
config.setClusterName("spring-session-cluster");
var join = config.getNetworkConfig().getJoin();
join.getTcpIpConfig().setEnabled(false);
join.getMulticastConfig().setEnabled(false);
join.getKubernetesConfig().setEnabled(true)
.setProperty("service-dns", "<multicast-service-name>");
var attribute = new AttributeConfig()
.setName(Hazelcast4IndexedSessionRepository.PRINCIPAL_NAME_ATTRIBUTE)
.setExtractorClassName(Hazelcast4PrincipalNameExtractor.class.getName());
config.getMapConfig(Hazelcast4IndexedSessionRepository.DEFAULT_SESSION_MAP_NAME)
.addAttributeConfig(attribute)
.addIndexConfig(new IndexConfig(IndexType.HASH, Hazelcast4IndexedSessionRepository.PRINCIPAL_NAME_ATTRIBUTE));
var serializer = new SerializerConfig();
serializer.setImplementation(new HazelcastSessionSerializer())
.setTypeClass(MapSession.class);
config.getSerializationConfig().addSerializerConfig(serializer);
return Hazelcast.newHazelcastInstance(config);
}
When we run 2 pods for this application, we see the below ERROR log:
com.hazelcast.internal.cluster.impl.operations.SplitBrainMergeValidationOp
Message: [<private-ip>]:5701 [spring-session-cluster] [4.2] Target is this node! -> [<private-ip>]:5701
Can someone please explain how to fix this error, still using "service-dns" configuration?

Answer 1

You need to enable headless mode for your service in openshift.
https://github.com/hazelcast/hazelcast-kubernetes#dns-lookup

Answer 2

Just add configuration for split brain protection
SplitBrainProtectionConfig splitBrainProtectionConfig = new SplitBrainProtectionConfig();
splitBrainProtectionConfig.setName("splitBrainProtectionRuleWithFourMembers")
.setEnabled(true)
.setMinimumClusterSize(4);
MapConfig mapConfig = new MapConfig();
mapConfig.setSplitBrainProtectionName("splitBrainProtectionRuleWithFourMembers");
Config config = new Config();
config.addSplitBrainProtectionConfig(splitBrainProtectionConfig);
config.addMapConfig(mapConfig);
You can read more about this in hazelcast documentation:
https://docs.hazelcast.com/imdg/4.2/network-partitioning/split-brain-protection.html