JBoss Cache JGroups Membership GMS TCPPING - caching

Question

I am using JBoss Cache (tried versions 3.2.7 and 3.1.0) to have a replicated Map for caching data between Application Servers. In the past I did some checks if it works out and it did. My testenvironment always were 2 nodes in the same network segment.
Since IT deparments sometimes have a problem with UDP, we use TCP (TCPPING for discovery).
Now a customer reported problems with our nodes losing their sync, not replicating data.
They have 4 nodes in 2 subnets (2 and 2). They say when they onbly use 2 nodes in any subnet it works. When they start the third, the problems begin.
The logfile state a lot of "merge" problems indicating a partitioning problem.
So I did my own tests in my company. My setup is my Laptop under Windows and two virtual machines with Ubuntu. The virtual machines use bridged networking interfaces. DHCP is used and our IT department provides my 3 nodes with different subnet IPs. My Host Laptop is in a different net than the virtual machines are. TCP Communication between the nodes works. There should be no firewall involved.
So much to my setup.
I wrote a little Program that just initializes JBoss Cache, gets the cache(MAP) and in an intervall changes values in the map, and afterwards showing the content of the whole map. Pretty simple, 2 Classes involved.
My JBoss Cache setup is as follows:
<?xml version="1.0" encoding="UTF-8"?>
<jbosscache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="urn:jboss:jbosscache-core:config:3.1">
<clustering mode="replication" clusterName="${jgroups.clustername:DEFAULT}">
<stateRetrieval timeout="20000" fetchInMemoryState="true" />
<sync replTimeout="20000" />
<jgroupsConfig>
<TCP start_port="${jgroups.tcpping.start_port:7800}" loopback="true" recv_buf_size="20000000"
send_buf_size="640000" discard_incompatible_packets="true"
max_bundle_size="64000" max_bundle_timeout="30"
use_incoming_packet_handler="true" enable_bundling="false"
use_send_queues="false" sock_conn_timeout="3000"
skip_suspected_members="true" use_concurrent_stack="true"
thread_pool.enabled="true" thread_pool.min_threads="1"
thread_pool.max_threads="25" thread_pool.keep_alive_time="5000"
thread_pool.queue_enabled="false" thread_pool.queue_max_size="100"
thread_pool.rejection_policy="run" oob_thread_pool.enabled="true"
oob_thread_pool.min_threads="1" oob_thread_pool.max_threads="8"
oob_thread_pool.keep_alive_time="5000"
oob_thread_pool.queue_enabled="false"
oob_thread_pool.queue_max_size="100"
oob_thread_pool.rejection_policy="run" />
<TCPPING timeout="3000"
initial_hosts="${jgroups.tcpping.initial_hosts:localhost[7800],localhost[7801]}"
port_range="3" num_initial_members="3" />
<MERGE2 max_interval="100000" min_interval="20000" />
<MERGE3 max_interval="100000" min_interval="20000" />
<FD_SOCK />
<FD timeout="10000" max_tries="5" shun="true" />
<VERIFY_SUSPECT timeout="1500" />
<BARRIER />
<pbcast.NAKACK use_mcast_xmit="false" gc_lag="0"
retransmit_timeout="300,600,1200,2400,4800" discard_delivered_msgs="true" />
<UNICAST timeout="300,600,1200" />
<pbcast.STABLE stability_delay="1000"
desired_avg_gossip="50000" max_bytes="400000" />
<VIEW_SYNC avg_send_interval="60000" />
<pbcast.GMS print_local_addr="true" join_timeout="6000"
shun="true" view_bundling="true" />
<FC max_credits="2000000" min_threshold="0.10" />
<FRAG2 frag_size="60000" />
<pbcast.STREAMING_STATE_TRANSFER />
</jgroupsConfig>
</clustering>
</jbosscache>
starting my test nodes I provide them with the following System properties.
-Djgroups.bind_addr=NODE1
-Djgroups.tcpping.initial_hosts=NODE1[7900],NODE2[7900],NODE3[7900]
-Djgroups.tcpping.start_port=7900
From the logging messages I can see the GMS message that the NODE address is indeed as specified NODE-X[7900] for all nodes.
NODE1-3 are given as IP numbers. Those IP numbers can be reached from other nodes.
NODE1,2 are in the same subnet
NODE3 is in a different subnet
I did an incredible amount of tests, changing the config, the JBossCache version, the combinations of nodes running etc.
Sometimes it works, sometimes it doesn't.
One cause that seems to influence if the members find each other is the initial hosts info. Depending on the order of the hosts, if there are additional hosts to the 3 given, leaving out the own ip out of the list the setup works or not. It also depends on the order the nodes are started.
I am sure it has to do with the JGroups Group membership. Maybe parameters need to be added to make it more robust.
I really would appreciate some hints on what to try in order to get the nodes talking to each other reliably.
I addition to trying to figure out the problems with JBoss Cache(JGroups) I did the same test using Hazelcast (TCP) instead. It PERFECTLY works without any problem, so the basic networking should work for the nodes.
I consider to switch to Hazelcast, but this requires redeployments in several of our customers IT departments and I would want to avoid that.