Apache Ignite cache freeze after a period of time - caching

Question

I have a cluster of 6 server and 1 client nodes. My client node does a lot of update and create jobs and there is also an expiry policy in order to catch the expired items.
But every day, the cluster freezes at least once. Even ignitevisor's cache command freeze during the call.
So I have looked at thread dump and I saw a strange thing that , there a a lot of same statements like this:
"pub-#39%null%" #51 prio=5 os_prio=0 tid=0x00007f9788623800 nid=0x1d02 waiting on condition [0x00007f9769ddc000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c004aaa8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
So there are plenty of threads waiting for a condition but somehow it never happens.
My cache config is like:
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<!--
Alter configuration below as needed.
-->
<bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<!-- Configure internal thread pool. -->
<property name="publicThreadPoolSize" value="64"/>
<!-- Configure system thread pool. -->
<property name="systemThreadPoolSize" value="32"/>
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
...
</list>
</property>
</bean>
</property>
</bean>
</property>
<property name="cacheConfiguration">
<list>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="asd1"/>
<property name="eagerTtl" value="true"/>
<property name="expiryPolicyFactory">
<bean class="javax.cache.configuration.FactoryBuilder.SingletonFactory">
<constructor-arg name="instance">
<bean class="javax.cache.expiry.TouchedExpiryPolicy">
<constructor-arg name="expiryDuration">
<bean class="javax.cache.expiry.Duration">
<constructor-arg name="timeUnit">
<value type="java.util.concurrent.TimeUnit">MILLISECONDS</value>
</constructor-arg>
<constructor-arg name="durationAmount" value="10800000"/>
</bean>
</constructor-arg>
</bean>
</constructor-arg>
</bean>
</property>
</bean>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="asd2"/>
<property name="eagerTtl" value="true"/>
<property name="expiryPolicyFactory">
<bean class="javax.cache.configuration.FactoryBuilder.SingletonFactory">
<constructor-arg name="instance">
<bean class="javax.cache.expiry.TouchedExpiryPolicy">
<constructor-arg name="expiryDuration">
<bean class="javax.cache.expiry.Duration">
<constructor-arg name="timeUnit">
<value type="java.util.concurrent.TimeUnit">MILLISECONDS</value>
</constructor-arg>
<constructor-arg name="durationAmount" value="86400000"/>
</bean>
</constructor-arg>
</bean>
</constructor-arg>
</bean>
</property>
</bean>
</list>
</property>
<property name="includeEventTypes" value="70"/>
</bean>
</beans>
I really need help here. Thx

Answer 1

This was discussed on Apache Ignite forum: http://apache-ignite-users.70518.x6.nabble.com/Apache-Ignite-cluster-freeze-after-a-period-of-time-td7726.html
Most likely the deadlock is caused by reordered keys within a putAll operation.