ISPN000217 while adding data to infinispan cache - spring-boot

Question

I have an application running on 4 nodes inside 2 clusters. The application is having cache configured using infinispan and SpringEmbeddedCacheManager. I am getting an intermittent issue while I am trying to add data to cache, please note that I am adding data as key value pair where my value will be always custom class created.
I just tried to change the cache type to replicated, local and invalidated, I have observed that I am not having issue when using local or invalidated cache. Can anyone confirm if large object in distributed cache cause any issue.
Infinispan Config
<distributed-cache name="apigw-access-cache" owners="1" segments="20" mode="SYNC" statistics="false">
<eviction max-entries="10" strategy="LIRS"/>
<expiration max-idle="360000" lifespan="3600000"/>
</distributed-cache>
Infinispan Version
<dependency>
<groupId>org.infinispan</groupId>
<artifactId>infinispan-spring4</artifactId>
<version>7.0.3.Final</version><!--$NO-MVN-MAN-VER$ -->
</dependency>
<dependency>
<groupId>org.infinispan</groupId>
<artifactId>infinispan-cli-server</artifactId>
<version>7.0.0.CR1</version>
</dependency>
Errors
2019-12-04 09:44:23.361 [qtp1933072581-15447] ERROR o.i.i.InvocationContextInterceptor - ISPN000136: Execution error
org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node-10097-32028, see cause for remote stack trace
at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44) ~[infinispan-core-7.0.3.Final.jar!/:7.0.3.Final]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:381) ~[infinispan-core-7.0.3.Final.jar!/:7.0.3.Final]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:167) ~[infinispan-core-7.0.3.Final.jar!/:7.0.3.Final]
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:560) ~[infinispan-core-7.0.3.Final.jar!/:7.0.3.Final]
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:290) ~[infinispan-core-7.0.3.Final.jar!/:7.0.3.Final]
Caused by: java.lang.IllegalArgumentException: Can not set java.util.Set field Class.field to java.lang.String
at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:167) ~[na:1.8.0_121]
at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:171) ~[na:1.8.0_121]
at sun.reflect.UnsafeObjectFieldAccessorImpl.set(UnsafeObjectFieldAccessorImpl.java:81) ~[na:1.8.0_121]
at java.lang.reflect.Field.set(Field.java:764) ~[na:1.8.0_121]
Caused by: org.infinispan.commons.CacheException: Problems invoking command.
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:221)
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
Caused by: org.infinispan.commons.CacheException: Problems invoking command.
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:221) ~[infinispan-core-7.0.3.Final.jar!/:7.0.3.Final]
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460) ~[jgroups-3.6.1.Final.jar!/:3.6.1.Final]
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377) ~[jgroups-3.6.1.Final.jar!/:3.6.1.Final]

Answer 1

First off, you should not be using such an old version of Infinispan, you should upgrade to 9.4.17.Final
The stack trace fragments don't appear to be in the right order, but Can not set java.util.Set field Class.field to java.lang.String is because you two nodes have different versions of the same class.
The biggest difference between distributed and invalidation caches is that distributed caches replicate values to other nodes, while invalidation caches send an invalidation message that includes only the key. If an invalidation cache works, then the problem is almost certainly that one of your value classes has changed and one of the nodes still has the old version.