Opendaylight Data Tree poisoned after leader switch - cluster-computing

Question

We are using Nitrogen version (SR1) of ODL. We are trying 2 node cluster and when follower becomes leader (after running the server for 5 - 6 hrs) we observe below exception in karaf.log. When below exception happens we are unable to access MDSAL for any read/write operations.
We call switchAllLocalShardsState API to change follower to be new leader based on "Akka Member Removed" event.
1) What does "poisoned" and "No Progress Exception" refer here.
2) Does "TERM" that we pass as an argument to switchAllLocalShardsState
cause this issue. If yes, please explain the significance of TERM
and also let us know why we are facing this issue only after
running the server for a long time.
2018-06-04 11:58:25,452 | ERROR | tAdminThread #20 | 130 - com.fujitsu.fnc.sdn.fw-scheduler-odl - 5.1.0.SNAPSHOT | SchedulerServiceImpl | Get Schedule List Transaction failed : ReadFailedException{message=read execution failed, errorList=[RpcError [message=read execution failed, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=ReadFailedException{message=read execution failed, errorList=[RpcError [message=read execution failed, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight.controller.cluster.access.client.NoProgressException: No progress in 31198 seconds]]}]]}
2018-06-04 11:58:25,452 | WARN | tAdminThread #12 | 261 - com.fujitsu.fnc.sdnfw.security-odl - 5.1.0.SNAPSHOT | IDMLightServer | getContainer: {}
java.util.concurrent.ExecutionException: ReadFailedException{message=read execution failed, errorList=[RpcError [message=read execution failed, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight.controller.cluster.access.client.NoProgressException: No progress in 31198 seconds]]}
at org.opendaylight.yangtools.util.concurrent.MappingCheckedFuture.wrapInExecutionException(MappingCheckedFuture.java:65)[583:org.opendaylight.yangtools.util:1.2.1]
at org.opendaylight.yangtools.util.concurrent.MappingCheckedFuture.get(MappingCheckedFuture.java:78)[583:org.opendaylight.yangtools.util:1.2.1]
at com.fujitsu.fnc.sdnfw.aaa.idmlight.impl.IDMLightServer.getUsermgmt(IDMLightServer.java:3056)[261:com.fujitsu.fnc.sdnfw.security-odl:5.1.0.SNAPSHOT]
at com.fujitsu.fnc.sdnfw.aaa.idmlight.impl.IDMLightServer.init(IDMLightServer.java:1977)[261:com.fujitsu.fnc.sdnfw.security-odl:5.1.0.SNAPSHOT]
at com.fujitsu.fnc.sdnfw.aaa.idmlight.impl.IDMLightServer.handleEvent(IDMLightServer.java:3251)[261:com.fujitsu.fnc.sdnfw.security-odl:5.1.0.SNAPSHOT]
at Proxyadd8c855_db4c_4e72_b600_2ca57cba8d4d.handleEvent(Unknown Source)[:]
at org.apache.felix.eventadmin.impl.handler.EventHandlerProxy.sendEvent(EventHandlerProxy.java:415)[393:org.apache.karaf.services.eventadmin:4.0.10]
at org.apache.felix.eventadmin.impl.tasks.HandlerTask.run(HandlerTask.java:90)[393:org.apache.karaf.services.eventadmin:4.0.10]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_66]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_66]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_66]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_66]
at java.lang.Thread.run(Thread.java:745)[:1.8.0_66]
Suppressed: java.lang.IllegalStateException: Uncaught exception occured during closing transaction
at org.opendaylight.controller.cluster.databroker.AbstractDOMBrokerTransaction.closeSubtransactions(AbstractDOMBrokerTransaction.java:92)[497:org.opendaylight.controller.sal-distributed-datastore:1.6.1]
at org.opendaylight.controller.cluster.databroker.DOMBrokerReadOnlyTransaction.close(DOMBrokerReadOnlyTransaction.java:50)[497:org.opendaylight.controller.sal-distributed-datastore:1.6.1]
at org.opendaylight.controller.md.sal.binding.impl.BindingDOMReadTransactionAdapter.close(BindingDOMReadTransactionAdapter.java:36)[484:org.opendaylight.controller.sal-binding-broker-impl:1.6.1]
at com.fujitsu.fnc.sdnfw.aaa.idmlight.impl.IDMLightServer.getUsermgmt(IDMLightServer.java:3062)[261:com.fujitsu.fnc.sdnfw.security-odl:5.1.0.SNAPSHOT]
... 10 more
Caused by: java.lang.IllegalStateException: Connection ConnectedClientConnection{client=ClientIdentifier{frontend=member-1-frontend-datastore-config, generation=1}, cookie=0, poisoned=org.opendaylight.controller.cluster.access.client.NoProgressException: No progress in 31198 seconds, backend=ShardBackendInfo{actor=Actor[akka.tcp://opendaylight-cluster-data#u446.nms.fnc.fujitsu.com:2550/user/shardmanager-config/member-2-shard-default-config#-1532234096], sessionId=0, version=BORON, maxMessages=1000, cookie=0, shard=default, dataTree=absent}} has been poisoned
at org.opendaylight.controller.cluster.access.client.AbstractClientConnection.commonEnqueue(AbstractClientConnection.java:198)[467:org.opendaylight.controller.cds-access-client:1.2.1]

Answer 1

The NoProgressException indicates you've enabled the new tell-based that was initially introduced in Nitrogen. It's designed to be more resilient to transient comm failures but is still experimental and there have been fixes since Nitrogen. I would suggest disabling it.
Also, there's a better way to implement 2-node primary/secondary by making the secondary non-voting and then promoting the secondary to leader by switching it to voting when the primary fails. This is documented in the online clustering guide in the Geo redundancy section which describes it for 6 nodes but you can use it for 2 nodes - same concept.