Microsoft Orleans in kubernetes StatefulSet POD crashes after several restart - consul

Question

Microsoft Orleans v3.4.3
Consul Clustering
Running in K8S
siloBuilder
.UseConsulClustering(opt =>
{
opt.Address = new Uri(AppConfig.Orleans.ConsulUrl);
opt.AclClientToken = AppConfig.Orleans.AclClientToken;
})
.Configure<ClusterOptions>(options =>
{
options.ClusterId = AppConfig.Orleans.ClusterID;
options.ServiceId = AppConfig.Orleans.ServiceID;
})
.siloBuilder.UseKubernetesHosting();
I configured the labels and environment variables for my POD accordingly to the doc.
- name: ORLEANS_SERVICE_ID #Required by Orleans
valueFrom:
fieldRef:
fieldPath: metadata.labels['orleans/serviceId']
- name: ORLEANS_CLUSTER_ID #Required by Orleans
valueFrom:
fieldRef:
fieldPath: metadata.labels['orleans/clusterId']
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.labels['statefulset.kubernetes.io/pod-name']
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
It is a StatefulSet with only 1 POD for testing.
On initial startup, it works well.
However, every time when I restart the POD, a new entry is created in Consul.
And It crashes in subsequent startup.
The log says
System.AggregateException: One or more errors occurred. (Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Failed to get response from: [S10.18.123.218:11111:361110184])
---> Orleans.Runtime.MembershipService.OrleansClusterConnectivityCheckFailedException: Failed to get ping responses from 1 of 1 active silos. Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster. Successfully contacted: []. Failed to get response from: [S10.18.123.218:11111:361110184]
at Orleans.Runtime.MembershipService.MembershipAgent.ValidateInitialConnectivity()
at Orleans.Runtime.MembershipService.MembershipAgent.BecomeActive()
at Orleans.Runtime.MembershipService.MembershipAgent.<>c__DisplayClass26_0.<<Orleans-ILifecycleParticipant<Orleans-Runtime-ISiloLifecycle>-Participate>g__OnBecomeActiveStart|6>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStart(CancellationToken ct)
at Orleans.LifecycleSubject.OnStart(CancellationToken ct)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute()
at Orleans.Runtime.Silo.StartAsync(CancellationToken cancellationToken)
at Orleans.Hosting.SiloHost.StartAsync(CancellationToken cancellationToken)
at Orleans.Hosting.SiloHostedService.StartAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at UBS.OrleansServer.EntryPoint.Start() in /app/UBS/OrleansServer/EntryPoint.cs:line 102
--- End of inner exception stack trace ---
I have to remove all the entries in Consul then restart the POD, then everything works fine.
The POD_NAME is the same for StatefulSet's POD, is it correct that each POD restart creates a new entry in Consul?
What could be the cause?
Thanks in advance
UPDATE
After several rounds crashes and restart, finally it does not crash any more. And in log I see the following message
ProcessTableUpdate (called from DeclareDead) membership table: 5 silos, 1 are Active, 4 are Dead, Version=<31, 28123>. All silos: [SiloAddress=S10.18.123.244:11111:361163684 SiloName=ubs-job-dev-0 Status=Active, SiloAddress=S10.18.123.200:11111:361158057 SiloName=ubs-job-dev-0 Status=Dead, SiloAddress=S10.18.123.210:11111:361161905 SiloName=ubs-job-dev-0 Status=Dead, SiloAddress=S10.18.123.217:11111:361157424 SiloName=ubs-job-dev-0 Status=Dead, SiloAddress=S10.18.123.244:11111:361163558 SiloName=ubs-job-dev-0 Status=Dead]
The SiloName never changes and there is only one POD in StatefulSet, but it sees 5 silos, 4 of them are dead. It seems each new POD, even if pod name does not change, is seen as a new silo. Is that expected?

Answer 1

(Failed to get ping responses from 1 of 1 active silos.
Newly joining silos validate connectivity with all active silos that have recently updated their 'I Am Alive' value before joining the cluster.
Successfully contacted: []. Failed to get response from: [S10.18.123.218:11111:361110184])
Looks like your membershiptable (in consul) thinks that you already have active silos in it. When your 'new' silo comes up and looks in the membershiptable it sees these active silos at the table's IP addresses.
To keep the cluster correct, a newly joining silo must be able to communicate with the existing silos. However if the membership table is incorrect (ip address with status 3/active) then you have a problem where the new silo is trying to ping the active silos and not being able to reach them will fail to join and fast itself.
You have a couple of solutions:
clear the consul table when deploying your solution
change the deploymentid on every deployment.
You obviously found the first solution (clear the table)
see silo lifecycle