Initial call on ServiceFabric proxy is VERY slow - performance

Question

Whenever a I'm calling one service fabric service from another, the first call on the proxy is VERY slow i.e. 100x slower than all subsequent calls. I've put timings in that record the time immediately before the call and then the time immediately in the service method being called and this can easily be over 60 seconds! The service fabric cluster is a standalone cluster running on 12 nodes/VM's.
Interestingly the length of time the first call takes seems to be related to the number of nodes i.e. if I deactivate half the nodes the time is reduced (though not by half). Also when running the exact same code on a dev cluster running on my local PC the length of time the first call take is typically around 8 second with subsequent calls taking < 10ms on either system. In addition, creating another proxy to the same service in the same client process still result in fast call times, it seem as if the proxy factory (which I believe SF caches per client process) is created on first use of the proxy and take a very long time.
Interestingly no exceptions are thrown and the services actually work!
So my question is, why does it take so long the first time a call is made from one service to another on a proxy created with ServiceProxy.Create()?

Answer 1

According to The SF remoting docs (See below, emphasis mine), ServiceProxy.Create is a wrapper around ServiceProxyFactory, and the first call also involves setting up the factory for the subsequent calls.
ServiceProxyFactory is a factory that creates proxy for different remoting interfaces. If you use API ServiceProxy.Create for creating proxy, then framework creates the singleton ServiceProxyFactory. It is useful to create one manually when you need to override IServiceRemotingClientFactory properties. Factory is an expensive operation. ServiceProxyFactory maintains cache of communication client. Best practice is to cache ServiceProxyFactory for as long as possible.

Answer 2

I have not experienced slow resolution anywhere near what you have, however I create my proxies when my API service starts up using dependency injection.
The way I have my system set up is that the stateless API service (asp.net core) communicates with the backend SF services.
It's possible that I am actually experiencing a longer delay, but by the time I go to use the application the resolution process has already started and finished, rather than the resolution starting when I make the first request to the app.
private void InitializeContainer(IApplicationBuilder app)
{
// Add application presentation components:
Container.RegisterMvcControllers(app);
Container.RegisterMvcViewComponents(app);
// Add application services.
Container.Register(() => ServiceProxy.Create<IContestService>(FabricUrl.ContestService), Lifestyle.Transient);
Container.Register(() => ServiceProxy.Create<IFriendService>(FabricUrl.FriendService), Lifestyle.Transient);
Container.Register(() => ServiceProxy.Create<IUserService>(FabricUrl.UserService), Lifestyle.Transient);
Container.Register(() => ServiceProxy.Create<IBillingService>(FabricUrl.BillingService), Lifestyle.Transient);
Container.RegisterSingleton(AutoMapperApi.Configure());
// Cross-wire ASP.NET services (if any). For instance:
Container.RegisterSingleton(app.ApplicationServices.GetService<ILoggerFactory>());
// NOTE: Prevent cross-wired instances as much as possible.
// See: https://simpleinjector.org/blog/2016/07/
}