Hazelcast client metrics have no value (are always 0.0) - spring-boot

Question

Background / Project Setup:
We are developing a (micro-) service in Springboot (`2.0.4.RELEASE`) with JCache (`javax.cache:cache-api:1.1.0`). We recently switched from Ehcache to Hazelcast (`3.10.4`) to have a central cache cluster for our distributed microservices.
We furthermore use Prometheus (`io.micrometer:micrometer-registry-prometheus:1.0.6`) to export important metrics. After switching, the exported cache metrics do not have any value other than 0.0.
Details:
I use the following Spring configuration for Hazelcast (deleted non-relevant imports)
import org.springframework.cache.CacheManager;
import com.hazelcast.client.HazelcastClient;
import com.hazelcast.client.config.ClientConfig;
import com.hazelcast.core.HazelcastInstance;
import com.hazelcast.spring.cache.HazelcastCacheManager;
#Configuration
public class HazelcastCacheConfig {
#Bean
public ClientConfig config() {
ClientConfig config = new ClientConfig();
// set group and network config
return config;
}
#Bean
#DependsOn("config")
public HazelcastInstance hazelcastInstance() {
return HazelcastClient.newHazelcastClient(config());
}
#Bean
#DependsOn("hazelcastInstance")
public CacheManager cacheManager() {
return new HazelcastCacheManager(hazelcastInstance());
}
}
Our project requires to create caches dynamically on the fly. So I implemented a custom CacheResolver to create and register these caches and their corresponding metrics:
import org.springframework.boot.actuate.metrics.cache.CacheMetricsRegistrar;
import org.springframework.cache.Cache;
import org.springframework.cache.CacheManager;
import io.micrometer.core.instrument.binder.cache.HazelcastCacheMetrics;
import io.micrometer.prometheus.PrometheusMeterRegistry;
#Component
public class CacheManagement implements CacheResolver {
#Autowired
CacheManager cacheManager;
#Autowired
CacheMetricsRegistrar cacheMetricsRegistrar;
#Autowired
PrometheusMeterRegistry meterRegistry;
#Override
public Collection<? extends Cache> resolveCaches(CacheOperationInvocationContext<?> context) {
String cacheName = context.getMethod().getAnnotation(Cacheable.class).cacheNames()[0];
Cache cache = cacheManager.getCache("some Name");
// checks if cache already exists in io.micrometer.prometheus.PrometheusMeterRegistry
if (!cacheRegistered(cache)) {
if (cache.getNativeCache() instanceof IMap<?, ?>)
HazelcastCacheMetrics.monitor(meterRegistry, (IMap<?, ?>) cache.getNativeCache(), /*some tags*/);
// same result with this
// cacheMetricsRegistrar.bindCacheToRegistry(cache, /*some tags*/)
}
}
return cache;
}
Finally I annotate the chacheable methods with
#Cacheable(
cacheNames = "someGeneratedName",
cacheResolver = "cacheManagement",
keyGenerator = "cacheKeyGenerator",
unless = /*..*/,
condition = /*..*/
)
public Object someCacheableMethod(Object... someParameters) {
// logic
}
Now caching works great. The caches are generated at runtime and through debugging I could verify that the caching mechanism works as expected. The metrics are also exported through Prometheus. The only problem is that all caching related metrics always have a value of 0.0.
With debugging I discovered, that the setHits(long hits) method in com.hazelcast.monitor.impl.LocalMapStatsImpl is never called. So when Prometheus scraping leads to getHits() being called, it always returns 0.
What else I tried:
Let Spring generate the CacheManager bean: same result, Spring wraps a HazelcastClientCacheManager in the generated CacheManager bean.
Inject a JCacheCacheManager bean in CacheManagement. Spring still wraps a HazelcastClientCacheManager bean in the JCacheCacheManager but now only JCache cache metrics are exported, none of the Hazelcast specific ones (like cache_partition_gets_total which I thought has to be exported as an alternative to the cache_gets_total{result="miss"} metric according to micrometer issue #586). All values are still 0.0
One last thought/idea that I have is that caching metrics need to enabled on the Hazelcast members somehow but I could not find any information on this.

Answer 1

#steve-mcgarrett, JCache stats in Hazelcast is disabled by default. You need to enable it, either programmatically or adding below config to hazelcast.xml file:
<cache name="default">
<statistics-enabled>false</statistics-enabled>
</cache>
Please see: http://docs.hazelcast.org/docs/latest/manual/html-single/index.html#jcache-declarative-configuration