Fixing "The web application [ROOT] created a ThreadLocal with key of type [com.netflix.hystrix.Hystrix$1]" - spring-boot

Question

I'm using Netflix cloud in a spring-boot API Gateway and I building the spring-boot app as a WAR that can be run standalone or deployed to a Tomcat container. When the spring-boot app is redeployed in Tomcat, for instance, using the Cargo Maven plugin, Tomcat MemoryLeakDetection complains about "The web application [ROOT] created a ThreadLocal with key of type [com.netflix.hystrix.Hystrix$1]".
After redeploying my spring-boot app 9 times Tomcat runs out of memory. How do I remove the Hystrix ThreadLocal so that it doesn't cause the WebappClassLoader instance to stick around every time I redeploy my spring-boot app. The result is that each redeploy leaves an instance of WebappClassLoader that can't be Garbage Collected because of the Hystrix ThreadLocal?
Here's the out-of-memory stacktrace:
29-Apr-2016 12:21:50.721 SEVERE [http-apr-8080-exec-38] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [my-spring-boot-app] created a ThreadLocal with key of type [com.netflix.hystrix.Hystrix$1] (value [com.netflix.hystrix.Hystrix$1#8302924]) and a value of type [java.util.LinkedL
ist] (value [[]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
29-Apr-2016 12:22:00.484 INFO [http-apr-8080-exec-38] org.apache.catalina.startup.HostConfig.undeploy Undeploying context [/my-spring-boot-app]
29-Apr-2016 12:22:09.353 INFO [http-apr-8080-exec-33] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive D:\apache-tomcat-8.0.28\webapps\my-spring-boot-app.war
12:23:08,056 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
12:23:08,089 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - Setting ReconfigureOnChangeFilter scanning period to 30 seconds
12:23:08,089 |-INFO in ReconfigureOnChangeFilter{invocationCounter=0} - Will scan for changes in [[D:\apache-tomcat-8.0.28\webapps\my-spring-boot-app\WEB-INF\classes\logba
ck.xml]] every 30 seconds.
12:23:08,089 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - Adding ReconfigureOnChangeFilter as a turbo filter
12:23:08,106 |-INFO in ch.qos.logback.core.joran.action.StatusListenerAction - Added status listener of type [ch.qos.logback.core.status.OnConsoleStatusListener]
12:23:08,118 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
12:23:08,136 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [CONSOLE]
12:23:08,176 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
java.lang.OutOfMemoryError: Metaspace
Dumping heap to java_pid15972.hprof ...
Heap dump file created [206407579 bytes in 1.201 secs]
29-Apr-2016 12:23:10.870 SEVERE [http-apr-8080-exec-33] org.apache.tomcat.util.modeler.BaseModelMBean.invoke Exception invoking method check
java.lang.OutOfMemoryError: Metaspace
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2496)
at org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:860)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1302)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1167)
at org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:97)
at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:55)
at org.slf4j.LoggerFactory.bind(LoggerFactory.java:141)
at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:655)
at org.springframework.boot.context.web.SpringBootServletInitializer.onStartup(SpringBootServletInitializer.java:84)
at org.springframework.web.SpringServletContainerInitializer.onStartup(SpringServletContainerInitializer.java:175)

Answer 1

Ok, I think I solved my own problem such that Tomcat MemoryLeakDetection no longer complains about the Hystrix ThreadLocal anymore. What I did was I added a custom ServletContextListener and in the "contextDestroyed()" method I found a way to forcibly remove the private static ThreadLocal from the com.netflix.hystrix.Hystrix class. This appears to solve the problem. Here's an excerpt from my custom servlet listener:
/**
* The listener interface for receiving ServletContext events.
* The class that is interested in processing a ServletContext
* event implements this interface, and the object created
* with that class is registered with a component using the
* component's <code>addServletContextListener<code> method. When
* the ServletContext event occurs, that object's appropriate
* method is invoked.
*
* #see ServletContextEvent
*/
#Component
public class GatewayServletContextListener implements ServletContextListener{
private static final Logger LOG = LoggerFactory.getLogger(GatewayServletContextListener.class);
#Override
public void contextInitialized(ServletContextEvent arg0) {
LOG.info("Servlet context listener observed context initialized");
}
#Override
public void contextDestroyed(ServletContextEvent arg0) {
LOG.info("Servlet context listener observed context destroyed");
cleanupThreadLocals();
}
/**
* Cleanup thread locals.
*/
private void cleanupThreadLocals() {
try {
LOG.info("Cleaning up ThreadLocals ...");
Field currentCommandField = ReflectionUtils.findField(Hystrix.class, "currentCommand");
Preconditions.checkNotNull(currentCommandField);
ReflectionUtils.makeAccessible(currentCommandField);
#SuppressWarnings("rawtypes")
ThreadLocal currentCommand = (ThreadLocal)currentCommandField.get(null);
Preconditions.checkNotNull(currentCommand);
currentCommand.remove();
LOG.info("Forcibly removed Hystrix 'currentCommand' ThreadLocal");
LOG.info("Done cleaning up ThreadLocals");
} catch(Exception e) {
LOG.warn(e.getMessage(), e);
}
}
}
And here's the Tomcat log on my spring-boot app redeploy:
2016-05-03/10:11:08.646/PDT [http-apr-8080-exec-3] INFO x.y.z.listeners.GatewayServletContextListener - Servlet context listener observed context destroyed
2016-05-03/10:11:08.648/PDT [http-apr-8080-exec-3] INFO x.y.z.listeners.GatewayServletContextListener - Cleaning up ThreadLocals ...
2016-05-03/10:11:08.652/PDT [http-apr-8080-exec-3] INFO x.y.z.listeners.GatewayServletContextListener - Forcibly removed Hystrix 'currentCommand' ThreadLocal
2016-05-03/10:11:08.654/PDT [http-apr-8080-exec-3] INFO x.y.z.listeners.GatewayServletContextListener - Done cleaning up ThreadLocals