I am working on a spring batch application, which is using a comboPooledDataSource to connect to the database. And the application is running on tomcat9.
When I stop the pod(container) to deploy another version of application, the logs shows that the application has been stoped, but before destroying the container it throws some warnings.
19-Aug-2020 09:58:16.235 WARNING [Thread-13] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [] appears to have started a thread named [Timer-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:552)
java.util.TimerThread.run(Timer.java:505)
19-Aug-2020 09:58:16.235 WARNING [Thread-13] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [] appears to have started a thread named [C3P0PooledConnectionPoolManager[]-DeferredStatementDestroyerThread-#0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)
19-Aug-2020 09:58:16.236 WARNING [Thread-13] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [] appears to have started a thread named [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser.run(BlockSource.java:327)
19-Aug-2020 09:58:16.236 WARNING [Thread-13] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [] appears to have started a thread named [Resource Destroyer in BasicResourcePool.close()] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
java.net.SocketInputStream.read(SocketInputStream.java:171)
java.net.SocketInputStream.read(SocketInputStream.java:141)
oracle.net.ns.Packet.receive(Packet.java:311)
oracle.net.ns.DataPacket.receive(DataPacket.java:105)
oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:305)
oracle.net.ns.NetInputStream.read(NetInputStream.java:249)
oracle.net.ns.NetInputStream.read(NetInputStream.java:171)
oracle.net.ns.NetInputStream.read(NetInputStream.java:89)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:123)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:79)
oracle.jdbc.driver.T4CMAREngineStream.unmarshalUB1(T4CMAREngineStream.java:429)
oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:397)
oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257)
oracle.jdbc.driver.T4C7Ocommoncall.doOLOGOFF(T4C7Ocommoncall.java:59)
oracle.jdbc.driver.T4CConnection.logoff(T4CConnection.java:844)
oracle.jdbc.driver.PhysicalConnection.close(PhysicalConnection.java:2498)
com.mchange.v2.c3p0.impl.NewPooledConnection.close(NewPooledConnection.java:642)
com.mchange.v2.c3p0.impl.NewPooledConnection.closeMaybeCheckedOut(NewPooledConnection.java:255)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.destroyResource(C3P0PooledConnectionPool.java:622)
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask.run(BasicResourcePool.java:1076)
com.mchange.v2.resourcepool.BasicResourcePool.destroyResource(BasicResourcePool.java:1101)
com.mchange.v2.resourcepool.BasicResourcePool.destroyResource(BasicResourcePool.java:1062)
com.mchange.v2.resourcepool.BasicResourcePool.access$100(BasicResourcePool.java:44)
com.mchange.v2.resourcepool.BasicResourcePool$5.run(BasicResourcePool.java:1316)
19-Aug-2020 09:58:16.239 SEVERE [Thread-13] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal#19931e16]) and a value of type [org.apache.logging.log4j.core.LoggerContext] (value [org.apache.logging.log4j.core.LoggerContext#205f573e]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
I Have checked other related questions where it says
**
These are just warnings, tomcat has removed or stopped it forcibly.
**
But I want to make changes in my application. So that, no other warnings are to be shown from application design point of view.
Could anyone have any idea, what to check in the application and how to solve it.
Related
I added the following filter to my spring boot application.
#Component
public class RequestFilter implements Filter {
#Override
public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException {
HttpServletRequest req = (HttpServletRequest) request;
HttpServletResponse res = (HttpServletResponse) response;
String authorizationHeader = res.getHeader("Authorization");
System.out.println("auth :" + authorizationHeader);
chain.doFilter(req, res);
}
}
When I try to start the application locally it fails to start with this error message:
10-May-2022 17:09:43.268 SEVERE [RMI TCP Connection(2)-127.0.0.1] org.apache.catalina.core.StandardContext.startInternal One or more Filters failed to start. Full details will be found in the appropriate container log file
10-May-2022 17:09:43.269 SEVERE [RMI TCP Connection(2)-127.0.0.1] org.apache.catalina.core.StandardContext.startInternal Context [] startup failed due to previous errors
10-May-2022 17:09:43.369 SEVERE [RMI TCP Connection(2)-127.0.0.1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal#8811a2]) and a value of type [org.springframework.security.core.context.SecurityContextImpl] (value [SecurityContextImpl [Null authentication]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
10-May-2022 17:09:43.369 SEVERE [RMI TCP Connection(2)-127.0.0.1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [ROOT] created a ThreadLocal with key of type [com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier$1] (value [com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier$1#429304b7]) and a value of type [com.hazelcast.internal.util.MutableInteger] (value [com.hazelcast.internal.util.MutableInteger#378734f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
[2022-05-10 05:09:43,390] Artifact *****.war exploded: Error during artifact deployment. See server log for details.
10-May-2022 17:09:45.941 INFO [hz.client_1.internal-2] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [com.hazelcast.client.impl.protocol.codec.ClientPingCodec]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [com.hazelcast.client.impl.protocol.codec.ClientPingCodec]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1440)
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1428)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1267)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1228)
at com.hazelcast.client.impl.connection.tcp.HeartbeatManager.checkConnection(HeartbeatManager.java:94)
at com.hazelcast.client.impl.connection.tcp.HeartbeatManager.run(HeartbeatManager.java:78)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset$$$capture(FutureTask.java:308)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
The error suggests to find more details in the container logs.
When check the tomcat error logs on my local machine, there are no errors logged to it.
Any idea what might be causing this error? The problem is there is no relevant information in the tomcat log to debug this.
Suppose it is 2016. I am building a very simple Java EE app with Spring for DI, jdbc template and web, Oracle for persistence and Deploy it to Tomcat. Sounds easy, not sure if it could be more trivial.
There are the following most recent stable versions:
Tomcat 8.5
Oracle jdbc drivers v 12.x
and Spring 4.3.x
Tomcat recommends putting jdbc drivers to $CATALINA_BASE/lib, so I follow this recommendation. Oracle recommends using their UCP pool and tutorials at oracle.com also suggest putting ucp.jar together with ojdbc.jar (to Tomcat's lib folder). I use Spring to manage lifecycle of UCP pool and pass it as a datasource to JdbcTemplate.
I use a single dedicated server at production and for the best experience of my users I use a Tomcat's Parallel deployment feature. There is nothing very special about this feature, it allows to deploy a new version with no downtime and automatically (and gracefully) undeploy an old version when there are no active sessions left for it.
The missing ResultSetMetaData problem
The unexpected problem I may have after deploying a new version of application with such a simple setup:
INFO [http-nio-8080-exec-6] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [java.sql.ResultSetMetaData]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [java.sql.ResultSetMetaData]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1427)
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1415)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1254)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1215)
at com.sun.proxy.$Proxy31.getMetaData(Unknown Source)
at org.springframework.jdbc.core.SingleColumnRowMapper.mapRow(SingleColumnRowMapper.java:89)
at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:93)
at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:60)
at org.springframework.jdbc.core.JdbcTemplate$1QueryStatementCallback.doInStatement(JdbcTemplate.java:465)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:407)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:477)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:487)
at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:497)
at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:503)
at example.App.rsMetadataTest(App.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:204)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:854)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:765)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:655)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1726)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
And now the app is broken. Any subsequent attempt to make a call involving ResultSetMetaData (i.e. jdbcTemplate.queryForObject("select 'hello' from dual", String.class)) will fail with:
java.lang.NoClassDefFoundError: java/sql/ResultSetMetaData
com.sun.proxy.$Proxy31.getMetaData(Unknown Source)
org.springframework.jdbc.core.SingleColumnRowMapper.mapRow(SingleColumnRowMapper.java:89)
org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:93)
org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:60)
org.springframework.jdbc.core.JdbcTemplate$1QueryStatementCallback.doInStatement(JdbcTemplate.java:465)
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:407)
org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:477)
org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:487)
org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:497)
org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:503)
example.App.rsMetadataTest(App.java:82)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:204)
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:854)
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:765)
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
javax.servlet.http.HttpServlet.service(HttpServlet.java:655)
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
How to reproduce
Unfortunately I do not understand the root cause of the exception. The ResultSetMetaData is a JDK class, how it can be not found? Was it unloaded? At least after some experiments I know exactly the minimum steps required to reproduce it:
deploy 1st version of an app and init db pool (i.e. with a simple connection, but which DOES NOT involve ResulstSetMetaData, i.e. jdbcTempalte.query()).
deploy the 2nd version of an app
wait for the 1st version to undeploy (as gracefully as possible)
and make a call which involves ResultSetMetaData.
Boom! The ResultSetMetaData not found again and the app is broken.
This bug does not depend on Tomcat's Parallel deployment feature. You can have the most recent (9.x) Tomcat with stock configuration, 2 different webapps using the same Oracle jdbc driver, deploy it in the order and under the same conditions I described above and get the same error.
Also I would like to add that the following statement from Tomcat is incorrect:
this web application instance has been stopped already
I know exactly that the 2nd (just deployed) app gets invoked (not the unloaded one), it is alive and could not be stopped. But it fails at reaching ResultSetMetaData on it's way.
With the help of docker-compose I did many experiments to isolate the problem and see what can fix it. One thing that fixes the problem is putting ucp.jar to .war, not into Tomcat's lib.
That's the reason for the question in the title:
Should Oracle's ucp.jar reside in Tomcat's lib or be bundled to application's war?
ucp.jar itself is not a jdbc driver which gets registered with a global service-provider. Do you put HikariCP to Tomcat's lib? I do not think so. And bundling ucp to webapp fixes the ResultSetMetaData problem. Are there any other reasons for ucp.jar to be placed to Tomcat's lib?
Broken reflection
Unfortunately moving ucp.jar to war by setting compile or runtime scope for it in Maven can lead to another problem:
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'oracleDataSource' defined in example.App: Initialization of bean failed; nested exception is java.lang.ArrayStoreException: sun.reflect.annotation.AnnotationTypeMismatchExceptionProxy
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:562)
....
... 64 more
Caused by: java.lang.ArrayStoreException: sun.reflect.annotation.AnnotationTypeMismatchExceptionProxy
at sun.reflect.annotation.AnnotationParser.parseEnumArray(AnnotationParser.java:744)
...
at java.lang.Class.getAnnotations(Class.java:3446)
at org.springframework.transaction.annotation.AnnotationTransactionAttributeSource.determineTransactionAttribute(AnnotationTransactionAttributeSource.java:152)
The context won't start as soon as you add #EnableTransactionManagement in your Spring Java config or <tx:annotation-driven/> if you prefer XML. But I do want to use #Transactional annotations in my app. So I am stuck again. Here at least I was able to understand the problem. Spring 4 tries to read annotations on PoolDataSourceImpl to see if the bean needs to proxied to support annotation-based transaction control. The Class#getAnnotations() fails to read annotations on the PoolDataSourceImpl class, because oracle.jdbc.logging.annotations.Feature exists in both jars (ucp and jdbc). And there are 2 class loaders having different instances of Class<oracle.jdbc.logging.annotations.Feature>. The part of introspection capabilities on PoolDataSourceImpl is broken with a weird ArrayStoreExceotion!
The presence of such an error is an argument for keeping both Oracle jars in the same classpath.
If you faced the above problems in 2016 (when there was no higher versions of Oracle driver), what would you do? I am asking this, because the project I work on is a bit stuck in the past. Earlier, upgrading Oracle driver had led to unexpected and unobvious problems in production, so at the nearest release we are hesitant to update the jdbc driver. But since the project was recently upgraded from Tomcat 7 to Tomcat 8, there is now a risk to face the missing ResultSetMetaData problem, which should be solved.
I forgot to say: you might face the stacktrace complaining on missing ResultSetMetaData in a previous version of Tomcat: 7.x. But it did not spoil the observable behaviour. Unlike Tomcat 9.x and 8.x, Tomcat 7.x printed the exception once, but somehow managed to execute the query and successfully handle the request. Tomcat 7.x did not break the app. Does it mean that modern Tomcat has the regression which Tomcat 7.x did not have?
The potential memory leak Tomcat warnings
What I also do not like at redeployment is the following lines at logs:
WARNING [Catalina-utility-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [app##1] appears to have started a thread named [Timer-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:502)
java.util.TimerThread.mainLoop(Timer.java:526)
java.util.TimerThread.run(Timer.java:505)
WARNING [Catalina-utility-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [app##1] appears to have started a thread named [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser.run(BlockSource.java:329)
WARNING [Catalina-utility-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [app##1] appears to have started a thread named [InterruptTimer] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:502)
java.util.TimerThread.mainLoop(Timer.java:526)
java.util.TimerThread.run(Timer.java:505)
Is it possible to fix them at all? From my tests they are not caused by UCP, but rather come from ojdbc.jar. I did not find any solution here. Neither latest version of ojdbc8 (or ojdbc11), nor using other pools or lifecycle methods of Oracle's UniversalConnectionPoolManager (as suggested here) have helped here.
If you replace ojdbc with postgres database and driver, you won't see similar warnings and your logs will be clean.
The source code
I did not provide any code in the post, it is already pretty long, but I created a repo with the minimal application example and parameterised docker-compose test. So you can easily play with it and reproduce all the problems I mentioned with a single command: docker-compose rm -fs && docker-compose up --build
I am aware that you mentioned I use Spring to manage lifecycle of UCP pool and pass it as a datasource to JdbcTemplate but my advice will be to create your datasource as a tomcat resource (i.e., at the context level):
<Resource
name="tomcat/UCPPool"
auth="Container"
<!-- Defines UCP or JDBC factory for connections -->
factory="oracle.ucp.jdbc.PoolDataSourceImpl"
<!-- Defines type of the datasource instance -->
type="oracle.ucp.jdbc.PoolDataSource"
description="UCP Pool in Tomcat"
<!-- Defines the Connection Factory to get the physical connections -->
connectionFactoryClassName="oracle.jdbc.pool.OracleDataSource”
minPoolSize="2"
maxPoolSize="60"
initialPoolSize="15"
autoCommit="false"
user="scott"
password="tiger"
<!-- FCF is auto-enabled in 12.2. Use this property only if you are using Pre 12.2 UCP
fastConnectionFailoverEnabled=”true” -->
<!-- Database URL -->
url="jdbc:oracle:thin:#(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=proddbclust
er-scan)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=proddb)))"
</Resource>
The example is obtained from the guide provided for Oracle when describing Configure Tomcat for UCP.
And try to acquire a reference to that datasource through JNDI:
#Bean
public DataSource dataSource() {
final JndiDataSourceLookup dsLookup = new JndiDataSourceLookup();
dsLookup.setResourceRef(false);
DataSource dataSource = dsLookup.getDataSource("tomcat/UCPPool");
return dataSource;
}
You are very likely facing a class loading issue and putting ucp.jar together with ojdbc.jar in your $CATALINA_BASE/lib and configuring this JNDI lookup can solve the problem.
Regarding your warnings, please, consider read this related SO question, especially this answer: it seems that there is a bug in the Oracle JDBC drive and an update to driver version 12.2 should solve the problem.
P.S.: Great question, very well documented!!
The messages "appears to have started a thread named [...] but has failed to stop it" point directly at the heart of the problem, which is a very common issue when re-deploying webapps within a webapp container, whether tomcat or jetty or any other. The issue is that some long-running threads were started by the app, but not explicitly shutdown, so they keep running, and hence they keep an instance of the WebappClassLoader for this webapp in memory, which references the classes previously loaded by it. When you then redeploy the same webapp, a new distinct WebappClassLoader with the same resources is created, which however doesn't have access to the classes loaded by the prior incarnation that the JVM is still referencing, thus leading to the NoClassDefFoundError.
There are only three general means of dealing with this:
a) Always restart the webapp container when redeploying webapps.
b) Fix all code in the webapps so that all such long-running threads are shut down. This means implementing ServletContextListeners that will perform explicit shutdown operations, stopping pool management threads etc. when the ServletContext is stopped (i.e. the webapp is undeployed).
c) Relocate the offending code so it is not loaded by the WebappClassLoader but by the SystemClassLoader, and thus never goes out of scope. In this case you would achieve that by moving the ojdbc.jar to the system classpath (tomcat/lib) and the datasource definition to the server configuration file (tomcat/conf/server.xml). It is anyway a bad practice to include database drivers within a webapp, such fundamental code should be centrally located so that only one instance of it runs within the JVM. Having these inside webapps can lead to conflicts.
I have a Spring Boot app with the Spring Cloud binder for Kinesis. Everything works fine, but when I stop the app I get the following exception:
2020-11-23 18:40:36.292 INFO --- [extShutdownHook] a.i.k.KinesisMessageDrivenChannelAdapter : stopped KinesisMessageDrivenChannelAdapter{shardOffsets=[KinesisShardOffset{iteratorType=TRIM_HORIZON, sequenceNumber='null', timestamp=null, stream='product-master-data-updated-event-stream', shard='shardId-000000000000', reset=false}], consumerGroup='marketing-campaign'}
2020-11-23 18:40:36.299 ERROR --- [s-shard-locks-1] a.i.k.KinesisMessageDrivenChannelAdapter : ShardConsumerManager Thread [org.springframework.integration.aws.inbound.kinesis.KinesisMessageDrivenChannelAdapter$ShardConsumerManager#71bd846d] has been interrupted
java.lang.InterruptedException: sleep interrupted
at java.base/java.lang.Thread.sleep(Native Method)
at org.springframework.integration.aws.inbound.kinesis.KinesisMessageDrivenChannelAdapter.sleep(KinesisMessageDrivenChannelAdapter.java:668)
at org.springframework.integration.aws.inbound.kinesis.KinesisMessageDrivenChannelAdapter.access$1400(KinesisMessageDrivenChannelAdapter.java:100)
at org.springframework.integration.aws.inbound.kinesis.KinesisMessageDrivenChannelAdapter$ShardConsumerManager.run(KinesisMessageDrivenChannelAdapter.java:1431)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
I'm using
implementation 'org.springframework.cloud:spring-cloud-stream-binder-kinesis:2.0.3.RELEASE'
springBoot: "2.2.7.RELEASE",
How can I stop the app gracefully? An exception on every deployment doesn't look nice.
The exception is just logged under ERROR level. You can move it for now to the FATAL logging level for that a.i.k.KinesisMessageDrivenChannelAdapter category.
We just need to fix this KinesisMessageDrivenChannelAdapter.ShardConsumerManager to avoid exception logging and rethrowing when the channel adapter is already in a stopped state.
Feel free to raise a GH issue and we will see what and how we can fix over there.
The thread interruption in that logic is correct behavior and should not effect anything in your application.
UPDATE
Related GH issue: https://github.com/spring-projects/spring-integration-aws/issues/187
We have an application that uses JMS with Tibco messaging server. It’s implemented with Spring Boot.
Our sessions are transacted sessions and the acknowledge mode is set to auto.
We can receive the messages sent to the queue but for some reason that commit() method fails.
This is the error:
2018-07-13 15:50:35.858 WARN 2576 --- [enerContainer-8] o.s.j.l.DefaultMessageListenerContainer : Execution of JMS message listener failed, and no ErrorHandler has been set.
javax.jms.TransactionRolledBackException: Commit failed
2018-07-13 15:50:35.914 WARN 2576 --- [enerContainer-8] o.s.j.l.DefaultMessageListenerContainer : Setup of JMS message listener invoker failed for destination 'queue' - trying to recover. Cause: Commit failed; nested exception is javax.jms.JMSException: operation='ack' cause='Illegal state' queue='queue'
I should mention two things:
I see the commit() failure and stack trace, only when I run in debug mode. If I let the war to just run, I don’t see any trace of commit failure in the logs. I can tell that we received the messages, but can’t tell whether the commit() failed or not.
If we use the default JMS implementations of Spring Boot which is ActiveMQ, then the commit() does not fail whether its in debug mode or not, it only happens with Tibco.
Does anybody have any idea that why the commit() fails? When the commit fails, the transaction is rolled back and the messages won’t be cleared from the queue.
Why I see this error only in debug mode? OR: it indeed, does happen when I run the war but just not logged?
Below please find the complete stack trace:
15:50:35.858 [DefaultMessageListenerContainer-8] WARN o.s.j.l.DefaultMessageListenerContainer - Execution of JMS message listener failed, and no ErrorHandler has been set.
javax.jms.TransactionRolledBackException: Commit failed
at com.tibco.tibjms.Tibjmsx.buildException(Tibjmsx.java:596)
at com.tibco.tibjms.TibjmsxSessionImp._confirmTransacted(TibjmsxSessionImp.java:3251)
at com.tibco.tibjms.TibjmsxSessionImp._confirm(TibjmsxSessionImp.java:3643)
at com.tibco.tibjms.TibjmsxSessionImp._commit(TibjmsxSessionImp.java:2898)
at com.tibco.tibjms.TibjmsxSessionImp.commit(TibjmsxSessionImp.java:4860)
at org.springframework.jms.support.JmsUtils.commitIfNecessary(JmsUtils.java:218)
at org.springframework.jms.listener.AbstractMessageListenerContainer.commitIfNecessary(AbstractMessageListenerContainer.java:776)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:680)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:318)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:257)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1189)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1179)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1076)
at java.lang.Thread.run(Unknown Source)
2018-07-13 15:50:35.914 WARN 2576 --- [enerContainer-8] o.s.j.l.DefaultMessageListenerContainer : Setup of JMS message listener invoker failed for destination 'queue' - trying to recover. Cause: Commit failed; nested exception is javax.jms.JMSException: operation='ack' cause='Illegal state' queue='queue'
2018-07-13 15:50:36.543 INFO 2576 --- [enerContainer-8] o.s.j.l.DefaultMessageListenerContainer : Successfully refreshed JMS Connection
15:50:36.543 [DefaultMessageListenerContainer-8] INFO o.s.j.l.DefaultMessageListenerContainer - Successfully refreshed JMS Connection
This EMS exception happens when some of your messages within that transaction expired in the meantime. Just tried this one myself.
When you run outside of debugging mode, things seem to be quick enough. But when you are in debugging mode, your time before you commit() is much longer. So if your producer's time-to-live expires in the meantime, the commit assumes that something is wrong (EMS knows that there should be e.g. 5 messages in this transaction, but one of them already expired) and throws this exception.
To verify your producer's time-to-live, see the last option of your MessageProducer.send() call or msgProducer.setTimeToLive() and note, this is a time in millseconds, not seconds. Another way to check if this is the issue, is to run in debug mode and be 'really quick' with continuing your thread. That might change your behaviour.
We have a spring-boot based web application and using these components below
jmsTemplate for ActiveMQ connection
Spring quartz scheduler for scheduling task
spring-boot-starter-data-mongodb and mongo-java-driver as persistence
Logback as logging
When we stoped the web application by tomcat manager, there will have warning or severe log as below
13-Jul-2016 08:54:44.688 INFO [http-nio-80-exec-19] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
13-Jul-2016 08:55:08.376 WARNING [http-nio-80-exec-18] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [demo] appears to have started a thread named [AsyncLogger-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:45)
com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:55)
com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:123)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
13-Jul-2016 08:55:08.380 WARNING [http-nio-80-exec-18] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [demo] appears to have started a thread named [schedulerFactoryBean_Worker-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
13-Jul-2016 08:55:08.383 WARNING [http-nio-80-exec-18] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [demo] appears to have started a thread named [schedulerFactoryBean_Worker-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
13-Jul-2016 08:55:08.387 WARNING [http-nio-80-exec-18] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [demo] appears to have started a thread named [schedulerFactoryBean_Worker-3] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
13-Jul-2016 08:55:08.389 WARNING [http-nio-80-exec-18] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [demo] appears to have started a thread named [schedulerFactoryBean_Worker-4] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
13-Jul-2016 08:55:08.390 WARNING [http-nio-80-exec-18] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [demo] appears to have started a thread named [schedulerFactoryBean_Worker-5] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
13-Jul-2016 08:55:08.391 WARNING [http-nio-80-exec-18] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [demo] appears to have started a thread named [schedulerFactoryBean_Worker-6] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
13-Jul-2016 08:55:08.393 WARNING [http-nio-80-exec-18] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [demo] appears to have started a thread named [schedulerFactoryBean_Worker-7] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
13-Jul-2016 08:55:08.394 WARNING [http-nio-80-exec-18] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [demo] appears to have started a thread named [schedulerFactoryBean_Worker-8] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
13-Jul-2016 08:55:08.395 WARNING [http-nio-80-exec-18] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [demo] appears to have started a thread named [schedulerFactoryBean_Worker-9] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
13-Jul-2016 08:55:08.397 WARNING [http-nio-80-exec-18] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [demo] appears to have started a thread named [schedulerFactoryBean_Worker-10] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
13-Jul-2016 08:55:08.398 WARNING [http-nio-80-exec-18] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [demo] appears to have started a thread named [ActiveMQ Session: ID:WIN-U2S1V56UQ42-60029-1468400094066-1:1:1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:502)
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:119)
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:42)
13-Jul-2016 08:55:08.400 SEVERE [http-nio-80-exec-18] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [demo] created a ThreadLocal with key of type [org.apache.axiom.util.UIDGenerator$1] (value [org.apache.axiom.util.UIDGenerator$1#53197bfc]) and a value of type [long[]] (value [[J#3ab7d938]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
13-Jul-2016 08:55:08.402 SEVERE [http-nio-80-exec-18] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [demo] created a ThreadLocal with key of type [org.apache.logging.log4j.core.async.AsyncLogger$Info$1] (value [org.apache.logging.log4j.core.async.AsyncLogger$Info$1#6ddb14cb]) and a value of type [org.apache.logging.log4j.core.async.AsyncLogger.Info] (value [org.apache.logging.log4j.core.async.AsyncLogger$Info#4f6422bc]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
It looks like the thread of logger, quartz scheduler and activeMQ connection have failed to shutdown properly. There will sometimes have something like mongoDB driver connection failed to closed(Sorry for that we have not capture the log yet).
We can not just shutdown tomcat but only closed the specified applicatoin is allowed. So that is important to make sure the application can be closed gracefully.
How can we do to let whole web application to be closed properly? Could we set something(like hook?) for container to handle the shutdown operation?
Or shall we handle these operations manually on ContextClosedEvent? and How to do so?