Hi I am using Spring Boot application.
And I am getting a weird exception.
First of all, I read some other posts talking about the way to hit the end point is wrong. But I am NOT even Hitting any end point. I just run the application and leave it.
java.lang.IllegalArgumentException: Invalid character found in method name [0x160x030x010x020x000x010x000x010xfc0x030x030xc4uy0xed0xf3jP0xae0x96|^0x9b0x1e0xe30x010xeb0x8bsa0xab9F-0x940xf00xecy0x190xf0m0xaf0xc4 ]. HTTP method names must be tokens
at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:419) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:269) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
and this happens not right after I run the application.
maybe 5mins after I run the application, it throws this error.
on top of that, when I change my application.properties to application.yml it throws this error right away.
In fact, the application seems to be working even after throwing this error.
I am guessing this is due to the some environment or configuration issue?
Please let me know if anyone has an idea.
Thank you so much in advance.
Related
Suppose it is 2016. I am building a very simple Java EE app with Spring for DI, jdbc template and web, Oracle for persistence and Deploy it to Tomcat. Sounds easy, not sure if it could be more trivial.
There are the following most recent stable versions:
Tomcat 8.5
Oracle jdbc drivers v 12.x
and Spring 4.3.x
Tomcat recommends putting jdbc drivers to $CATALINA_BASE/lib, so I follow this recommendation. Oracle recommends using their UCP pool and tutorials at oracle.com also suggest putting ucp.jar together with ojdbc.jar (to Tomcat's lib folder). I use Spring to manage lifecycle of UCP pool and pass it as a datasource to JdbcTemplate.
I use a single dedicated server at production and for the best experience of my users I use a Tomcat's Parallel deployment feature. There is nothing very special about this feature, it allows to deploy a new version with no downtime and automatically (and gracefully) undeploy an old version when there are no active sessions left for it.
The missing ResultSetMetaData problem
The unexpected problem I may have after deploying a new version of application with such a simple setup:
INFO [http-nio-8080-exec-6] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [java.sql.ResultSetMetaData]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [java.sql.ResultSetMetaData]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1427)
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1415)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1254)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1215)
at com.sun.proxy.$Proxy31.getMetaData(Unknown Source)
at org.springframework.jdbc.core.SingleColumnRowMapper.mapRow(SingleColumnRowMapper.java:89)
at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:93)
at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:60)
at org.springframework.jdbc.core.JdbcTemplate$1QueryStatementCallback.doInStatement(JdbcTemplate.java:465)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:407)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:477)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:487)
at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:497)
at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:503)
at example.App.rsMetadataTest(App.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:204)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:854)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:765)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:655)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1726)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
And now the app is broken. Any subsequent attempt to make a call involving ResultSetMetaData (i.e. jdbcTemplate.queryForObject("select 'hello' from dual", String.class)) will fail with:
java.lang.NoClassDefFoundError: java/sql/ResultSetMetaData
com.sun.proxy.$Proxy31.getMetaData(Unknown Source)
org.springframework.jdbc.core.SingleColumnRowMapper.mapRow(SingleColumnRowMapper.java:89)
org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:93)
org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:60)
org.springframework.jdbc.core.JdbcTemplate$1QueryStatementCallback.doInStatement(JdbcTemplate.java:465)
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:407)
org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:477)
org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:487)
org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:497)
org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:503)
example.App.rsMetadataTest(App.java:82)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:204)
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:854)
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:765)
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
javax.servlet.http.HttpServlet.service(HttpServlet.java:655)
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
How to reproduce
Unfortunately I do not understand the root cause of the exception. The ResultSetMetaData is a JDK class, how it can be not found? Was it unloaded? At least after some experiments I know exactly the minimum steps required to reproduce it:
deploy 1st version of an app and init db pool (i.e. with a simple connection, but which DOES NOT involve ResulstSetMetaData, i.e. jdbcTempalte.query()).
deploy the 2nd version of an app
wait for the 1st version to undeploy (as gracefully as possible)
and make a call which involves ResultSetMetaData.
Boom! The ResultSetMetaData not found again and the app is broken.
This bug does not depend on Tomcat's Parallel deployment feature. You can have the most recent (9.x) Tomcat with stock configuration, 2 different webapps using the same Oracle jdbc driver, deploy it in the order and under the same conditions I described above and get the same error.
Also I would like to add that the following statement from Tomcat is incorrect:
this web application instance has been stopped already
I know exactly that the 2nd (just deployed) app gets invoked (not the unloaded one), it is alive and could not be stopped. But it fails at reaching ResultSetMetaData on it's way.
With the help of docker-compose I did many experiments to isolate the problem and see what can fix it. One thing that fixes the problem is putting ucp.jar to .war, not into Tomcat's lib.
That's the reason for the question in the title:
Should Oracle's ucp.jar reside in Tomcat's lib or be bundled to application's war?
ucp.jar itself is not a jdbc driver which gets registered with a global service-provider. Do you put HikariCP to Tomcat's lib? I do not think so. And bundling ucp to webapp fixes the ResultSetMetaData problem. Are there any other reasons for ucp.jar to be placed to Tomcat's lib?
Broken reflection
Unfortunately moving ucp.jar to war by setting compile or runtime scope for it in Maven can lead to another problem:
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'oracleDataSource' defined in example.App: Initialization of bean failed; nested exception is java.lang.ArrayStoreException: sun.reflect.annotation.AnnotationTypeMismatchExceptionProxy
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:562)
....
... 64 more
Caused by: java.lang.ArrayStoreException: sun.reflect.annotation.AnnotationTypeMismatchExceptionProxy
at sun.reflect.annotation.AnnotationParser.parseEnumArray(AnnotationParser.java:744)
...
at java.lang.Class.getAnnotations(Class.java:3446)
at org.springframework.transaction.annotation.AnnotationTransactionAttributeSource.determineTransactionAttribute(AnnotationTransactionAttributeSource.java:152)
The context won't start as soon as you add #EnableTransactionManagement in your Spring Java config or <tx:annotation-driven/> if you prefer XML. But I do want to use #Transactional annotations in my app. So I am stuck again. Here at least I was able to understand the problem. Spring 4 tries to read annotations on PoolDataSourceImpl to see if the bean needs to proxied to support annotation-based transaction control. The Class#getAnnotations() fails to read annotations on the PoolDataSourceImpl class, because oracle.jdbc.logging.annotations.Feature exists in both jars (ucp and jdbc). And there are 2 class loaders having different instances of Class<oracle.jdbc.logging.annotations.Feature>. The part of introspection capabilities on PoolDataSourceImpl is broken with a weird ArrayStoreExceotion!
The presence of such an error is an argument for keeping both Oracle jars in the same classpath.
If you faced the above problems in 2016 (when there was no higher versions of Oracle driver), what would you do? I am asking this, because the project I work on is a bit stuck in the past. Earlier, upgrading Oracle driver had led to unexpected and unobvious problems in production, so at the nearest release we are hesitant to update the jdbc driver. But since the project was recently upgraded from Tomcat 7 to Tomcat 8, there is now a risk to face the missing ResultSetMetaData problem, which should be solved.
I forgot to say: you might face the stacktrace complaining on missing ResultSetMetaData in a previous version of Tomcat: 7.x. But it did not spoil the observable behaviour. Unlike Tomcat 9.x and 8.x, Tomcat 7.x printed the exception once, but somehow managed to execute the query and successfully handle the request. Tomcat 7.x did not break the app. Does it mean that modern Tomcat has the regression which Tomcat 7.x did not have?
The potential memory leak Tomcat warnings
What I also do not like at redeployment is the following lines at logs:
WARNING [Catalina-utility-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [app##1] appears to have started a thread named [Timer-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:502)
java.util.TimerThread.mainLoop(Timer.java:526)
java.util.TimerThread.run(Timer.java:505)
WARNING [Catalina-utility-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [app##1] appears to have started a thread named [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser.run(BlockSource.java:329)
WARNING [Catalina-utility-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [app##1] appears to have started a thread named [InterruptTimer] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:502)
java.util.TimerThread.mainLoop(Timer.java:526)
java.util.TimerThread.run(Timer.java:505)
Is it possible to fix them at all? From my tests they are not caused by UCP, but rather come from ojdbc.jar. I did not find any solution here. Neither latest version of ojdbc8 (or ojdbc11), nor using other pools or lifecycle methods of Oracle's UniversalConnectionPoolManager (as suggested here) have helped here.
If you replace ojdbc with postgres database and driver, you won't see similar warnings and your logs will be clean.
The source code
I did not provide any code in the post, it is already pretty long, but I created a repo with the minimal application example and parameterised docker-compose test. So you can easily play with it and reproduce all the problems I mentioned with a single command: docker-compose rm -fs && docker-compose up --build
I am aware that you mentioned I use Spring to manage lifecycle of UCP pool and pass it as a datasource to JdbcTemplate but my advice will be to create your datasource as a tomcat resource (i.e., at the context level):
<Resource
name="tomcat/UCPPool"
auth="Container"
<!-- Defines UCP or JDBC factory for connections -->
factory="oracle.ucp.jdbc.PoolDataSourceImpl"
<!-- Defines type of the datasource instance -->
type="oracle.ucp.jdbc.PoolDataSource"
description="UCP Pool in Tomcat"
<!-- Defines the Connection Factory to get the physical connections -->
connectionFactoryClassName="oracle.jdbc.pool.OracleDataSource”
minPoolSize="2"
maxPoolSize="60"
initialPoolSize="15"
autoCommit="false"
user="scott"
password="tiger"
<!-- FCF is auto-enabled in 12.2. Use this property only if you are using Pre 12.2 UCP
fastConnectionFailoverEnabled=”true” -->
<!-- Database URL -->
url="jdbc:oracle:thin:#(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=proddbclust
er-scan)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=proddb)))"
</Resource>
The example is obtained from the guide provided for Oracle when describing Configure Tomcat for UCP.
And try to acquire a reference to that datasource through JNDI:
#Bean
public DataSource dataSource() {
final JndiDataSourceLookup dsLookup = new JndiDataSourceLookup();
dsLookup.setResourceRef(false);
DataSource dataSource = dsLookup.getDataSource("tomcat/UCPPool");
return dataSource;
}
You are very likely facing a class loading issue and putting ucp.jar together with ojdbc.jar in your $CATALINA_BASE/lib and configuring this JNDI lookup can solve the problem.
Regarding your warnings, please, consider read this related SO question, especially this answer: it seems that there is a bug in the Oracle JDBC drive and an update to driver version 12.2 should solve the problem.
P.S.: Great question, very well documented!!
The messages "appears to have started a thread named [...] but has failed to stop it" point directly at the heart of the problem, which is a very common issue when re-deploying webapps within a webapp container, whether tomcat or jetty or any other. The issue is that some long-running threads were started by the app, but not explicitly shutdown, so they keep running, and hence they keep an instance of the WebappClassLoader for this webapp in memory, which references the classes previously loaded by it. When you then redeploy the same webapp, a new distinct WebappClassLoader with the same resources is created, which however doesn't have access to the classes loaded by the prior incarnation that the JVM is still referencing, thus leading to the NoClassDefFoundError.
There are only three general means of dealing with this:
a) Always restart the webapp container when redeploying webapps.
b) Fix all code in the webapps so that all such long-running threads are shut down. This means implementing ServletContextListeners that will perform explicit shutdown operations, stopping pool management threads etc. when the ServletContext is stopped (i.e. the webapp is undeployed).
c) Relocate the offending code so it is not loaded by the WebappClassLoader but by the SystemClassLoader, and thus never goes out of scope. In this case you would achieve that by moving the ojdbc.jar to the system classpath (tomcat/lib) and the datasource definition to the server configuration file (tomcat/conf/server.xml). It is anyway a bad practice to include database drivers within a webapp, such fundamental code should be centrally located so that only one instance of it runs within the JVM. Having these inside webapps can lead to conflicts.
I have a vanilla Spring Boot web application, with some controllers, REST services etc. There is no notion of thread pools, shutdown hooks etc. in the webapp's sources. The app itself does not use any other services, like external databases.
The app worked well until it started to die within two or three days on average after being started. In the logs everything looks normal until this lonely message:
--- [extShutdownHook] o.s.s.concurrent.ThreadPoolTaskExecutor : Shutting down ExecutorService 'taskExecutor'
that has the last timestamp, even if shown in the logs a bit out of chronological order. The message is preceded by this:
java.lang.IllegalArgumentException: Invalid character found in the HTTP protocol [HTTP/1.10x0aHost:]
at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:559) ~[tomcat-embed-core-9.0.43.jar!/:9.0.43]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:261) ~[tomcat-embed-core-9.0.43.jar!/:9.0.43]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.43.jar!/:9.0.43]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:887) ~[tomcat-embed-core-9.0.43.jar!/:9.0.43]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1684) ~[tomcat-embed-core-9.0.43.jar!/:9.0.43]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.43.jar!/:9.0.43]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[na:na]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.43.jar!/:9.0.43]
at java.base/java.lang.Thread.run(Thread.java:832) ~[na:na]
which in turn has no time stamp, so it is difficult to say how these two are related or if they are related at all.
After that, the app no longer responds to requests. Where can I look for causes?
The log fragment you have quoted is not a cause but it is an effect of application beeing gracefully shut down
--- [extShutdownHook]
says this is a hook set on a JVM exit routine. So when, for example, you send a kill singal to the JVM, hooks set via setShutdownHook() will be called - and this is exactly what you see - thread pool is shutdown because JVM is requested to be terminated.
Now, I cannot state why it is terminated, but it is possible that you run out of resoureces (for whatever reasons) and OS (linux?) is killing the most resource consuming process. It is also possible that OS is killing long running processes due to some settings (happenned to me in the past that hoster was killing long runing process)
If you are using scope as provided for tomcat dependency in pom.xml. Try removing that.
It might be the reason of graceful shutdown of tomcat when it finds ideal threads.
--- [extShutdownHook] o.s.s.concurrent.ThreadPoolTaskExecutor : Shutting down ExecutorService 'taskExecutor'
I've developed and deployed a Spring Boot REST application in an Amazon EC2 instance. Then I created a CloudFront distribution with behavior of "Redirect HTTP to HTTPS" and origin configuration as "HTTP Only" to be able to access the Rest API via HTTPS.
Normally, everything works fine while the app is accessed through http with EC2's self domain. But when I started to access through https CloudFront link, the application starts dying silently after a while.
What actions should I take to prevent app being terminated?
Here is what I saw in log files. But this is not the crash log, app keeps running after this log:
2020-09-09 00:09:15.877 INFO 27720 --- [nio-8080-exec-3] o.apache.coyote.http11.Http11Processor : Error parsing HTTP request header
Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in the HTTP protocol [HTTP/1.10x0aHost:]
at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:560) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:260) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.36.jar!/:9.0.36]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat-embed-core-9.0.36.jar!/:9.0.36]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590) [tomcat-embed-core-9.0.36.jar!/:9.0.36]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.36.jar!/:9.0.36]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_252]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_252]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.36.jar!/:9.0.36]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_252]
2020-09-09 11:24:35.544 INFO 27720 --- [nio-8080-exec-5] o.apache.coyote.http11.Http11Processor : Error parsing HTTP request header
Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in the HTTP protocol [HTTP/1.10x0aHost:]
at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:560) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:260) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.36.jar!/:9.0.36]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat-embed-core-9.0.36.jar!/:9.0.36]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590) [tomcat-embed-core-9.0.36.jar!/:9.0.36]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.36.jar!/:9.0.36]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_252]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_252]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.36.jar!/:9.0.36]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_252]
LOGBACK: No context given for c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy#532445947
That's all, no other exception, no shutdown logs, no other clue unfortunately...
I'm using following command to run the application:
sudo nohup java -Dspring.profiles.active=stage -jar myapplication.jar &
Spring Boot version is 2 and EC2 instance is Amazon Linux 2018.3
Edit: I'm adding some images about CloudFront configurations.
Origin Configurations:
Behavior Configurations:
Configuring Supported HTTP Versions of CloudFront distribution
as
HTTP/1.1, HTTP/1.0
instead of
HTTP/2, HTTP/1.1, HTTP/1.0
solved the problem.
The exception in the original post is still present, but it's no longer terminating.
I think the application serving the Rest API doesn't support HTTP 2, and when redirected from CloudFront some HTTP 2 request arrives. Then handling not supported version of a http request somehow causes some fatal errors resulting termination. Please correct me if I'm wrong.
As Steffen mentioned, HTTPS traffic is being forwarded to your application. Your Cloud Front configuration should do SSL termination and then forward the request to your application.
I got a case where the application consumes messages and produces messages as a response to the consumed messages. This is done using kafka transactions, BUT the app also has a scheduled job that sends Kafka messages at regular intervals (also using transactions since it sends to two topics).
When the scheduled job starts sending, I get this exception:
org.apache.kafka.common.KafkaException: TransactionalId aura-transaction-1: Invalid transition attempted from state IN_TRANSACTION to state IN_TRANSACTION
Anyone know what might be the reason?
I'm considering trying with different kafkaTemplates (+ producer factory) to see if that fixes the issue. Since then I can assign a new transaction-id-prefix to the scheduled job. Currently they have the same.
Consumer uses a basic #KafkaListener that is already registered in a transaction from the KafkaMessageListenerContainer. It then produces a message using KafkaTemplate.send(Object).
The scheduled job uses the KafkaTemplate.executeInTransaction functionality and sends to two topics.
Versions:
Spring Boot 2.1.1
Spring Kafka: 2.2.2
StackTrace:
org.apache.kafka.common.KafkaException: TransactionalId person-identhendelse-lager-1.privat-person-fregIdenthendelse-v1.0: Invalid transition attempted from state IN_TRANSACTION to state IN_TRANSACTION
at org.apache.kafka.clients.producer.internals.TransactionManager.transitionTo(TransactionManager.java:758)
at org.apache.kafka.clients.producer.internals.TransactionManager.transitionTo(TransactionManager.java:751)
at org.apache.kafka.clients.producer.internals.TransactionManager.beginTransaction(TransactionManager.java:216)
at org.apache.kafka.clients.producer.KafkaProducer.beginTransaction(KafkaProducer.java:606)
at org.springframework.kafka.core.DefaultKafkaProducerFactory$CloseSafeProducer.beginTransaction(DefaultKafkaProducerFactory.java:459)
at org.springframework.kafka.core.KafkaTemplate.executeInTransaction(KafkaTemplate.java:278)
at no.nav.person.identhendelse.lager.app.aggregat.AggregatIdenthendelsePublisher.sendForPerson(AggregatIdenthendelsePublisher.java:52)
at no.nav.person.identhendelse.lager.app.aggregat.AggregatScheduledTask.aggregate(AggregatScheduledTask.java:54)
at no.nav.person.identhendelse.lager.app.aggregat.AggregatScheduledTask$$FastClassBySpringCGLIB$$7f682c33.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)
at io.micrometer.core.aop.TimedAspect.timedMethod(TimedAspect.java:77)
at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at no.nav.person.utils.precondition.feature.annotation.PreconditionMethodInterceptor.invoke(PreconditionMethodInterceptor.java:22)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
at no.nav.person.identhendelse.lager.app.aggregat.AggregatScheduledTask$$EnhancerBySpringCGLIB$$e0b597f7.aggregate(<generated>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Added example code:
https://github.com/Lg87/kafka-transaction-example
See readme.md and FIND KafkaException to see the exception that occurs.
When asking questions like this, always provide version information.
Show your code and the complete stack trace.
You mentioned transactionTemplate - don't use a template as well as executeInTransaction - they are redundant since they both start a transaction.
We recently fixed a problem where such "nested" transactions were broken.
EDIT
I found the problem; when using producerPerConsumerPartition (default true), producers used by the container should not be added to cache for use by arbitrary KafkaTemplate operations.
As a work-around, use a different DefaultKafkaProducerFactory for the stand-alone template operations.
https://github.com/spring-projects/spring-kafka/issues/908
I am creating a Rest based web service using spring boot,while deploying in tomcat am not getting error.But when i'm trying to hit the RestController's method which is annotated with #RequestMapping and trying to print a string in the browser.
Am getting the error:
type Exception report
message Circular view path [error]: would dispatch back to the current
handler URL [/services-0.0.1-SNAPSHOT/error] again. Check your
ViewResolver setup! (Hint: This may be the result of an unspecified
view, due to default view name generation.)
description The server encountered an internal error that prevented it from fulfilling this request.
exception
javax.servlet.ServletException: Circular view path [error]: would dispatch back to the current handler URL [/services-0.0.1-SNAPSHOT/error] again. Check your ViewResolver setup! (Hint: This may be the result of an unspecified view, due to default view name generation.)
org.springframework.web.servlet.view.InternalResourceView.prepareForRendering(InternalResourceView.java:205)
org.springframework.web.servlet.view.InternalResourceView.renderMergedOutputModel(InternalResourceView.java:145)
org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:303)
org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1228)
org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1011)
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:955)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:877)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:857)
javax.servlet.http.HttpServlet.service(HttpServlet.java:618)
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)
javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextHeaderFilter.doFilterInternal(EndpointWebMvcAutoConfiguration.java:291)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:102)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
org.springframework.boot.actuate.autoconfigure.MetricFilterAutoConfiguration$MetricsFilter.doFilterInternal(MetricFilterAutoConfiguration.java:90)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101)
org.springframework.boot.context.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:101)
org.springframework.boot.context.web.ErrorPageFilter.forwardToErrorPage(ErrorPageFilter.java:173)
org.springframework.boot.context.web.ErrorPageFilter.handleException(ErrorPageFilter.java:156)
org.springframework.boot.context.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:119)
org.springframework.boot.context.web.ErrorPageFilter.access$000(ErrorPageFilter.java:59)
org.springframework.boot.context.web.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:88)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
org.springframework.boot.context.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:101)
I am not able to figure out this issues :(
Any help in this regard will be appreciable.
Check whether your controller class marked with #RestController. If not, and you just annotated it with #Controller, you have to change the #Controller annotation to #RestController. You can have same result by adding #ResponseBody to your controller class or every controller mothods that is used for restful API.