I am trying to configure spring data with cassandra.
But I am getting bellow error , when my app is deploying in tomcat.
When I check the connection, it is available to the given port. (127.0.0.1:9042). I have include stack trace and spring configuration bellow.
Does anyone having idea on this error?
Full stack trace :
2015-12-06 17:46:25 ERROR web.context.ContextLoader:331 - Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cassandraSession': Invocation of init method failed; nested exception is com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /127.0.0.1:9042 (com.datastax.driver.core.exceptions.InvalidQueryException: unconfigured table schema_keyspaces))
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1572)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:736)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:759)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:434)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:106)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4994)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5492)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:649)
at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1245)
at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1895)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /127.0.0.1:9042 (com.datastax.driver.core.exceptions.InvalidQueryException: unconfigured table schema_keyspaces))
at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:223)
at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:78)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1230)
at com.datastax.driver.core.Cluster.init(Cluster.java:157)
at com.datastax.driver.core.Cluster.connect(Cluster.java:245)
at com.datastax.driver.core.Cluster.connect(Cluster.java:278)
at org.springframework.cassandra.config.CassandraCqlSessionFactoryBean.afterPropertiesSet(CassandraCqlSessionFactoryBean.java:82)
at org.springframework.data.cassandra.config.CassandraSessionFactoryBean.afterPropertiesSet(CassandraSessionFactoryBean.java:43)
===================================================================
Spring Configuration :
<?xml version="1.0" encoding="UTF-8"?>
<beans:beans ...>
<cassandra:cluster id="cassandraCluster"
contact-points="127.0.0.1" port="9042" />
<cassandra:converter />
<cassandra:session id="cassandraSession" cluster-ref="cassandraCluster"
keyspace-name="blood" />
<cassandra:template id="cqlTemplate" />
<cassandra:repositories base-package="com.blood.dao.nosql" />
<cassandra:mapping entity-base-packages="com.blood.domain.nosql" />
</beans:beans>
The problem is that Spring Data Cassandra (as of December 2015 when I write this) does not provide support for Cassandra 3.x. Here's an excerpt from a conversation with one of the developers in the #spring channel on freenode:
[13:49] <_amicable> Hi all, does anybody know if spring data cassandra supports cassandra 3.x? All dependencies & datastax drivers seem to be 2.x
[13:49] <#_ollie> amicable: Not in the near future.
[13:49] <_amicable> _ollie: thanks.
[13:50] <_amicable> I'll go and look at the relative merits of 2.x vs 3.x then
;)
[13:51] <#_ollie> SD Cassandra is a community project (so far) and its progress highly depends on how much time the developers can actually spend on it.
[13:51] <#_ollie> We will have someone joining the team in February 2016 to get the project more closely aligned to the core Spring Data projects.
It looks like you are using an older version of the driver with Cassandra 3.0. Cassandra 3.0 changed its internal schema metadata representation, and only the latest drivers can parse this metadata.
Use Java Cassandra driver 3.0.0-alpha5 to connect to Cassandra 3.0.
Related
Context
By default our SpringBoot application uses HikariCP for its connection pooling.
We are currently investigating how to increase the performance of our application, and want to check out how our load tests behave when using PgPool instead.
It is our understanding that HikariCP and PgPool will not work well with each other and thus that we should be able to disable HikariCP in order to test the performance when using PgPool appropriately.
Question
How does one ensure that no connection pooling ("CP") is used at all at the application level?
Precision
The ultimate goal being to plug PgPool in between our application and our database.
It's my (potentially flawed) understanding that HikariCP should be disabled for that, and that each getConnection() call should return a new connection fetch from the JDBC URL configured for our application. And that that JDBC URL should eventually point to PgPool instead of the DB, which will take care of returning pooled connections.
Attempts
First, we simply removed Hikari from the dependencies:
configurations.all { exclude group: 'com.zaxxer' }
But we'd get this error:
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.]
13:04:34.0242 - ERROR TenantId[] UserId[] TraceId[] Thread[main]
Logger[org.springframework.boot.SpringApplication]
Msg[Application run failed]
org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:163)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:577)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:434)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:338)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1343)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1332)
at com.sap.s4hana.eureka.business.centralinventory.application.Application.main(Application.java:17)
Caused by: org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat
at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:142)
at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.<init>(TomcatWebServer.java:104)
at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getTomcatWebServer(TomcatServletWebServerFactory.java:450)
at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getWebServer(TomcatServletWebServerFactory.java:199)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:182)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:160)
Caused by: org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat
... 8 common frames omitted
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'webMvcMetricsFilter' defined in class path resource [org/springframework/boot/actuate/autoconfigure/metrics/web/servlet/WebMvcMetricsAutoConfiguration.class]: Unsatisfied dependency expressed through method 'webMvcMetricsFilter' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'prometheusMeterRegistry' defined in class path resource [org/springframework/boot/actuate/autoconfigure/metrics/export/prometheus/PrometheusMetricsExportAutoConfiguration.class]: Initialization of bean failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'healthRegistryCustomizer' defined in class path resource
...(I interrupt the stacktrace there due to company policy)
nested exception is java.lang.IllegalStateException: No supported DataSource type found
I eventually got around to adding the following to our dependencies:
implementation 'org.apache.tomcat:tomcat-jdbc:10.0.14'
But then I'm left wondering:
Does Tomcat enable CP by default? (Which would thus mean I have not effectively removed application-level CP)
Is there not a way to simply add a configuration in application.properties to keep the servlet container but avoid CP? (So that I don't need to remove Hikari from the dependencies.)
I also tried adding (inspired by this SO answer):
spring.datasource.type=org.springframework.jdbc.datasource.DriverManagerDataSource
spring.datasource.driver-class-name=org.postgresql.Driver
And also tried replacing that DriverManagerDataSource with SimpleDriverDataSource, but to no apparent avail since I cannot seem to confirm that indeed no CP is used.
Suppose it is 2016. I am building a very simple Java EE app with Spring for DI, jdbc template and web, Oracle for persistence and Deploy it to Tomcat. Sounds easy, not sure if it could be more trivial.
There are the following most recent stable versions:
Tomcat 8.5
Oracle jdbc drivers v 12.x
and Spring 4.3.x
Tomcat recommends putting jdbc drivers to $CATALINA_BASE/lib, so I follow this recommendation. Oracle recommends using their UCP pool and tutorials at oracle.com also suggest putting ucp.jar together with ojdbc.jar (to Tomcat's lib folder). I use Spring to manage lifecycle of UCP pool and pass it as a datasource to JdbcTemplate.
I use a single dedicated server at production and for the best experience of my users I use a Tomcat's Parallel deployment feature. There is nothing very special about this feature, it allows to deploy a new version with no downtime and automatically (and gracefully) undeploy an old version when there are no active sessions left for it.
The missing ResultSetMetaData problem
The unexpected problem I may have after deploying a new version of application with such a simple setup:
INFO [http-nio-8080-exec-6] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [java.sql.ResultSetMetaData]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [java.sql.ResultSetMetaData]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1427)
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1415)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1254)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1215)
at com.sun.proxy.$Proxy31.getMetaData(Unknown Source)
at org.springframework.jdbc.core.SingleColumnRowMapper.mapRow(SingleColumnRowMapper.java:89)
at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:93)
at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:60)
at org.springframework.jdbc.core.JdbcTemplate$1QueryStatementCallback.doInStatement(JdbcTemplate.java:465)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:407)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:477)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:487)
at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:497)
at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:503)
at example.App.rsMetadataTest(App.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:204)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:854)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:765)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:655)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1726)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
And now the app is broken. Any subsequent attempt to make a call involving ResultSetMetaData (i.e. jdbcTemplate.queryForObject("select 'hello' from dual", String.class)) will fail with:
java.lang.NoClassDefFoundError: java/sql/ResultSetMetaData
com.sun.proxy.$Proxy31.getMetaData(Unknown Source)
org.springframework.jdbc.core.SingleColumnRowMapper.mapRow(SingleColumnRowMapper.java:89)
org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:93)
org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:60)
org.springframework.jdbc.core.JdbcTemplate$1QueryStatementCallback.doInStatement(JdbcTemplate.java:465)
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:407)
org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:477)
org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:487)
org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:497)
org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:503)
example.App.rsMetadataTest(App.java:82)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:204)
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:854)
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:765)
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
javax.servlet.http.HttpServlet.service(HttpServlet.java:655)
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
How to reproduce
Unfortunately I do not understand the root cause of the exception. The ResultSetMetaData is a JDK class, how it can be not found? Was it unloaded? At least after some experiments I know exactly the minimum steps required to reproduce it:
deploy 1st version of an app and init db pool (i.e. with a simple connection, but which DOES NOT involve ResulstSetMetaData, i.e. jdbcTempalte.query()).
deploy the 2nd version of an app
wait for the 1st version to undeploy (as gracefully as possible)
and make a call which involves ResultSetMetaData.
Boom! The ResultSetMetaData not found again and the app is broken.
This bug does not depend on Tomcat's Parallel deployment feature. You can have the most recent (9.x) Tomcat with stock configuration, 2 different webapps using the same Oracle jdbc driver, deploy it in the order and under the same conditions I described above and get the same error.
Also I would like to add that the following statement from Tomcat is incorrect:
this web application instance has been stopped already
I know exactly that the 2nd (just deployed) app gets invoked (not the unloaded one), it is alive and could not be stopped. But it fails at reaching ResultSetMetaData on it's way.
With the help of docker-compose I did many experiments to isolate the problem and see what can fix it. One thing that fixes the problem is putting ucp.jar to .war, not into Tomcat's lib.
That's the reason for the question in the title:
Should Oracle's ucp.jar reside in Tomcat's lib or be bundled to application's war?
ucp.jar itself is not a jdbc driver which gets registered with a global service-provider. Do you put HikariCP to Tomcat's lib? I do not think so. And bundling ucp to webapp fixes the ResultSetMetaData problem. Are there any other reasons for ucp.jar to be placed to Tomcat's lib?
Broken reflection
Unfortunately moving ucp.jar to war by setting compile or runtime scope for it in Maven can lead to another problem:
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'oracleDataSource' defined in example.App: Initialization of bean failed; nested exception is java.lang.ArrayStoreException: sun.reflect.annotation.AnnotationTypeMismatchExceptionProxy
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:562)
....
... 64 more
Caused by: java.lang.ArrayStoreException: sun.reflect.annotation.AnnotationTypeMismatchExceptionProxy
at sun.reflect.annotation.AnnotationParser.parseEnumArray(AnnotationParser.java:744)
...
at java.lang.Class.getAnnotations(Class.java:3446)
at org.springframework.transaction.annotation.AnnotationTransactionAttributeSource.determineTransactionAttribute(AnnotationTransactionAttributeSource.java:152)
The context won't start as soon as you add #EnableTransactionManagement in your Spring Java config or <tx:annotation-driven/> if you prefer XML. But I do want to use #Transactional annotations in my app. So I am stuck again. Here at least I was able to understand the problem. Spring 4 tries to read annotations on PoolDataSourceImpl to see if the bean needs to proxied to support annotation-based transaction control. The Class#getAnnotations() fails to read annotations on the PoolDataSourceImpl class, because oracle.jdbc.logging.annotations.Feature exists in both jars (ucp and jdbc). And there are 2 class loaders having different instances of Class<oracle.jdbc.logging.annotations.Feature>. The part of introspection capabilities on PoolDataSourceImpl is broken with a weird ArrayStoreExceotion!
The presence of such an error is an argument for keeping both Oracle jars in the same classpath.
If you faced the above problems in 2016 (when there was no higher versions of Oracle driver), what would you do? I am asking this, because the project I work on is a bit stuck in the past. Earlier, upgrading Oracle driver had led to unexpected and unobvious problems in production, so at the nearest release we are hesitant to update the jdbc driver. But since the project was recently upgraded from Tomcat 7 to Tomcat 8, there is now a risk to face the missing ResultSetMetaData problem, which should be solved.
I forgot to say: you might face the stacktrace complaining on missing ResultSetMetaData in a previous version of Tomcat: 7.x. But it did not spoil the observable behaviour. Unlike Tomcat 9.x and 8.x, Tomcat 7.x printed the exception once, but somehow managed to execute the query and successfully handle the request. Tomcat 7.x did not break the app. Does it mean that modern Tomcat has the regression which Tomcat 7.x did not have?
The potential memory leak Tomcat warnings
What I also do not like at redeployment is the following lines at logs:
WARNING [Catalina-utility-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [app##1] appears to have started a thread named [Timer-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:502)
java.util.TimerThread.mainLoop(Timer.java:526)
java.util.TimerThread.run(Timer.java:505)
WARNING [Catalina-utility-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [app##1] appears to have started a thread named [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser.run(BlockSource.java:329)
WARNING [Catalina-utility-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [app##1] appears to have started a thread named [InterruptTimer] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:502)
java.util.TimerThread.mainLoop(Timer.java:526)
java.util.TimerThread.run(Timer.java:505)
Is it possible to fix them at all? From my tests they are not caused by UCP, but rather come from ojdbc.jar. I did not find any solution here. Neither latest version of ojdbc8 (or ojdbc11), nor using other pools or lifecycle methods of Oracle's UniversalConnectionPoolManager (as suggested here) have helped here.
If you replace ojdbc with postgres database and driver, you won't see similar warnings and your logs will be clean.
The source code
I did not provide any code in the post, it is already pretty long, but I created a repo with the minimal application example and parameterised docker-compose test. So you can easily play with it and reproduce all the problems I mentioned with a single command: docker-compose rm -fs && docker-compose up --build
I am aware that you mentioned I use Spring to manage lifecycle of UCP pool and pass it as a datasource to JdbcTemplate but my advice will be to create your datasource as a tomcat resource (i.e., at the context level):
<Resource
name="tomcat/UCPPool"
auth="Container"
<!-- Defines UCP or JDBC factory for connections -->
factory="oracle.ucp.jdbc.PoolDataSourceImpl"
<!-- Defines type of the datasource instance -->
type="oracle.ucp.jdbc.PoolDataSource"
description="UCP Pool in Tomcat"
<!-- Defines the Connection Factory to get the physical connections -->
connectionFactoryClassName="oracle.jdbc.pool.OracleDataSource”
minPoolSize="2"
maxPoolSize="60"
initialPoolSize="15"
autoCommit="false"
user="scott"
password="tiger"
<!-- FCF is auto-enabled in 12.2. Use this property only if you are using Pre 12.2 UCP
fastConnectionFailoverEnabled=”true” -->
<!-- Database URL -->
url="jdbc:oracle:thin:#(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=proddbclust
er-scan)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=proddb)))"
</Resource>
The example is obtained from the guide provided for Oracle when describing Configure Tomcat for UCP.
And try to acquire a reference to that datasource through JNDI:
#Bean
public DataSource dataSource() {
final JndiDataSourceLookup dsLookup = new JndiDataSourceLookup();
dsLookup.setResourceRef(false);
DataSource dataSource = dsLookup.getDataSource("tomcat/UCPPool");
return dataSource;
}
You are very likely facing a class loading issue and putting ucp.jar together with ojdbc.jar in your $CATALINA_BASE/lib and configuring this JNDI lookup can solve the problem.
Regarding your warnings, please, consider read this related SO question, especially this answer: it seems that there is a bug in the Oracle JDBC drive and an update to driver version 12.2 should solve the problem.
P.S.: Great question, very well documented!!
The messages "appears to have started a thread named [...] but has failed to stop it" point directly at the heart of the problem, which is a very common issue when re-deploying webapps within a webapp container, whether tomcat or jetty or any other. The issue is that some long-running threads were started by the app, but not explicitly shutdown, so they keep running, and hence they keep an instance of the WebappClassLoader for this webapp in memory, which references the classes previously loaded by it. When you then redeploy the same webapp, a new distinct WebappClassLoader with the same resources is created, which however doesn't have access to the classes loaded by the prior incarnation that the JVM is still referencing, thus leading to the NoClassDefFoundError.
There are only three general means of dealing with this:
a) Always restart the webapp container when redeploying webapps.
b) Fix all code in the webapps so that all such long-running threads are shut down. This means implementing ServletContextListeners that will perform explicit shutdown operations, stopping pool management threads etc. when the ServletContext is stopped (i.e. the webapp is undeployed).
c) Relocate the offending code so it is not loaded by the WebappClassLoader but by the SystemClassLoader, and thus never goes out of scope. In this case you would achieve that by moving the ojdbc.jar to the system classpath (tomcat/lib) and the datasource definition to the server configuration file (tomcat/conf/server.xml). It is anyway a bad practice to include database drivers within a webapp, such fundamental code should be centrally located so that only one instance of it runs within the JVM. Having these inside webapps can lead to conflicts.
Yesterday we had a power outage and were able to get all of our machines back online with the exception of one box.
When firing up our application we see the log
Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.levelsbeyond.search.elasticsearch.ElasticSearchTransportClientProvider]: Constructor threw exception; nested exception is org.elasticsearch.client.transport.NoNodeAvailableException: No node available (org.mule.api.lifecycle.InitialisationException)
at org.mule.config.builders.AbstractConfigurationBuilder.configure(AbstractConfigurationBuilder.java:52)
at org.mule.config.builders.AbstractResourceConfigurationBuilder.configure(AbstractResourceConfigurationBuilder.java:78)
at org.mule.context.DefaultMuleContextFactory.createMuleContext(DefaultMuleContextFactory.java:97)
at org.mule.config.builders.MuleXmlBuilderContextListener.createMuleContext(MuleXmlBuilderContextListener.java:169)
at org.mule.config.builders.MuleXmlBuilderContextListener.initialize(MuleXmlBuilderContextListener.java:98)
at org.mule.config.builders.MuleXmlBuilderContextListener.contextInitialized(MuleXmlBuilderContextListener.java:74)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4939)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5434)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:633)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:983)
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1660)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
and then consistently repeated we see
2014-09-19 11:35:19,200 WARN [org.elasticsearch.transport.netty] (elasticsearch[Dominic Fortune][transport_client_worker][T#5]{New I/O worker #5}) - <[Dominic Fortune] Message not fully read (response) for [12] handler future(org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler$1#625badaa), error [true], resetting>
Everything was working perfectly fine until the power outage. This is a single node in the cluster and it is running on the same machine as the java application (centos 6.5) so I know this isn't the same issue you keep finding on SO and on google that states this issue is caused by different version of elasticsearch and/or Java.
Does anyone no how to recover from this and get back up and running?
Thanks.
Turns out that when the power went out, a restart triggered an auto update of elasticsearch and the upgraded version didn't support the transport drivers in use.
I am trying to deploy iceScrum in using tomcat. I couldn't start the app in the manager, it is showing the above FAIL message. In ice scrum.log
2014-06-24 08:42:26,983 [http-bio-8080-exec-2] ERROR org.springframework.web.context.ContextLoader - Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'pluginManager' defined in ServletContext resource [/WEB-INF/applicationContext.xml]: Invocation of init method failed; nested exception is java.lang.NullPointerException: Cannot invoke method getAt() on null object
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException: Cannot invoke method getAt() on null object
... 3 more
I have no idea what went wrong. I followed all the steps mentioned in https://www.kagilum.com/documentation/install-guide/#application-server. Have anyone come across this kind of issues?
P.S. I am an iOS dev.
icescrum dosent support Java 8 yet.
iceScrum requires Java 6 or 7. Please note that early versions of Java
6 may not work properly and that Java 8 support is planned but not
available yet.
changing the java version to 7 fixed the issue. And the app runs without any problems.
I want to enable OAuth2 provider in my web application, which is based on Grails 2.2.2. However I'm struggling with spring-security-oauth2-provider plugin.
spring-security-oauth2-provider plugin uses spring-security-oauth2 library. I'm trying to run plugin version 1.0.4-SNAPSHOT from Git, which uses spring-security-oauth2-1.0.4.RELEASE library.
After plugin installation, my application will not start, saying that it could not initialize "oauth2ProviderFilter" bean with this exception and stack trace:
| Error 2013-06-15 23:51:51,434 [localhost-startStop-1] ERROR context.GrailsContextLoader - Error initializing the application: Error creating bean with name 'oauth2ProviderFilter': Initialization of bean failed; nested exception is java.lang.reflect.MalformedParameterizedTypeException
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'oauth2ProviderFilter': Initialization of bean failed; nested exception is java.lang.reflect.MalformedParameterizedTypeException
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:527)
at org.codehaus.groovy.grails.commons.spring.ReloadAwareAutowireCapableBeanFactory.doCreateBean(ReloadAwareAutowireCapableBeanFactory.java:122)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:607)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:925)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:472)
at org.codehaus.groovy.grails.commons.spring.DefaultRuntimeSpringConfiguration.getApplicationContext(DefaultRuntimeSpringConfiguration.java:153)
at org.codehaus.groovy.grails.commons.spring.GrailsRuntimeConfigurator.configure(GrailsRuntimeConfigurator.java:170)
at org.codehaus.groovy.grails.commons.spring.GrailsRuntimeConfigurator.configure(GrailsRuntimeConfigurator.java:127)
at org.codehaus.groovy.grails.web.context.GrailsConfigUtils.configureWebApplicationContext(GrailsConfigUtils.java:121)
at org.codehaus.groovy.grails.web.context.GrailsContextLoader.initWebApplicationContext(GrailsContextLoader.java:107)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:111)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4887)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5381)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1559)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1549)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.reflect.MalformedParameterizedTypeException
at sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl.validateConstructorArguments(ParameterizedTypeImpl.java:60)
at sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl.<init>(ParameterizedTypeImpl.java:53)
at sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl.make(ParameterizedTypeImpl.java:95)
at sun.reflect.generics.factory.CoreReflectionFactory.makeParameterizedType(CoreReflectionFactory.java:105)
at sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:140)
at sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
at sun.reflect.generics.repository.ConstructorRepository.getParameterTypes(ConstructorRepository.java:94)
at java.lang.reflect.Method.getGenericParameterTypes(Method.java:291)
at java.beans.FeatureDescriptor.getParameterTypes(FeatureDescriptor.java:387)
at java.beans.MethodDescriptor.setMethod(MethodDescriptor.java:114)
at java.beans.MethodDescriptor.<init>(MethodDescriptor.java:72)
at java.beans.MethodDescriptor.<init>(MethodDescriptor.java:56)
at java.beans.Introspector.getTargetMethodInfo(Introspector.java:1130)
at java.beans.Introspector.getBeanInfo(Introspector.java:414)
at java.beans.Introspector.getBeanInfo(Introspector.java:161)
at org.springframework.beans.CachedIntrospectionResults.<init>(CachedIntrospectionResults.java:217)
at org.springframework.beans.CachedIntrospectionResults.forClass(CachedIntrospectionResults.java:142)
at org.springframework.beans.BeanWrapperImpl.getCachedIntrospectionResults(BeanWrapperImpl.java:324)
at org.springframework.beans.BeanWrapperImpl.getPropertyDescriptors(BeanWrapperImpl.java:331)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.filterPropertyDescriptorsForDependencyCheck(AbstractAutowireCapableBeanFactory.java:1242)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1101)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
After investigationg on oauth2 library documentation (here), I found out that oauth2ProviderFilter is an instance of OAuth2AuthenticationProcessingFilter. After inspecting source of this class (github), it looks like there are few properties (authenticationEntryPoint, authenticationManager, authencticationDetailsSource) that need to be instantiated. I thought that there might be a problem with dependency injection, so I tried to define oauth2ProviderFilter bean in resources.groovy with references to instances defined by Spring Security Core plugin.
I placed this in my resources.groovy:
beans = {
oauth2ProviderFilter(OAuth2AuthenticationProcessingFilter){
authenticationEntryPoint = ref('basicAuthenticationEntryPoint')
authenticationManager = ref('authenticationManager')
authenticationDetailsSource = ref('authenticationDetailsSource')
}
}
That did not solve the problem, there is still error saying that this filter can't be instantiated.
Since I'm no spring expert, do you think that this error might be connected with dependency injection during bean creation? Where might be the problem?
Is it possible that spring-security-oauth2 library is designed for spring framework version which and grails might be using different framework version and this might cause problems?
What are next steps I could take to find the cause of the problem and eventually fix the problem?
What happens when you run the release version of the plugin and not the SNAPSHOT? I found that when I work with latest versions of grails, many times they break existing plugins. So what I would do is...
Try to use release version of plugin with your 2.2.2 grails.
If that does not work, I would step back my grails version and try an older one with the released plugin.
If an older version does not work, I might be missing something in my setup, so I would try to figure out what it is.
If things work in older versions and it looks like my setup is good based on setting it up, then I would ask on the grails nabble on what is up and/or open a JIRA ticket.
Another thing to consider is how young/old the plugin is and how many previous bugs it had. We have many times either decided to debug the plugin because it was a worthwhile effort, but sometimes we decided to not use it because it had too many issues and were going to keep us back in the future.