While starting the application once the scheduler starts running , I get the following error
Caused by: java.sql.SQLException: MaxOpenPreparedStatements limit reached
Below is the properties
<entry key="jdbc.preparedStatementCacheSize" value="500" />
<entry key="jdbc.initialSize" value="5" />
<entry key="jdbc.maxActive" value="100" />
<entry key="jdbc.maxIdle" value="${jdbc.initialSize}" />
<entry key="jdbc.minIdle" value="0" />
<entry key="jdbc.validationQuery" value="select database()" />
<entry key="jdbc.maxWait" value="16000"/>
The bean datasource
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource">
<property name="driverClassName" value="${jdbc.driverClassName}"/>
<property name="url" value="${jdbc.url}"/>
<property name="username" value="${jdbc.username}"/>
<property name="password" value="${jdbc.password}"/>
<property name="poolPreparedStatements" value="true" />
<property name="maxOpenPreparedStatements" alue="${jdbc.preparedStatementCacheSize}"/>
<property name="initialSize" value="${jdbc.initialSize}" />
<property name="maxActive" value="${jdbc.maxActive}" />
<property name="maxIdle" value="${jdbc.maxIdle}" />
<property name="minIdle" value="${jdbc.minIdle}" />
<property name="validationQuery" value="${jdbc.validationQuery}" />
</bean>
Instead of org.apache.commons.dbcp.BasicDataSource if I use
org.springframework.jdbc.datasource.DriverManagerDataSource then I wont get this exception.
I am using Spring 2.5. Please help me in resolving this exception.
Related
Tomcat 8
<bean id="dataSource" class="org.apache.tomcat.jdbc.pool.DataSource"
destroy-method="close">
<property name="driverClassName" value="com.mysql.cj.jdbc.Driver" />
<property name="url" value="${database.url}" />
<property name="username" value="${database.username}" />
<property name="password" value="${database.password}" />
<property name="initialSize" value="10" />
<property name="maxActive" value="200" />
<property name="minIdle" value="10" />
<property name="maxIdle" value="50" />
<property name="maxWait" value="50000" />
<property name="fairQueue" value="true" />
<property name="logAbandoned" value="true" />
<property name="removeAbandoned" value="true" />
<property name="testOnBorrow" value="true" />
<property name="timeBetweenEvictionRunsMillis" value="60000" />
<property name="validationQuery" value="${database.validationQuery}" />
</bean>
App is running on tomcat 8 and starts giving the error after 15 20 mins
SQL state [null]; error code [0]; Connection has already been closed.; nested exception is java.sql.SQLException: Connection has already been closed.
at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1542)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:667)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:713)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:738)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:794)
at org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.query(NamedParameterJdbcTemplate.java:209)
at org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.queryForList(NamedParameterJdbcTemplate.java:303)
at org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.queryForList(NamedParameterJdbcTemplate.java:310)
Increase the value of removeAbandonedTimeOut in tomcat-resources.xml so that it is set to the longest running query the application may have.
removed this <property name="removeAbandoned" value="true" /> after change app is working fine
(I hope this below makes sense, I had a bit of a struggle with formatting).
We have an application (currently being refreshed from technology of circa 2005) whose owners require the use of Oracle AQ for message handling.
We are using the Spring JDBC extensions, and when the application begins to listen to the queue it gets a ClassCastException saying "X cannot be cast to oracle.jdbc.internal.OracleConnection".
Where X is either a WrappedConnectionJDK7 from jboss when we use a datasource defined in Wildfly e.g.
<bean id="dataSource-supsrep"
class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="oracle.jdbc.driver.OracleDriver" />
<property name="url" value="jdbc:oracle:thin:#localhost:1521:XE" />
<property name="username" value="...." />
<property name="password" value="...." />
</bean>
or X is a PoolGuardConnectionWrapper from commons.dbcp when we define the datasource like this:
<bean id="dataSource-supsrep"
class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="oracle.jdbc.driver.OracleDriver" />
<property name="url" value="jdbc:oracle:thin:#localhost:1521:XE" />
<property name="username" value="...." />
<property name="password" value="...." />
</bean>
Message Listener bean is defined like this :
<bean id="messageListener" class="xxx.report.dispatch.DispatchMDB" />
Message Listener container bean is like this :
<bean id="reportProcessorXslfoHigh"
class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="sessionTransacted" value="true" />
<property name="connectionFactory" ref="AQconnectionFactory" />
<property name="destinationName" value="Q1_XSLFO_HIGH" />
<property name="messageListener" ref="messageListener" />
<property name="transactionManager" ref="transactionManager" />
<property name="concurrentConsumers" value="1" />
<property name="maxConcurrentConsumers" value="1" />
<property name="receiveTimeout" value="30000" />
<property name="idleTaskExecutionLimit" value="30" />
<property name="idleConsumerLimit" value="1" />
</bean>
The connection factory is like this :
<orcl:aq-jms-connection-factory id="AQconnectionFactory"
data-source="dataSource-supsrep" native-jdbc- extractor="oracleNativeJdbcExtractor" />
and it uses a Native JDBC Extractor :
<bean id="oracleNativeJdbcExtractor"
class="org.springframework.jdbc.support
.nativejdbc.OracleJdbc4NativeJdbcExtractor" />
So my question is - can this be overcome by Spring configuration details - or is there another way round it? Clearly the underlying connection is needed but after 3 days of research I cannot see how it can be done.
Any clues gratefully received !
Thanks,
John
Skip exception class functionality is not working properly when MongoItemreader is being used.
Issues:
1. Spring batch is stuck at a particular data row (in my case 15). It calls MessageContextReadConverter and onReadError for same row number 15 -- 3 times.
2. skiplistener is never invoked.
Scenario:
Suppose 20 rows are fetched from MongoDB and at row number 15 configurable exception ".ConversionFailedException" is thrown from a custom class derived from Converter, (MessageContextReadConverter implements Converter)
. MessageContextReadConverter is set during mongoTemplate creation,.
Now, exception thrown at row number 15 is received in custom readlistener (public void onReadError(Exception ex)).
Configuration XML:
<context:property-placeholder location="classpath:application.properties" />
<context:component-scan base-package="com.XXX.YYY.batch.kernel" />
<context:component-scan base-package="com.XXX.YYY.batch.dao" />
<context:annotation-config />
<!-- Enable Annotation based Declarative Transaction Management -->
<tx:annotation-driven proxy-target-class="true" transaction-manager="transactionManager" />
<!-- Creating TransactionManager Bean, since JDBC we are creating of type DataSourceTransactionManager -->
<bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="dataSource" />
</bean>
<batch:job id="txnLogJob" restartable="false">
<batch:step id="txnload">
<tasklet allow-start-if-complete="true" transaction-manager="transactionManager">
<chunk reader="txnLogReader" processor="txnLogProcessor"
writer="txnLogItemWriter" commit-interval="10" skip-limit="3">
<skippable-exception-classes>
<include class="org.springframework.core.convert.ConversionFailedException" />
</skippable-exception-classes>
</chunk>
<listeners>
<listener ref="stepListener" />
<listener ref="chunklistener" />
<listener ref="readlistener" />
<listener ref="skiplistener" />
</listeners>
</tasklet>
</batch:step>
<batch:listeners>
<batch:listener ref="completionListener" />
</batch:listeners>
</batch:job>
<bean id="txnLogReader" class="org.springframework.batch.item.data.MongoItemReader"
scope="step">
<property name="template" ref="mongoTemplate" />
<property name="query" value=" { }" />
<property name="pageSize" value="50" />
<property name="sort">
<map>
<entry key="audit_info.created_on"
value="#{T(org.springframework.data.domain.Sort.Direction).ASC}" />
</map>
</property>
<property name="collection" value="txnlog" />
<property name="targetType" value="com.XXX.YYY.kernel.msg.MessageContext" />
</bean>
<bean id="completionListener"
class="com.XXX.YYY.batch.listeners.JobCompletionNotificationListener" />
<bean id="stepListener"
class="com.XXX.YYY.batch.listeners.StepExecutionListener" />
<bean id="chunklistener"
class="com.XXX.YYY.batch.listeners.ChunkExecutionListener" />
<bean id="readlistener"
class="com.XXX.YYY.batch.listeners.ReadExecutionListener" />
<bean id="skiplistener"
class="com.XXX.YYY.batch.listeners.SkipExecutionListener" />
<bean id="jobParametersDAOImpl" class="com.XXX.YYY.batch.dao.JobParametersDAOImpl" />
<bean id="batchLoader" class="com.XXX.YYY.batch.kernel.BatchLoader" />
<bean id="batchjobParameter" class="com.XXX.YYY.batch.dao.Batch_Job_Parameters" />
<!-- <bean id="reportWriter" class="org.springframework.batch.item.data.MongoItemWriter">
<property name="template" ref="mongoTemplate" /> <property name="collection"
value="txnlog" /> name of the collection to write </bean> -->
<bean id="txnLogItemWriter" class="org.springframework.batch.item.file.FlatFileItemWriter"
scope="step">
<property name="shouldDeleteIfExists" value="true" />
<property name="resource" value="file:target/test-outputs/output.txt" />
<property name="lineAggregator">
<bean
class="org.springframework.batch.item.file.transform.PassThroughLineAggregator" />
</property>
</bean>
<bean id="txnLogProcessor"
class="com.XXX.YYY.batch.processor.MessageContextItemProcessor" />
<bean id="jobLauncher"
class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
<property name="jobRepository" ref="jobRepository" />
</bean>
<bean id="jobRepository"
class="org.springframework.batch.core.repository.support.JobRepositoryFactoryBean">
<property name="databaseType" value="MYSQL" />
<property name="dataSource" ref="dataSource" />
<property name="transactionManager" ref="transactionManager" />
</bean>
<bean id="dataSource" class="com.XXX.YYY.common.DataSource"
destroy-method="close">
<property name="driverClassName" value="${jdbc.driverClassName}" />
<property name="url" value="${jdbc.url}" />
<property name="username" value="${jdbc.username}" />
<property name="password" value="${jdbc.password}" />
<property name="connectionProperties" value="${jdbc.connectionProperties}" />
<property name="initialSize" value="${jdbc.initialSize}" />
<property name="maxTotal" value="${jdbc.maxTotal}" />
<property name="maxIdle" value="${jdbc.maxIdle}" />
<property name="minIdle" value="${jdbc.minIdle}" />
<property name="maxWaitMillis" value="${jdbc.maxWaitMillis}" />
<property name="testOnBorrow" value="${jdbc.testOnBorrow}" />
<property name="testWhileIdle" value="${jdbc.testWhileIdle}" />
<property name="testOnReturn" value="${jdbc.testOnReturn}" />
<property name="validationQuery" value="${jdbc.validationQuery}" />
</bean>
I was using the SimpleMessageStore ( memory ) as the default Store for an Aggregator, but as i need
persistent store, i started using a jdbc solution backed up by Mysql 5.5
But i'm really concerned about its performance, as using a PerfTest ( databene's contiperf ), memory store achieved up to 6000 messages/sec, whereas using mysql ( or redis, or mongo ), i can't achieve more than 15-20 messages/sec.
<int:aggregator id="XDRAggegator"
input-channel="mqChannel"
output-channel="publishChannel"
message-store="jdbc-messageStore"
....
<bean id="reaper" class="org.springframework.integration.store.MessageGroupStoreReaper">
<property name="messageGroupStore" ref="jdbc-messageStore"/>
<property name="timeout" value="${msg.timeout}"/>
</bean>
<task:scheduler id="taskScheduler"/>
<task:scheduled-tasks scheduler="taskScheduler">
<task:scheduled ref="reaper" method="run" fixed-rate="${reaper.rate}"/>
</task:scheduled-tasks>
<task:executor id="taskExecutor" pool-size="1"/>
<task:annotation-driven executor="taskExecutor" scheduler="taskScheduler"/>
<int-jdbc:message-store id="jdbc-messageStore" data-source="dataSource" />
<bean id="dataSource" class="com.jolbox.bonecp.BoneCPDataSource" >
<property name="driverClass" value="com.mysql.jdbc.Driver" />
<property name="jdbcUrl" value="jdbc:mysql://${database.host}:3306/XDR?autoReconnect=true&cachePrepStmts=true&prepStmtCacheSize=50" />
<property name="username" value="${database.username}"/>
<property name="password" value="${database.password}"/>
<property name="idleConnectionTestPeriodInMinutes" value="1"/>
<property name="idleMaxAgeInMinutes" value="2"/>
<property name="connectionTimeoutInMs" value="1000" />
<property name="maxConnectionsPerPartition" value="10"/>
<property name="minConnectionsPerPartition" value="4"/>
<property name="partitionCount" value="4"/>
<property name="acquireIncrement" value="2"/>
<property name="statementsCacheSize" value="10"/>
<property name="closeConnectionWatch" value="false"/>
</bean>
Am i missing something ?
I'm using default schema that comes into spring-integration-jdbc library.
I was wondering if I can run multiple c3p0 datasources for one DB, something like:
<bean id="dataSource1" class = "com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method="close">
<property name="driverClass" value="${db.driverClassName}"/>
<property name="jdbcUrl" value="${db.url}/schema1"/>
<property name="user" value="${db.username}"/>
<property name="password" value="${db.password}"/>
<property name="acquireIncrement" value="1" />
<property name="idleConnectionTestPeriod" value="100"/>
<property name="minPoolSize" value="5" />
<property name="maxPoolSize" value="50" />
<property name="maxIdleTime" value="1800" />
</bean>
<bean id="dataSource2" class = "com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method="close">
<property name="driverClass" value="${db.driverClassName}"/>
<property name="jdbcUrl" value="${db.url}/schema2"/>
<property name="user" value="${db.username}"/>
<property name="password" value="${db.password}"/>
<property name="acquireIncrement" value="1" />
<property name="idleConnectionTestPeriod" value="100"/>
<property name="minPoolSize" value="5" />
<property name="maxPoolSize" value="50" />
<property name="maxIdleTime" value="1800" />
</bean>
They will be used by difference persistence services.
Thanks.
That's absolutely fine. You'll run into some configuration issues like:
autowiring DataSource by type won't work
#Transactional/declarative transactions will work only against one, selected DataSource. Alternatively you'll have to manually tell which transaction manager you want to use (thus you'll need two transaction managers as well: transactionManager1 and transactionManager2):
#Transactional("transactionalManager2")
Besides, there's nothing wrong with this configuration. Actually it's a pretty good idea (+1): if some layers/components of your application saturate the pool, others can still operate.
The only thing I recommend is to use lesser known abstract/parent bean declarations to avoid repetition:
<bean id="dataSource" abstract="true" class="com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method="close">
<property name="driverClass" value="${db.driverClassName}"/>
<property name="user" value="${db.username}"/>
<property name="password" value="${db.password}"/>
<property name="acquireIncrement" value="1" />
<property name="idleConnectionTestPeriod" value="100"/>
<property name="minPoolSize" value="5" />
<property name="maxPoolSize" value="50" />
<property name="maxIdleTime" value="1800" />
</bean>
<bean id="dataSource1" parent="dataSource">
<property name="jdbcUrl" value="${db.url}/schema1"/>
</bean>
<bean id="dataSource2" parent="dataSource">
<property name="jdbcUrl" value="${db.url}/schema2"/>
</bean>
See also:
What is meant by abstract="true" in spring?