Spring Boot - PoolExhaustedException - spring

I recently rebuild all my Spring 4 Projects with the latest Spring Boot Starter Framework.
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.5.7.RELEASE</version>
</parent>
Everything is working fine so far, except that i am experiencing PoolExhaustedException randomly across all my rebuild projects.
The exact exception is:
org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-nio-8080-exec-4] Timeout: Pool empty. Unable to fetch a connection in 20 seconds, none available[size:50; busy:50; idle:0; lastwait:20000].
The last exception i received was from a controller which is downloading a document from my database:
#RequestMapping(value = "/getDocument/{value}/{text}", method = RequestMethod.GET)
public void get(HttpServletResponse response, #PathVariable String value, #PathVariable String text){
try {
Document ufile = documentService.getDocumentByID(Integer.parseInt(value));
response.setContentType(ufile.getType());
response.setContentLength(ufile.getContent().length);
FileCopyUtils.copy(ufile.getContent(), response.getOutputStream());
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
Service:
#Transactional
public Document getDocumentByID(int id) {
Document r = this.documentDAO.getDocumentByID(id);
return r;
}
DAO:
#Transactional
public Document getDocumentByID(int id)
{
Session session = this.sessionFactory.getCurrentSession();
Document p = (Document) session.load(Document.class, new Integer(id));
return p;
}
I already tried to annotate the Controller with #Transactional which temporarily solved the problem, but led to other transactional errors so i had to remove it.
I also tried to increase the pool via project.properties too 100 which only delayed the problem but didn't solve it.
project.properties
spring.datasource.tomcat.max-wait=20000
spring.datasource.tomcat.max-active=50
spring.datasource.tomcat.max-idle=20
spring.datasource.tomcat.min-idle=15
spring.jpa.properties.hibernate.dialect = org.hibernate.dialect.MySQLDialect
spring.jpa.properties.hibernate.id.new_generator_mappings = false
spring.jpa.properties.hibernate.format_sql = true
spring.mvc.view.prefix: /WEB-INF/views/
spring.mvc.view.suffix: .jsp
spring.jpa.properties.hibernate.current_session_context_class=org.springframework.orm.hibernate4.SpringSessionContext
spring.jpa.hibernate.naming.physical-strategy=org.hibernate.boot.model.naming.PhysicalNamingStrategyStandardImpl

Not sure if the properties with these names have any effect.
I'd also try to put a breakpoint and inspect the Tomcat DataSource directly to view the values to be convinced (can be easily autowired in a test).
Looking at the Spring Boot Tomcat Datasource Configuration Tests I see some interesting options like minEvictableIdleTimeMillis, timeBetweenEvictionRunsMillis and maxWait all prefixed with spring.datasource.tomcat., so give them a try.

Related

Include newly added data sources into route Data Source object without restarting the application server

Implemented Spring's AbstractRoutingDatasource by dynamically determining the actual DataSource based on the current context.
Refered this article : https://www.baeldung.com/spring-abstract-routing-data-source.
Here on spring boot application start up . Created a map of contexts to datasource objects to configure our AbstractRoutingDataSource. All these client context details are fetched from a database table.
#Bean
#DependsOn("dataSource")
#Primary
public DataSource routeDataSource() {
RoutingDataSource routeDataSource = new RoutingDataSource();
DataSource defaultDataSource = (DataSource) applicationContext.getBean("dataSource");
List<EstCredentials> credentials = LocalDataSourcesDetailsLoader.getAllCredentails(defaultDataSource); // fetching from database table
localDataSourceRegistrationBean.registerDataSourceBeans(estCredentials);
routeDataSource.setDefaultTargetDataSource(defaultDataSource);
Map<Object, Object> targetDataSources = new HashMap<>();
for (Credentials credential : credentials) {
targetDataSources.put(credential.getEstCode().toString(),
(DataSource) applicationContext.getBean(credential.getEstCode().toString()));
}
routeDataSource.setTargetDataSources(targetDataSources);
return routeDataSource;
}
The problem is if i add a new client details, I cannot get that in routeDataSource. Obvious reason is that these values are set on start up.
How can I achieve to add new client context and I had to re intialize the routeDataSource object.
Planning to write a service to get all the client context newly added and reset the routeDataSource object, no need to restart the server each time any changes in the client details.
A simple solution to this situation is adding #RefreshScope to the bean definition:
#Bean
#Primary
#RefreshScope
public DataSource routeDataSource() {
RoutingDataSource routeDataSource = new RoutingDataSource();
DataSource defaultDataSource = (DataSource) applicationContext.getBean("dataSource");
List<EstCredentials> credentials = LocalDataSourcesDetailsLoader.getAllCredentails(defaultDataSource); // fetching from database table
localDataSourceRegistrationBean.registerDataSourceBeans(estCredentials);
routeDataSource.setDefaultTargetDataSource(defaultDataSource);
Map<Object, Object> targetDataSources = new HashMap<>();
for (Credentials credential : credentials) {
targetDataSources.put(credential.getEstCode().toString(),
(DataSource) applicationContext.getBean(credential.getEstCode().toString()));
}
routeDataSource.setTargetDataSources(targetDataSources);
return routeDataSource;
}
Add Spring Boot Actuator as a dependency:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
Then trigger the refresh endpoint POST to /actuator/refresh to update the DataSource (actually every refresh scoped bean).
So this will depend on how much you know about the datasources to be added, but you could set this up as a multi-tenant project. Another example of creating new datasources:
#Autowired private Map <String, Datasource> mars2DataSources;
public void addDataSourceAtRuntime() {
DataSourceBuilder dataSourcebuilder = DataSourcebuilder.create(
MultiTenantJPAConfiguration.class.getclassloader())
.driverclassName("org.postgresql.Driver")
.username("postgres")
.password("postgres")
.url("Jdbc: postgresql://localhost:5412/somedb");
mars2DataSources("tenantX", datasourcebuilder.build())
}
Given that you are using Oracle, you could also use its database change notification features.
Think of it as a listener in the JDBC driver that gets notified whenever something changes in your database table. So upon receiving a change, you could reinitialize/add datasources.
You can find a tutorial of how to do this here: https://docs.oracle.com/cd/E11882_01/java.112/e16548/dbchgnf.htm#JJDBC28820
Though, depending on your organization database notifications need some extra firewall settings for the communication to work.
Advantage: You do not need to manually call the REST Endpoint if something changes, (though Marcos Barberios answer is perfectly valid!)

Problem with connection to Neo4j test container using Spring boot 2 and JUnit5

Problem with connection to Neo4j test container using Spring boot 2 and JUnit5
int test context. Container started successfully but spring.data.neo4j.uri property has a wrong default port:7687, I guess this URI must be the same when I call neo4jContainer.getBoltUrl().
Everything works fine in this case:
#Testcontainers
public class ExampleTest {
#Container
private static Neo4jContainer neo4jContainer = new Neo4jContainer()
.withAdminPassword(null); // Disable password
#Test
void testSomethingUsingBolt() {
// Retrieve the Bolt URL from the container
String boltUrl = neo4jContainer.getBoltUrl();
try (
Driver driver = GraphDatabase.driver(boltUrl, AuthTokens.none());
Session session = driver.session()
) {
long one = session.run("RETURN 1",
Collections.emptyMap()).next().get(0).asLong();
assertThat(one, is(1L));
} catch (Exception e) {
fail(e.getMessage());
}
}
}
But SessionFactory is not created for the application using autoconfiguration following to these recommendations - https://www.testcontainers.org/modules/databases/neo4j/
When I try to create own primary bean - SessionFactory in test context I get the message like this - "URI cannot be returned before the container is not loaded"
But Application runs and works perfect using autoconfiguration and neo4j started in a container, the same cannot be told about the test context
You cannot rely 100% on Spring Boot's auto configuration (for production) in this case because it will read the application.properties or use the default values for the connection.
To achieve what you want to, the key part is to create a custom (Neo4j-OGM) Configuration bean. The #DataNeo4jTest annotation is provided by the spring-boot-test-autoconfigure module.
#Testcontainers
#DataNeo4jTest
public class TestClass {
#TestConfiguration
static class Config {
#Bean
public org.neo4j.ogm.config.Configuration configuration() {
return new Configuration.Builder()
.uri(databaseServer.getBoltUrl())
.credentials("neo4j", databaseServer.getAdminPassword())
.build();
}
}
// your tests
}
For a broader explanation have a look at this blog post. Esp. the section Using with Neo4j-OGM and SDN.

Spring boot webflux TEXT_EVENT_STREAM_VALUE is not working

I'm using spring-boot with the dependency spring-boot-starter-webflux,
i want to get one data per second with browser,
when i use spring-boot version 2.1.4,the code is working,
but 2.1.5 or greater version not working,
i will get all of the data at 10 sesonds later,not one data per second
I want to get the reason,or others i should do
I find spring-boot update the dependency of netty in 2.1.5,
so if i add the dependency in my pom.xml with
<dependency>
<groupId>io.projectreactor.netty</groupId>
<artifactId>reactor-netty</artifactId>
<version>0.8.8.RELEASE</version>
</dependency>
it working
#RestController
#RequestMapping("/demo")
public class DemoController {
// just get a string per second
#GetMapping(value = "",produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<String> getMsg(){
return Flux.fromStream(new Random().ints(10).mapToObj(intStream ->
{
try {
TimeUnit.SECONDS.sleep(1);
} catch (InterruptedException e) {
e.printStackTrace();
}
return "this is data "+intStream;
}));
}
}
I believe this will achieve what it is you are going for.
#GetMapping(value = "",produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<String> getMsg(){
return Flux.fromStream(new Random()
.ints(10)
.mapToObj(value -> "this is data " + value))
.delayElements(Duration.ofSeconds(1));
}

Hibernate 4 -> 5 upgrade leads to LazyInitializationException

I upgraded a project from Spring 4.3.9.RELEASE + Hibernate 4.3.11.Final to Spring Boot 2.1.4.RELEASE and Hibernate 5.3.9.Final. The queries are still working fine, but I'm getting LazyInitializationException with some #OneToMany class members.
First I retrieve the object, which has a reference to a #OneToMany List, from the #Transaction service. The collection is returned to the controller, and from there it goes back to Spring to be serialized into a json. The controller has #RestController, so it knows what to do.
In Spring 4.3.9.RELEASE + Hibernate 4.3.11.Final everything was fine, even though OpenEntityManagerInView wasn't enabled by configuration and the collection wasn't loaded with EAGER mode. But in Spring Boot 2.1.4.RELEASE and Hibernate 5.3.9.Final the same thing doesn't work anymore. I've tried enabling OEMIV, by setting spring.jpa.open-in-view=true, but even this doesn't seem to work or it's being overriden somewhere.
If I enable EAGER loading mode for that collection, everything works fine.
pom.xml
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
#Entity
#JsonSerialize(using = TemplateSerializer.class)
public class Template implements Serializable {
#Id
#GeneratedValue
private Long id;
private String name;
#ManyToOne
private ObjFormat objFormat;
#OneToOne
#JoinColumn(name = "event_id")
#OnDelete(action = OnDeleteAction.CASCADE)
private Event event;
#OneToMany
#JoinColumn(name = "category_id")
private List<Category> linkToCategories;
The problem is caused by field linkToCategories. If I configure #OneToMany(fetch = FetchType.EAGER) everything works fine.
Application configuration:
#Bean
public LocalSessionFactoryBean sessionFactory(DataSource dataSource) throws ClassNotFoundException {
LocalSessionFactoryBean localSessionFactoryBean = new LocalSessionFactoryBean();
localSessionFactoryBean.setDataSource(dataSource);
localSessionFactoryBean.setPackagesToScan("com.project.backend.model",
"com.project.backend.hibernate.converters");
return localSessionFactoryBean;
}
#Bean
public HibernateTransactionManager transactionManager(SessionFactory sessionFactory) {
return new HibernateTransactionManager(sessionFactory);
}
Later edit:
After a lot of debugging, the difference between the old and the new Hibernate functionality is in the HibernateTransactionManager. In the method doGetTransaction(), in Hibernate 4 it finds the SessionHolder object when calling
TransactionSynchronizationManager.getResource(getSessionFactory())
while in Hibernate 5 it doesn't.
SessionHolder sessionHolder =
(SessionHolder) TransactionSynchronizationManager.getResource(getSessionFactory());
if (sessionHolder != null) {
if (logger.isDebugEnabled()) {
logger.debug("Found thread-bound Session [" + sessionHolder.getSession() + "] for Hibernate transaction");
}
txObject.setSessionHolder(sessionHolder);
}
else if (this.hibernateManagedSession) {
try {
Session session = this.sessionFactory.getCurrentSession();
if (logger.isDebugEnabled()) {
logger.debug("Found Hibernate-managed Session [" + session + "] for Spring-managed transaction");
}
txObject.setExistingSession(session);
}
catch (HibernateException ex) {
throw new DataAccessResourceFailureException(
"Could not obtain Hibernate-managed Session for Spring-managed transaction", ex);
}
}
In the method doBegin, a new session is created and set on the txObject for every request.
if (txObject.getSessionHolder() == null || txObject.getSessionHolder().isSynchronizedWithTransaction()) {
Interceptor entityInterceptor = getEntityInterceptor();
Session newSession = (entityInterceptor != null ?
getSessionFactory().withOptions().interceptor(entityInterceptor).openSession() :
getSessionFactory().openSession());
if (logger.isDebugEnabled()) {
logger.debug("Opened new Session [" + newSession + "] for Hibernate transaction");
}
txObject.setSession(newSession);
}
My experience with Hibernate is fairly small, so here I'm stuck. It's probably a configuration thing, but I can't find it.
As M. Deinum was saying, the Spring 4.3.9.RELEASE + Hibernate 4.3.11.Final configuration was loading OpenSessionInViewFilter, which explains why all the queries were going through successfully. After configuring the same filter in Spring Boot, everything is back to normal. Add the following bean to register the filter:
#Bean
public FilterRegistrationBean<OpenSessionInViewFilter> registerOpenSessionInViewFilterBean() {
FilterRegistrationBean<OpenSessionInViewFilter> registrationBean = new FilterRegistrationBean<>();
OpenSessionInViewFilter filter = new OpenSessionInViewFilter();
registrationBean.setFilter(filter);
return registrationBean;
}
The next step is to replace plain Hibernate with JPA, and OpenSessionInViewFilter with OpenEntityManagerInViewFilter.
Thanks M. Deinum.
#xxxToMany annotations inicates that fetch type is LAZY by default. It means that you need to initialize collection your entity refers to.
Eg.
#Entity
public class Book {
#OneToMany
public List<Author> authors;
}
There is few ways to resolve this. You can modify #OneToMany annotation with:
#OneToMany(FetcType=FetchType.EAGER)
Or to make a method where you will initialize authors eg.:
public void initializeAuthors(Book book) {
Book b = em.find(Book.class, book.getId());
List<Author> authors = new ArrayList<>(b.getAuthors());
book.setAuthors(authors);
}
If you have #NamedQueries on your entities, you can do that by adding LEFT JOIN FETCH on your collections.

ActiveMQ fails to start (EOFException: null)

I'm having issues start my spring web app with camel and ActiveMQ.
The particular error I'm getting is not very descriptive:
16:18:53.552 [localhost-startStop-1] ERROR o.a.a.b.BrokerService - Failed to start Apache ActiveMQ ([activemq.myworkingdomain.com, ID:Ricardos-MacBook-Air.local-65257-1453738732697-0:2], {})
java.io.EOFException: null
at java.io.DataInputStream.readBoolean(DataInputStream.java:244) ~[na:1.8.0_60]
at org.apache.activemq.openwire.v11.SubscriptionInfoMarshaller.looseUnmarshal(SubscriptionInfoMarshaller.java:133) ~[activemq-client-5.13.0.jar:5.13.0]
at org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:366) ~[activemq-client-5.13.0.jar:5.13.0]
at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:277) ~[activemq-client-5.13.0.jar:5.13.0]
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBTopicMessageStore$1.execute(KahaDBStore.java:755) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.kahadb.page.Transaction.execute(Transaction.java:769) ~[kahadb-5.7.0.jar:5.7.0]
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBTopicMessageStore.getAllSubscriptions(KahaDBStore.java:749) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBTopicMessageStore.<init>(KahaDBStore.java:663) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.store.kahadb.KahaDBStore.createTopicMessageStore(KahaDBStore.java:920) ~[activemq-core-5.7.0.jar:5.7.0]
at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.createTopicMessageStore(KahaDBPersistenceAdapter.java:100) ~[activemq-core-5.7.0.jar:5.7.0]
I'm sticking with java and simple spring stuff no xml:
#Bean
public CamelContext camelContext() {
final CamelContext camelContext = new DefaultCamelContext();
camelContext.addComponent("activemq", activeMQComponent());
try {
CamelConfigurator.addRoutesToCamel(camelContext);
camelContext.start();
} catch (final Exception e) {
LOGGER.error("Failed to start the camel context", e);
}
LOGGER.info("Started the Camel context and components");
return camelContext;
}
#Bean
public ActiveMQComponent activeMQComponent() {
final ActiveMQComponent activeMQComponent = new ActiveMQComponent();
activeMQComponent.setConfiguration(jmsConfiguration());
activeMQComponent.setTransacted(true);
activeMQComponent.setCacheLevelName("CACHE_CONSUMER");
return activeMQComponent;
}
#Bean
public JmsConfiguration jmsConfiguration() {
final JmsConfiguration jmsConfiguration = new JmsConfiguration(pooledConnectionFactory());
jmsConfiguration.setConcurrentConsumers(CONCURRENT_CONSUMERS);
return jmsConfiguration;
}
#Bean
public PooledConnectionFactory pooledConnectionFactory() {
final PooledConnectionFactory pooledConnectionFactory = new PooledConnectionFactory(
activeMQConnectionFactory());
pooledConnectionFactory.setMaxConnections(MAX_CONNECTIONS_TO_POOL_FACTORY);
// pooledConnectionFactory.start();
return pooledConnectionFactory;
}
#Bean
public ActiveMQConnectionFactory activeMQConnectionFactory() {
return new ActiveMQConnectionFactory(username, password, activeMQBrokerURL);
}
I've been trying to change the order in which things load also the routes and what's included in them, deleting the kahadb local folder but nothing seems to work or even point me in the right location.
Your stacktrace shows that your application in using
activemq-client-5.13.0
activemq-core-5.7.0
I am pretty sure that the version mismatch is responsible for this error.
Can you just import activemq-all 5.13.0 and try again?
According to your stacktrace, seems like you are experiencing issues with KahaDB (the persistence engine by default for Apache ActiveMQ).
I believe KahaDB has a bug within it and you can find information about it on the Apache issues Web Page.
I had this issue once, but it strange since it seems like it happens at random, sometimes it works, sometime it fails.
I managed to fix this issue by choosing a different persistence engine for ActiveMQ. Maybe you can give it a try and tell me if it works for you. Hope this information can help you.

Resources