Hibernate session closed for no apparent reason - spring

Question

I've been struggling with this for days and i really don't know what's happening here.
I have a few processes that are triggered by a user and happen on a seperate thread, which does some operations, and updates an entry in the DB about it's progress which the user can retrieve.
this has all been working fine until recently, when suddenly, sometimes, seemingly uncorrelated with anything, these processes fail on their first attempt to lazy load an entity. They fail with one of a few different errors, all of which eventually stem from the hibernate seesion being closed somehow:
org.hibernate.SessionException: Session is closed. The read-only/modifiable setting is only accessible when the proxy is associated with an open session.
-
org.hibernate.SessionException: Session is closed!
-
org.hibernate.LazyInitializationException: could not initialize proxy - the owning Session was closed
-
org.hibernate.exception.GenericJDBCException: Could not read entity state from ResultSet : EntityKey[com.rdthree.plenty.domain.crops.CropType#1]
-
java.lang.NullPointerException
at com.mysql.jdbc.ResultSetImpl.checkColumnBounds(ResultSetImpl.java:766)
i'm using spring #Transactional to manage my transactions
here's my config:
#Bean
public javax.sql.DataSource dataSource() {
// update TTL so that the datasource will pick up DB failover - new IP
java.security.Security.setProperty("networkaddress.cache.ttl", "30");
HikariDataSource ds = new HikariDataSource();
String jdbcUrl = "jdbc:mysql://" + plentyConfig.getHostname() + ":" + plentyConfig.getDbport() + "/"
+ plentyConfig.getDbname() + "?useSSL=false";
ds.setJdbcUrl(jdbcUrl);
ds.setUsername(plentyConfig.getUsername());
ds.setPassword(plentyConfig.getPassword());
ds.setConnectionTimeout(120000);
ds.setMaximumPoolSize(20);
return ds;
}
#Bean
public JpaTransactionManager transactionManager(EntityManagerFactory entityManagerFactory) {
JpaTransactionManager manager = new JpaTransactionManager();
manager.setEntityManagerFactory(entityManagerFactory);
return manager;
}
#Bean
#Autowired
public LocalContainerEntityManagerFactoryBean entityManagerFactory(javax.sql.DataSource dataSource) {
LocalContainerEntityManagerFactoryBean factory = new LocalContainerEntityManagerFactoryBean();
factory.setDataSource(dataSource);
factory.setPackagesToScan("com.rdthree.plenty.domain");
Properties properties = new Properties();
properties.put("org.hibernate.flushMode", "ALWAYS");
properties.put("hibernate.cache.use_second_level_cache", "true");
properties.put("hibernate.cache.use_query_cache", "true");
properties.put("hibernate.cache.region.factory_class",
"com.rdthree.plenty.config.PlentyInfinispanRegionFactory");
properties.put("hibernate.cache.infinispan.statistics", "true");
properties.put("hibernate.cache.infinispan.query", "distributed-query");
properties.put("hibernate.enable_lazy_load_no_trans", "true");
if (plentyConfig.getProfile().equals(PlentyConfig.UNIT_TEST)
|| plentyConfig.getProfile().equals(PlentyConfig.PRODUCTION_INIT)) {
properties.put("hibernate.cache.infinispan.cfg", "infinispan-local.xml");
} else {
properties.put("hibernate.cache.infinispan.cfg", "infinispan.xml");
}
factory.setJpaProperties(properties);
HibernateJpaVendorAdapter adapter = new HibernateJpaVendorAdapter();
adapter.setShowSql(false);
adapter.setDatabasePlatform("org.hibernate.dialect.MySQLDialect");
factory.setJpaVendorAdapter(adapter);
return factory;
}
the way it works is that the thread fired by the user iterates over a collection of plans and applies each of them, the process that applies the plan also updates the progress entity on the DB.
the whole thread bean is marked at transactional:
#Component
#Scope("prototype")
#Transactional(propagation = Propagation.REQUIRES_NEW, isolation = Isolation.READ_COMMITTED)
public class TemplatePlanApplicationThreadBean extends AbstractPlentyThread implements TemplatePlanApplicationThread {
...
#Override
public void run() {
startProcessing();
try {
logger.trace(
"---Starting plan manifestation for " + fieldCropReplaceDatePlantationDates.size() + " fields---");
List<Plan> plans = new ArrayList<>();
for (FieldCropReplaceDatePlantationDates obj : fieldCropReplaceDatePlantationDates) {
for (TemplatePlan templatePlan : obj.getTemplatePlans()) {
try {
plans.add(planService.findActivePlanAndManifestTemplatePlan(templatePlan, organization,
obj.getPlantationDate(), obj.getReplacementStartDate(), obj.getFieldCrop(),
autoSchedule, schedulerRequestArguments, planApplicationProgress, false));
} catch (ActivityException e) {
throw new IllegalArgumentException(e);
}
}
Plan plan = plans.get(plans.size() - 1);
plan = planService.getEntityById(plan.getId());
if (plan != null) {
planService.setUnscheduledPlanAsSelected(plan);
}
plans.clear();
}
if (planApplicationProgressService.getEntityById(planApplicationProgress.getId()) != null) {
planApplicationProgressService.deleteEntity(planApplicationProgress.getId());
}
} catch (Exception e) {
logger.error(PlentyUtils.extrapolateStackTrace(e));
connector.createIssue("RdThreeLLC", "plenty-web",
new GitHubIssueRequest("Template plan application failure",
"```\n" + PlentyUtils.extrapolateStackTrace(e) + "\n```", 1, new ArrayList<>(),
Lists.newArrayList("plentytickets")));
planApplicationProgress.setFailed(true);
planApplicationProgressService.saveEntity(planApplicationProgress);
} finally {
endProcessing();
}
}
here is the method called by the thread:
#Override
#Transactional(propagation = Propagation.REQUIRES_NEW)
public synchronized Plan findActivePlanAndManifestTemplatePlan(TemplatePlan templatePlan,
ServiceProviderOrganization organization, Date plantationDate, Date replacementDate, FieldCrop fieldCrop,
boolean autoSchedule, SchedulerRequestArguments schedulerRequestArguments, Progress planProgress,
boolean commit) throws ActivityException {
Plan oldPlan = getLatestByFieldCrop(fieldCrop);
List<Activity> activitiesToRemove = oldPlan != null
? findActivitiesToRemoveAfterDate(oldPlan, replacementDate != null ? replacementDate : new Date())
: new ArrayList<>();
List<PlanExpense> planExpensesToRemove = oldPlan != null
? findPlanExpensesToRemoveAfterDate(oldPlan, replacementDate != null ? replacementDate : new Date())
: new ArrayList<>();
Date oldPlanPlantationDate = oldPlan != null ? inferPlanDates(oldPlan).getPlantationDate() : null;
if (oldPlan != null) {
if (commit) {
oldPlan.setReplaced(true);
}
buildPlanProfitProjectionForPlanAndField(oldPlan, Sets.newHashSet(activitiesToRemove),
Sets.newHashSet(planExpensesToRemove));
}
if (commit) {
for (Activity activity : activitiesToRemove) {
activityService.deleteEntity(activity.getId());
}
for (PlanExpense planExpense : planExpensesToRemove) {
planExpenseService.deleteEntity(planExpense.getId());
}
}
oldPlan = oldPlan != null ? getEntityById(oldPlan.getId()) : null;
Plan plan = manifestTemplatePlan(templatePlan, oldPlan, organization,
plantationDate != null ? plantationDate : oldPlanPlantationDate, replacementDate, fieldCrop,
autoSchedule, schedulerRequestArguments, planProgress, commit);
if (!commit) {
setPlanAllocationsUnscheduled(plan);
}
return plan;
}
the thing that kills me is that the error happens only sometimes, so i can't really debug it and i can't correlate it with anything.
Any ideas about what would cause the session to close?
i tried turning off any other threads, so basically this is the only one that's running, didn't help
Thanks