Spring AOP NullPointerException after running successfully for an extended period of time - spring-boot

Question

This is a problem which has stumped myself and two of my colleagues for a few days now.
We are receiving a NullPointerException after our spring-boot microservice has been running without a hitch anywhere from a few minutes to a few hours and has received a few hundred to few thousand requests. This issue started after a few beans were changed to being request-scoped due to a requirements change.
Classes (all objects are autowired/constructed at microservice boot):
// New class introduced to accommodate requirements change.
#Repository("databaseUserAccountRepo")
public class DatabaseAccountUserRepoImpl implements UserLdapRepo {
private final DatabaseAccountUserRepository databaseAccountUserRepository;
#Autowired
public DatabaseAccountUserRepoImpl(
#Qualifier("databaseAccountUserRepositoryPerRequest") final DatabaseAccountUserRepository databaseAccountUserRepository
) {
this.databaseAccountUserRepository = databaseAccountUserRepository;
}
// ...snip...
}
// ==============================================================================
// New class introduced to accommodate requirements change.
#Repository("databaseAccountUserRepository")
public interface DatabaseAccountUserRepository
extends org.springframework.data.repository.CrudRepository {
// ...snip...
}
// ==============================================================================
#Repository("ldapUserAccountRepo")
public class UserLdapRepoImpl implements UserLdapRepo {
// ...snip...
}
// ==============================================================================
#Component
public class LdapUtils {
private final UserLdapRepo userLdapRepo;
#Autowired
public LdapUtils(
#Qualifier("userLdapRepoPerRequest") final UserLdapRepo userLdapRepo
) {
this.userLdapRepo = userLdapRepo;
}
// ...snip...
public Object myMethod(/* whatever */) {
// ...snip...
return userLdapRepo.someMethod(/* whatever */);
}
}
// ==============================================================================
// I have no idea why the original developer decided to do it this way.
// It's worked fine up until now so I see no reason to change it unless
// I really need to.
public class AuthenticationContext {
private static final ThreadLocal<String> organizationNameThreadLocal = new ThreadLocal<>();
// ...snip...
public static void setOrganizationName(String organizationName) {
organizationNameThreadLocal.set(organizationName);
}
public static String getOrganizationName() {
return organizationNameThreadLocal.get();
}
public static void clear() {
organizationNameThreadLocal.remove();
}
// ...snip...
}
// ==============================================================================
public class AuthenticationContextInterceptor extends HandlerInterceptorAdapter {
#Override
public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler)
throws Exception {
AuthenticationContext.setOrganizationName(request.getHeader("customer-id"));
return true;
}
#Override
public void afterCompletion(HttpServletRequest request, HttpServletResponse response,
Object handler, Exception ex) throws Exception {
AuthenticationContext.clear();
}
}
Code to request-scope:
#Configuration
// We have some aspects in our codebase, so this might be relevant.
#EnableAspectJAutoProxy(proxyTargetClass = true)
public class ServiceConfiguration {
// ...snip...
#Bean
#Scope(value = WebApplicationContext.SCOPE_REQUEST, proxyMode = ScopedProxyMode.TARGET_CLASS)
public UserLdapRepo userLdapRepoPerRequest(
final Map<String, String> customerIdToUserLdapRepoBeanName
) {
final String customerId = AuthenticationContext.getOrganizationName();
final String beanName = customerIdToUserLdapRepoBeanName.containsKey(customerId)
? customerIdToUserLdapRepoBeanName.get(customerId)
: customerIdToUserLdapRepoBeanName.get(null); // default
return (UserLdapRepo) applicationContext.getBean(beanName);
}
#Bean
public Map<String, String> customerIdToUserLdapRepoBeanName(
#Value("${customers.user-accounts.datastore.use-database}") final String[] customersUsingDatabaseForAccounts
) {
final Map<String, String> customerIdToUserLdapRepoBeanName = new HashMap<>();
customerIdToUserLdapRepoBeanName.put(null, "ldapUserAccountRepo"); // default option
if (customersUsingDatabaseForAccounts != null && customersUsingDatabaseForAccounts.length > 0) {
Arrays.stream(customersUsingDatabaseForAccounts)
.forEach(customerId ->
customerIdToUserLdapRepoBeanName.put(customerId, "databaseUserAccountRepo")
);
}
return customerIdToUserLdapRepoBeanName;
}
// Given a customer ID (taken from request header), returns the
// DatabaseAccountUserRepository instance for that particular customer.
// The DatabaseAccountUserRepositoryProvider is NOT request-scoped.
// The DatabaseAccountUserRepositoryProvider is basically just a utility
// wrapper around a map of String -> DatabaseAccountUserRepository.
#Bean
#Scope(value = WebApplicationContext.SCOPE_REQUEST, proxyMode = ScopedProxyMode.TARGET_CLASS)
public DatabaseAccountUserRepository databaseAccountUserRepositoryPerRequest(
final DatabaseAccountUserRepositoryProvider databaseAccountUserRepositoryProvider
) {
final String customerId = AuthenticationContext.getOrganizationName();
return databaseAccountUserRepositoryProvider.getRepositoryFor(customerId);
}
// ...snip...
}
The stack trace:
java.lang.NullPointerException: null
at org.springframework.aop.framework.adapter.DefaultAdvisorAdapterRegistry.getInterceptors(DefaultAdvisorAdapterRegistry.java:81)
at org.springframework.aop.framework.DefaultAdvisorChainFactory.getInterceptorsAndDynamicInterceptionAdvice(DefaultAdvisorChainFactory.java:89)
at org.springframework.aop.framework.AdvisedSupport.getInterceptorsAndDynamicInterceptionAdvice(AdvisedSupport.java:489)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:659)
at com.mycompany.project.persistence.useraccount.ldap.UserLdapRepoImpl$$EnhancerBySpringCGLIB$$b6378f51.someMethod(<generated>)
at sun.reflect.GeneratedMethodAccessor304.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy209.findByFederatedInfo(Unknown Source)
at com.mycompany.project.util.LdapUtils.myMethod(LdapUtils.java:141)
The method in which the NPE is thrown is this guy:
//////////////////////////////////////////
// This is a method in Spring framework //
//////////////////////////////////////////
#Override
public MethodInterceptor[] getInterceptors(Advisor advisor) throws UnknownAdviceTypeException {
List<MethodInterceptor> interceptors = new ArrayList<MethodInterceptor>(3);
Advice advice = advisor.getAdvice(); // <<<<<<<<<< line 81
if (advice instanceof MethodInterceptor) {
interceptors.add((MethodInterceptor) advice);
}
for (AdvisorAdapter adapter : this.adapters) {
if (adapter.supportsAdvice(advice)) {
interceptors.add(adapter.getInterceptor(advisor));
}
}
if (interceptors.isEmpty()) {
throw new UnknownAdviceTypeException(advisor.getAdvice());
}
return interceptors.toArray(new MethodInterceptor[interceptors.size()]);
}
Most relevant dependencies:
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.5.10.RELEASE</version>
<relativePath/>
</parent>
<dependencies>
<!-- this results in spring-aop:4.3.14.RELEASE -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-aop</artifactId>
</dependency>
</dependencies>
The request header customer-id is set by our proxy, so it must be available on the request (we added logging to verify that this statement is true; it is).
We do not know the exact traffic pattern which can cause the NPE to start being triggered. Once triggered, all subsequent requests also result in an NPE.
We have several other request-scoped beans in this project; they are also selected using the customer-id. Several of said objects have existed in this project for months prior to this change. They do not exhibit this problem.
We believe that the userLdapRepoPerRequest() and databaseAccountUserRepositoryPerRequest() methods are working correctly - receiving the correct customer-id, is returning the correct object, etc...at least when the methods are hit. This was determined by adding logging to the body of those methods - a log message immediately upon entering the method which records the parameter, one log message verifying value of the customer-id, and one log message immediately before returning which records the value which is to be returned. Note: Our logging setup has a correlation ID present on each message, so we can keep track of what messages corresponds the same request.
It's almost as if Spring is losing track of a few of its proxied beans.
Anyone have any ideas on what's happening or anything you would like us to try? Any leads are much appreciated.