Unable to refresh OAuth token with custom AuthenticationManager and JWT tokens enabled - spring-boot

Question

We are facing a problem with Spring Security OAuth2 (v 2.0.8). The app is a Spring Boot application (v 1.2.6) with a custom authentication manager and it uses JWT tokens instead of default random value tokens. We are also using Spring Cloud Angel.SR4 and will upgrade to Brixton eventually but at the moment we "cannot" move to Spring Boot 1.3.
The application uses the default Spring Security configuration parameters to configure security for actuator endpoints:
security.user.name:user
security.user.password:password
This is the relevant part of the OAuth2 configuration:
#Configuration
#EnableAuthorizationServer
protected static class AuthorizationServerConfig extends AuthorizationServerConfigurerAdapter {
private AuthenticationManager oauthUserAuthenticationManager() {
Collection<UserDetails> users = new LinkedList<>();
users.add(new User("john", "doe", Arrays.asList(new SimpleGrantedAuthority("USER"))));
users.add(new User("jane", "doe", Arrays.asList(new SimpleGrantedAuthority("USER"), new SimpleGrantedAuthority("ADMIN"))));
UserDetailsManager manager = new InMemoryUserDetailsManager(users);
DaoAuthenticationProvider p = new DaoAuthenticationProvider();
p.setUserDetailsService(manager);
return new ProviderManager(Arrays.asList(p));
}
#Bean
public JwtAccessTokenConverter jwtAccessTokenConverter() {
JwtAccessTokenConverter converter = new JwtAccessTokenConverter();
Resource fsr = resourceLoader.getResource(keystore);
KeyPair keyPair =
new KeyStoreKeyFactory(fsr, keystorePassword.toCharArray())
.getKeyPair(keyPairAlias, keyPairPassword.toCharArray());
converter.setKeyPair(keyPair);
return converter;
}
public void configure(AuthorizationServerEndpointsConfigurer endpoints) throws Exception {
endpoints
.authenticationManager(oauthUserAuthenticationManager())
.accessTokenConverter(jwtAccessTokenConverter());
}
}
The application works fine when using password grant (grant_type=password) and tokens are successfully granted using john/doe or jane/doe. However, when trying to use the refresh token grant (grant_type=refresh_token), an exception is thrown: UsernameNotFoundException: john.
From debugging I have seen that the error originates in DefaultTokenServices.refreshAccessToken() (line 150) when the authenticationManager is used to verify that the user still exists:
user = authenticationManager.authenticate(user);
The debugger also reveals that an authentication with username user is accepted by the authenticationManager, so it looks as if the tokenServices uses the default authentication manager and the password token granter (ResourceOwnerPasswordTokenGranter) uses the custom authentication manager that was configured by the AuthorizationServerEndpointsConfigurer in our configuration.
We believe that what we need to do is to configure the same authentication manager in both places, but how?