Spring Boot - Kafka Consumer Bean Scope - spring-boot

I'm using a CacheManager in a Spring Boot application with SCOPE_REQUEST scope.
#Bean
#Scope(value = WebApplicationContext.SCOPE_REQUEST, proxyMode = ScopedProxyMode.TARGET_CLASS)
public CacheManager cacheManager() {
return new ConcurrentMapCacheManager();
}
I'm also using Kafka for communication between microservices. Actually I'm receiving an event through a Kafka consumer and I get the following error:
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'scopedTarget.cacheManager': Scope 'request' is not active for the current thread;
...
Caused by: java.lang.IllegalStateException: No thread-bound request found: Are you referring to request attributes outside of an actual web request, or processing a request outside of the originally receiving thread?
It's clear that the CacheManager bean is missing on the listener thread.
My goal is to have let the Spring Boot/Kafka framework to create the mean for each consumed Kafka events just as it's for the web requests.
I have no idea how I could achive that, could someone help me ?
Thank you so much,
Have a nice day!

#Gary Russel
That's true and false at the same time, meantime I succeed to find a solution, create the below class:
public class KafkaRequestScopeAttributes implements RequestAttributes {
private Map<String, Object> requestAttributeMap = new HashMap<>();
#Override
public Object getAttribute(String name, int scope) {
if (scope == RequestAttributes.SCOPE_REQUEST) {
return this.requestAttributeMap.get(name);
}
return null;
}
#Override
public void setAttribute(String name, Object value, int scope) {
if (scope == RequestAttributes.SCOPE_REQUEST) {
this.requestAttributeMap.put(name, value);
}
}
#Override
public void removeAttribute(String name, int scope) {
if (scope == RequestAttributes.SCOPE_REQUEST) {
this.requestAttributeMap.remove(name);
}
}
#Override
public String[] getAttributeNames(int scope) {
if (scope == RequestAttributes.SCOPE_REQUEST) {
return this.requestAttributeMap.keySet().toArray(new String[0]);
}
return new String[0];
}
#Override
public void registerDestructionCallback(String name, Runnable callback, int scope) {
// Not Supported
}
#Override
public Object resolveReference(String key) {
// Not supported
return null;
}
#Override
public String getSessionId() {
return null;
}
#Override
public Object getSessionMutex() {
return null;
}
}
then add the following two lines into your KafkaListener method's start and end:
RequestContextHolder.setRequestAttributes(new KafkaRequestScopeAttributes());
RequestContextHolder.resetRequestAttributes();
By doing that you can force to create the REQUEST_SCOPE in a Kafka Listener.

Request Scope is for web applications only; it can't be used with Kafka consumers.

Related

Spring Boot and Spring Data with Cassandra: Continue on failed database connection

I use Spring Boot and Spring Data with Cassandra. On application startup spring establishes a connection to the database to setup the schema and initialize spring data repositories. If the database is not available, the application won't start.
I want, that the application just logs an error and starts. Of course, I can't use the repositories anymore, but other services (rest controllers etc), which are independent from the database should work. It would also be nice to see in actuator healthcheck, that cassandra is down.
For JDBC, there is a spring.datasource.continue-on-error property. I couldn't find something similar for Cassandra.
I also tried to create a custom cassandra configuration and trying to catch Exception on CqlSession creation, but I couldn't achieve the desired behavior.
EDIT: As suggested by #adutra, I tried to set advanced.reconnect-on-init, Application tries to establish the connection, but the application is not fully initialized (e.g. REST controller are not reachable)
#Configuration
public class CustomCassandraConfiguration extends CassandraAutoConfiguration {
#Bean
public DriverConfigLoaderBuilderCustomizer driverConfigLoaderBuilderCustomizer() {
return builder -> builder.withBoolean(DefaultDriverOption.RECONNECT_ON_INIT, true);
}
}
EDIT2: I have now working example (application starts, custom health check for cassandra), but if feels pretty ugly:
CustomCassandraAutoConfiguration
#Configuration
public class CustomCassandraAutoConfiguration extends CassandraAutoConfiguration {
private final Logger logger = LoggerFactory.getLogger(getClass());
#Override
#Bean
public CqlSession cassandraSession(CqlSessionBuilder cqlSessionBuilder) {
try {
return super.cassandraSession(cqlSessionBuilder);
} catch (AllNodesFailedException e) {
logger.error("Failed to establish the database connection", e);
}
return new DatabaseNotConnectedFakeCqlSession();
}
#Bean
public CassandraReactiveHealthIndicator cassandraHealthIndicator(ReactiveCassandraOperations r, CqlSession session) {
if (session instanceof DatabaseNotConnectedFakeCqlSession) {
return new CassandraReactiveHealthIndicator(r) {
#Override
protected Mono<Health> doHealthCheck(Health.Builder builder) {
return Mono.just(builder.down().withDetail("connection", "was not available on startup").build());
}
};
}
return new CassandraReactiveHealthIndicator(r);
}
}
CustomCassandraDataAutoConfiguration
#Configuration
public class CustomCassandraDataAutoConfiguration extends CassandraDataAutoConfiguration {
public CustomCassandraDataAutoConfiguration(CqlSession session) {
super(session);
}
#Bean
public SessionFactoryFactoryBean cassandraSessionFactory(CqlSession session, Environment environment, CassandraConverter converter) {
SessionFactoryFactoryBean sessionFactoryFactoryBean = super.cassandraSessionFactory(environment, converter);
// Disable schema action if database is not available
if (session instanceof DatabaseNotConnectedFakeCqlSession) {
sessionFactoryFactoryBean.setSchemaAction(SchemaAction.NONE);
}
return sessionFactoryFactoryBean;
}
}
DatabaseNotConnectedFakeCqlSession (Fake session implementation)
public class DatabaseNotConnectedFakeCqlSession implements CqlSession {
#Override
public String getName() {
return null;
}
#Override
public Metadata getMetadata() {
return null;
}
#Override
public boolean isSchemaMetadataEnabled() {
return false;
}
#Override
public CompletionStage<Metadata> setSchemaMetadataEnabled( Boolean newValue) {
return null;
}
#Override
public CompletionStage<Metadata> refreshSchemaAsync() {
return null;
}
#Override
public CompletionStage<Boolean> checkSchemaAgreementAsync() {
return null;
}
#Override
public DriverContext getContext() {
return new DefaultDriverContext(new DefaultDriverConfigLoader(), ProgrammaticArguments.builder().build());
}
#Override
public Optional<CqlIdentifier> getKeyspace() {
return Optional.empty();
}
#Override
public Optional<Metrics> getMetrics() {
return Optional.empty();
}
#Override
public <RequestT extends Request, ResultT> ResultT execute( RequestT request, GenericType<ResultT> resultType) {
return null;
}
#Override
public CompletionStage<Void> closeFuture() {
return null;
}
#Override
public CompletionStage<Void> closeAsync() {
return null;
}
#Override
public CompletionStage<Void> forceCloseAsync() {
return null;
}
#Override
public Metadata refreshSchema() {
return null;
}
}
Any suggestions?
You can set the option datastax-java-driver.advanced.reconnect-on-init to true to achieve the effect you want. Its usage is explained in the configuration reference page in the driver docs:
Whether to schedule reconnection attempts if all contact points are unreachable on the first initialization attempt.
If this is true, the driver will retry according to the reconnection policy. The SessionBuilder.build() call - or the future returned by SessionBuilder.buildAsync() - won't complete until a contact point has been reached. If this is false and no contact points are available, the driver will fail with an AllNodesFailedException.
However be careful: with this option set to true, as stated above, any component trying to access a CqlSession bean, even if the session bean is lazy, will block until the driver is able to connect, and might block forever if the contact points are wrong.
If that's not acceptable for you, I would suggest that you wrap the CqlSession bean in another bean that will check if the future returned by SessionBuilder.buildAsync() is done or not, and either block, throw or return null, depending on the caller's expectations.
[EDIT] I've reached out internally to the DataStax Drivers team last night and adutra has responded so I'm withdrawing my response.

No thread-bound request found with Spring when Kafka receive a message

I 'm getting this error from my service
jvm org.hibernate.internal.ExceptionMapperStandardImpl {"#trace_info":"[availability-psql,eba16d49e23479cc,675789f41e0dda5b,eba16d49e23479cc,false]", "#message": "HHH000346: Error during managed flush [Error creating bean with name 'scopedTarget.infoUser': Scope 'request' is not active for the current thread; consider defining a scoped proxy for this bean if you intend to refer to it from a singleton; nested exception is java.lang.IllegalStateException: No thread-bound request found: Are you referring to request attributes outside of an actual web request, or processing a request outside of the originally receiving thread? If you are actually operating within a web request and still receive this message, your code is probably running outside of DispatcherServlet/DispatcherPortlet: In this case, use RequestContextListener or RequestContextFilter to expose the current request.]
This is because of I have a bean of scope #ScopeRequest. This problem show up when a new message from kafka is received and I try to update my data base with spring data. If I remove my #Transactional I don't have any problem to save the data.
#KafkaListener(topics = "#{kafkaMastersConfig.topics}", containerFactory = "mastersContainerFactory")
#Transactional
#Authorized
public void consumeWrapperMasterChangeEvent(#Payload String payload,
#Header(KafkaHeaders.RECEIVED_TOPIC) String topic, #Nullable #Header(AUTHORIZATION) String authorization) throws IOException {
try {
log.info("Received change event in masters: '{}'", payload);
RequestAttributes context = RequestContextHolder.currentRequestAttributes();
RequestContextHolder.setRequestAttributes(context);
changeProcessorFactory.getEntityChangeProcessor(getEntityFromTopic(topic)).processChange(payload);
} catch ( Exception e ) {
log.error("Error proccesing message {} ", e.getMessage());
} finally {
RequestContextHolder.resetRequestAttributes();
}
}
And here is the bean:
#RequestScope
#Component
#NoArgsConstructor
#Getter
#Setter
public class InfoUser {
private DecodedJWT jwt;
public String getCurrentUser() {
if (jwt == null) {
return null;
}
return jwt.getSubject();
}
public String getAuthorizationBearer() {
if (jwt == null) {
return null;
}
return jwt.getToken();
}
}
And this class:
public class CustomRequestScopeAttr implements RequestAttributes {
private Map<String, Object> requestAttributeMap = new HashMap<>();
#Override
public Object getAttribute(String name, int scope) {
if (scope == RequestAttributes.SCOPE_REQUEST) {
return this.requestAttributeMap.get(name);
}
return null;
}
#Override
public void setAttribute(String name, Object value, int scope) {
if (scope == RequestAttributes.SCOPE_REQUEST) {
this.requestAttributeMap.put(name, value);
}
}
#Override
public void removeAttribute(String name, int scope) {
if (scope == RequestAttributes.SCOPE_REQUEST) {
this.requestAttributeMap.remove(name);
}
}
#Override
public String[] getAttributeNames(int scope) {
if (scope == RequestAttributes.SCOPE_REQUEST) {
return this.requestAttributeMap.keySet().toArray(new String[0]);
}
return new String[0];
}
#Override
public void registerDestructionCallback(String name, Runnable callback, int scope) {
// Not Supported
}
#Override
public Object resolveReference(String key) {
// Not supported
return null;
}
#Override
public String getSessionId() {
return null;
}
#Override
public Object getSessionMutex() {
return null;
}
}
And futhermore I have an aspect class to save the authorization token:
#Aspect
#Component
#RequiredArgsConstructor
public class AuthorizationAspect {
private final AuthorizationDecoder authorizationDecoder;
private final ApplicationContext applicationContext;
#Around("#annotation(Authorized)")
public Object setInfoUser(ProceedingJoinPoint joinPoint) throws Throwable {
try {
String[] parameterNames = ((CodeSignature) joinPoint.getSignature()).getParameterNames();
Object[] args = joinPoint.getArgs();
Map<String, Object> arguments = new HashMap<>();
for (int i = 0; i < args.length; i++) {
if (null != args[i]) {
arguments.put(parameterNames[i], args[i]);
}
}
Object authorization = arguments.get("authorization");
RequestContextHolder.setRequestAttributes(new CustomRequestScopeAttr());
InfoUser infoUser = applicationContext.getBean(InfoUser.class);
infoUser.setJwt(authorizationDecoder.decodeToken((String) authorization));
return joinPoint.proceed();
} finally {
RequestContextHolder.resetRequestAttributes();
}
}
And the last class is trying to save de info:
#RequiredArgsConstructor
public class RoomChangeMaster implements ChangeMaster<Room> {
private final TimetableRepository timetableRepository;
private final AvailabilityRepository availabilityRepository;
#Override
public void processChange(Room entity, ActionEnum action) {
if (action == ActionEnum.updated) {
List<Timetable> timetables = (List<Timetable>) timetableRepository.findByRoomId(entity.getId());
Room room = timetables.get(0).getRoom();
room.setDescription(entity.getDescription());
room.setCode(entity.getCode());
timetables.forEach(timetable -> {
timetable.setRoom(room);
timetableRepository.save(timetable);
});
availabilityRepository
.updateAvailabilityRoomByRoomId(room, entity.getId());
} else {
throw new IllegalStateException("Unexpected value: " + action);
}
}
}
I have spent a lot of time finding out the problem, but so far, I was not able to know the problem. Any idea will be appreciate.
Thank you
RequestContextHolder is for Spring-MVC - it is for a Web request only and is populated with information from an HTTP request.
/**
* Holder class to expose the web request in the form of a thread-bound
* {#link RequestAttributes} object. The request will be inherited
* by any child threads spawned by the current thread if the
* {#code inheritable} flag is set to {#code true}.
*
...
There is no equivalent for listener containers (of any type) because there is no "incoming request".
Looks like your hibernate code is tightly tied to the web.
If you are trying to reuse existing code you need to decouple it and use some other technique to pass information between the layers (e.g. a custom equivalent of RequestContextHolder).
Finally, I have solved it changing the hiberante method save by saveAndFlush

Spring `#Autowire` field is `null` eventhough it works fine in other classes

Spring #Autowire field is null even though it works fine in other classes successfully.
public class SendRunner implements Runnable {
private String senderAddress;
#Autowired
private SubscriberService subscriberService;
public SendRunner(String senderAddress) {
this.senderAddress = senderAddress;
}
#Override
public void run() {
sendRequest();
}
private void sendRequest() {
try {
HashMap<String, String> dataMap = new HashMap<>();
dataMap.put("subscriberId", senderAddress);
HttpEntity<?> entity = new HttpEntity<Object>(dataMap, httpHeaders);
Subscriber subscriber = subscriberService.getSubscriberByMsisdn(senderAddress);
} catch (Exception e) {
logger.error("Error occurred while trying to send api request", e);
}
}
Also this class is managed as a bean in the dispatcher servlet :
<bean id="SendRunner" class="sms.dating.messenger.connector.SendRunner">
</bean>
In here i'm getting a null pointer exception for subscriberService. What would be the possible reason for this? Thanks in advance.
Can you please try with below code snippet
#Configuration
public class Someclass{
#Autowired
private SubscriberService subscriberService;
Thread subscriberThread = new Thread() {
#Override
public void run() {
try {
HashMap<String, String> dataMap = new HashMap<>();
dataMap.put("subscriberId", senderAddress);
HttpEntity<?> entity = new HttpEntity<Object>(dataMap, httpHeaders);
Subscriber subscriber = subscriberService.getSubscriberByMsisdn(senderAddress);
} catch (Exception e) {
logger.error("Error occurred while trying to send api request", e);
}
}
};
}
Can you please annotate your SendRunner class with #Component or #Service and include the SendRunner package in componentscanpackage
Your bean not in Spring Managed context, below can be the reasons.
Package sms.dating.messenger.connector not in Component scan.
You are moving out of the Spring context by creating an object with new (see below),
this way you will not get the autowired fields.
SendRunner sendRunner = new SendRunner () ,
sendRunner.sendRequest();
Just check how I implement. Hope this will help.
#RestController
public class RestRequest {
#Autowired
SendRunner sendRunner;
#RequestMapping("/api")
public void Uri() {
sendRunner.start();
}
}
SendRunner class
#Service
public class SendRunner extends Thread{
#Autowired
private SubscriberService subscriberService;
#Override
public void run() {
SendRequest();
}
private void SendRequest() {
System.out.println("Object is " + subscriberService);
String senderAddress = "address";
subscriberService.getSubscriberByMsisdn(senderAddress);
}
}
Below are the logs printed when I hit the REST api.
Object is com.example.demo.SubscriberService#40f33492

Spring Autowired Shared Queue NullPointerException

I'm using Spring for the first time and am trying to implement a shared queue wherein a Kafka listener puts messages on the shared queue, and a ThreadManager that will eventually do something multithreaded with the items it takes off the shared queue. Here is my current implementation:
The Listener:
#Component
public class Listener {
#Autowired
private QueueConfig queueConfig;
private ExecutorService executorService;
private List<Future> futuresThread1 = new ArrayList<>();
public Listener() {
Properties appProps = new AppProperties().get();
this.executorService = Executors.newFixedThreadPool(Integer.parseInt(appProps.getProperty("listenerThreads")));
}
//TODO: how can I pass an approp into this annotation?
#KafkaListener(id = "id0", topics = "bose.cdp.ingest.marge.boseaccount.normalized")
public void listener(ConsumerRecord<?, ?> record) throws InterruptedException, ExecutionException
{
futuresThread1.add(executorService.submit(new Runnable() {
#Override public void run() {
try{
queueConfig.blockingQueue().put(record);
// System.out.println(queueConfig.blockingQueue().take());
} catch (Exception e){
System.out.print(e.toString());
}
}
}));
}
}
The Queue:
#Configuration
public class QueueConfig {
private Properties appProps = new AppProperties().get();
#Bean
public BlockingQueue<ConsumerRecord> blockingQueue() {
return new ArrayBlockingQueue<>(
Integer.parseInt(appProps.getProperty("blockingQueueSize"))
);
}
}
The ThreadManager:
#Component
public class ThreadManager {
#Autowired
private QueueConfig queueConfig;
private int threads;
public ThreadManager() {
Properties appProps = new AppProperties().get();
this.threads = Integer.parseInt(appProps.getProperty("threadManagerThreads"));
}
public void run() throws InterruptedException {
ExecutorService executorService = Executors.newFixedThreadPool(threads);
try {
while (true){
queueConfig.blockingQueue().take();
}
} catch (Exception e){
System.out.print(e.toString());
executorService.shutdownNow();
executorService.awaitTermination(1, TimeUnit.SECONDS);
}
}
}
Lastly, the main thread where everything is started from:
#SpringBootApplication
public class SourceAccountListenerApp {
public static void main(String[] args) {
SpringApplication.run(SourceAccountListenerApp.class, args);
ThreadManager threadManager = new ThreadManager();
try{
threadManager.run();
} catch (Exception e) {
System.out.println(e.toString());
}
}
}
The problem
I can tell when running this in the debugger that the Listener is adding things to the queue. When the ThreadManager takes off the shared queue, it tells me the queue is null and I get an NPE. It seems like autowiring isn't working to connect the queue the listener is using to the ThreadManager. Any help appreciated.
This is the problem:
ThreadManager threadManager = new ThreadManager();
Since you are creating the instance manually, you cannot use the DI provided by Spring.
One simple solution is implement a CommandLineRunner, that will be executed after the complete SourceAccountListenerApp initialization:
#SpringBootApplication
public class SourceAccountListenerApp {
public static void main(String[] args) {
SpringApplication.run(SourceAccountListenerApp.class, args);
}
// Create the CommandLineRunner Bean and inject ThreadManager
#Bean
CommandLineRunner runner(ThreadManager manager){
return args -> {
manager.run();
};
}
}
You use SpringĀ“s programatic, so called 'JavaConfig', way of setting up Spring beans (classes annotated with #Configuration with methods annotated with #Bean). Usually at application startup Spring will call those #Bean methods under the hood and register them in it's application context (if scope is singleton - the default - this will happen only once!). No need to call those #Bean methods anywhere in your code directly... you must not, otherwise you will get a separate, fresh instance that possibly is not fully configured!
Instead, you need to inject the BlockingQueue<ConsumerRecord> that you 'configured' in your QueueConfig.blockingQueue() method into your ThreadManager. Since the queue seems to be a mandatory dependency for the ThreadManager to work, I'd let Spring inject it via constructor:
#Component
public class ThreadManager {
private int threads;
// add instance var for queue...
private BlockingQueue<ConsumerRecord> blockingQueue;
// you could add #Autowired annotation to BlockingQueue param,
// but I believe it's not mandatory...
public ThreadManager(BlockingQueue<ConsumerRecord> blockingQueue) {
Properties appProps = new AppProperties().get();
this.threads = Integer.parseInt(appProps.getProperty("threadManagerThreads"));
this.blockingQueue = blockingQueue;
}
public void run() throws InterruptedException {
ExecutorService executorService = Executors.newFixedThreadPool(threads);
try {
while (true){
this.blockingQueue.take();
}
} catch (Exception e){
System.out.print(e.toString());
executorService.shutdownNow();
executorService.awaitTermination(1, TimeUnit.SECONDS);
}
}
}
Just to clarify one more thing: by default the method name of a #Bean method is used by Spring to assign this bean a unique ID (method name == bean id). So your method is called blockingQueue, means your BlockingQueue<ConsumerRecord> instance will also be registered with id blockingQueue in application context. The new constructor parameter is also named blockingQueue and it's type matches BlockingQueue<ConsumerRecord>. Simplified, that's one way Spring looks up and injects/wires dependencies.

Issues integration testing a Spring MVC controller method that calls an async Spring service method

I have the following Spring MVC controller method:
#RequestMapping(value = "/sendPasswordReset", method = RequestMethod.POST, produces = "text/html")
public String sendPasswordResetInformation(#ModelAttribute #Validated({ ValidationGroups.PasswordReset.class }) PasswordResetInfo passwordResetInfo,
BindingResult bindingResult, Model model, final RedirectAttributes redirectAttributes, Locale locale) throws InterruptedException, ExecutionException {
if (preferenceService.isEmailAvailable(passwordResetInfo.getEmail())) {
bindingResult.rejectValue("email", "controller.preference.email_not_in_system");
}
if (bindingResult.hasErrors()) {
model.addAttribute("passwordResetInfo", passwordResetInfo);
return "preference/sendPasswordReset";
}
redirectAttributes.addFlashAttribute("flashMessage", messageSource.getMessage("controller.preference.password_reset_info_sent", null, locale));
Future<Void> future = preferenceService.sendPasswordResetInfo(passwordResetInfo.getEmail());//.get();//TODO is ".get()" ugly?
future.get();//NPE HERE!!
return "redirect:/preference/sendPasswordReset";
}
Here is the implementation of sendPasswordResetInfo:
#Async
#Override
public Future<Void> sendPasswordResetInfo(String email) {
Assert.hasText(email);
Member member = memberRepository.findByEmail(email);
try {
mailerService.doMailPasswordResetInfo(member);
return new AsyncResult<Void>(null);
} catch (MessagingException | MailSendException e) {
log.error("MessagingException | MailSendException", e);
throw new MailerException("MessagingException | MailSendException");
}
}
Here is how I am trying to integration test the controller method:
#Test
public void sendPasswordResetShouldHaveNormalInteractions() throws Exception {
when(preferenceService.isEmailAvailable(anyString())).thenReturn(Boolean.FALSE);
mockMvc.perform(post("/preference/sendPasswordReset")//
.param("email", VALID_EMAIL))//
.andExpect(redirectedUrl("/preference/sendPasswordReset"))//
.andExpect(flash().attributeExists("flashMessage"))//
.andExpect(flash().attributeCount(1));
verify(preferenceService).sendPasswordResetInfo(eq(VALID_EMAIL));
reset(preferenceService);
}
I systematically get a NullPointerException (in tests) in the controller method because the future object is null here:
future.get()
However the controller method runs fine when I use the app (outside of tests).
I have tried using a sync task executor as follows (to no avail):
#Profile(Profiles.TEST)
#Configuration
#EnableAsync
public class FakeAsyncConfiguration implements AsyncConfigurer {
#Override
public Executor getAsyncExecutor() {
SyncTaskExecutor taskExecutor = new SyncTaskExecutor();
return taskExecutor;
}
#Override
public AsyncUncaughtExceptionHandler getAsyncUncaughtExceptionHandler() {
return new SimpleAsyncUncaughtExceptionHandler();
}
}
My questions are:
Why is the Future object always null during the integration tests and
How can I ensure it is not null during my integration tests?

Resources