I've written integration tests for my Spring Boot Kafka (Consumer/Producer) service everything gone well. So I'm committing the offsets of my consumer manually after some processing.
I want to verify whether acknowledgment.acknowledge() was called in the consumer. Is is to possible verify?
Here is my method signature of the service:
#KafkaListener(topics = {TOPIC_XXX_V1}, containerFactory = "XXXListener")
private void consumer(#Payload XXXXRequestEvent xxxxRequestEvent, Acknowledgment acknowledgment) {
.....
// do something with the database
acknowledgment.acknowledge()
For the testing side I'm using #SpyBean for the Service and a MockBean for the database interaction. I want verify somehow whether in the test case the .acknowledge() was called. FYI: the .acknowledge() is a public abstract void method
As the Acknowledge instance is injected and created as part of Spring Kafka when consuming a message, I guess there is no way to use something like verify() of Mockito for this.
When writing a unit test instead you could pass a mocked version of Acknowledge here and then verify that this method was invoked. However, with a unit test, you can't test the actual consumption of a message (serialization, correct message handler, etc.).
So in your case, I would try to verify that your message was acknowledged by e.g. using Testcontainers to execute commands inside the Kafka container and ensuring that the already acknowledged message is not returned any more.
Another approach could be to create a Kafka client as part of your test and then try to consume messages from the same topic for X seconds and expect zero results. Awaitility might help you here.
Related
TLDR; How do you test a Reactive Function composition using the Test Binder?
I have a Spring Cloud Stream that uses Reactive Functions and I don't know how to test it. I don't see any official docs on how to do an Integration Test from input source to output destination binder.
In my specific case, I am connecting a Spring Integration flow using a Reactive Supplier and the IntegrationReactiveUtils.messageChannelToFlux() pattern. This works in a development environment - I can pull messages from RabbitMQ using the Spring Integration Flow and they enter the SCSt.
My SCSt has several function chained together, each one is reactive. They are composed like func1|func2|func3. I verified this works with a dev Rabbit (source) and Kafka (Destination).
I can't seem to figure out how to test this, and there doesn't seem to be any official documentation on testing a complete reactive stream. Right now I have code that roughly looks like this:
#Autowired
MessageChannel inputChannel;
#Autowired
private OutputDestination output;
#Test
void myTest() {
//omitted prep of var 'messageToSend'
this.inputChannel.send(messageToSend);
var outputMessage = output.receive(5000);
Assertions.assertNotNull(outputMessage.getPayload());
}
The error I receive is that output.receive(5000) returns null. I suspect a threading issue because I am not subscribing to the Flux and waiting for completion.
I have run a debugger in the Flux functions and see the message going all the way to the end with no errors or weirdness.
I figured this out actually. I had to specify the binder name. I had a test property spring.cloud.stream.bindings.processingStream set, which I thought made 2 new bindings (processingStream-in-0 and processingStream-out-0).
It turns out I had to set the binding name in the test code like output.receive(5000, "processingStream"), without the -out-0 suffix. I can now receive messages from the stream.
#Component
#RequiredArgsConstructor
public class EventListener {
private final EventProcessingService eventProcessingService;
#JmsListener(destination = "inputQueue", constainerFactory = "myContainerFactory)
public void receiveMessage(Message message) {
eventProcessingService.doSome(message).subscribe(); // return Mono<Void>
}
}
#Service
public class EventProcessingService {
public Mono<Void> doSome(Message message) {
//...
}
}
#Configuration
#RequiredArgsConstructor
public class MqIntegration {
private final ConnectionFactory connectionFactory;
#Bean
public Publisher<Message<String>> mqReactiveFlow() {
return IntegrationFlows
.from(Jms.messageDrivenChannelAdapter(this.connectionFactory)
.destination("testQueue"))
.channel(MessageChannels.queue())
.toReactivePublisher();
}
}
I have some webflux application which interacts with ibm mq and a JmsListener which listens for messages from the queue when a message is received EventProcessingService makes requests to other services depending on the messages.
I would like to know how I can create a JmsListener that works with reactive threads using Spring Integration. In other words I want to know if it is possible to create an Integration flow which will receive messages from the queue and call the EvenProcessingService when the messages are received so that it does not have a negative effect on the threads inside webflux application
I think we need to clean up some points in your question.
WebFlux is not a project by itself. It is Spring Framework module about Web on top of reactive server: https://docs.spring.io/spring-framework/docs/current/reference/html/web-reactive.html#spring-webflux
The #JmsListener is a part of another Spring Framework module - spring-jms. And there is nothing relevant to threads used by reactive server for WebFlux layer. https://docs.spring.io/spring-framework/docs/current/reference/html/integration.html#jms
Spring Integration is a separate project which implement EIP on top of Spring Framework dependency injection container. It indeed has its own WebFlux module for channel adapters on top of WebFlux API in Spring Framework: https://docs.spring.io/spring-integration/docs/current/reference/html/webflux.html#webflux. And it also has a JMS module on top of JMS module from Spring Framework: https://docs.spring.io/spring-integration/docs/current/reference/html/jms.html#jms. However there is nothing related to #JmsLisntener since its Jms.messageDrivenChannelAdapter() fully covers that functionality and from a big height it does it the same way - via MessageListenerContainer.
All of this is might not be relevant to the question, but it is better to have a clear context of what you are asking so we will feel that we are on the same page with you.
Now trying to answer to your concern.
As long as you don't deal with JMS from WebFlux layer (#RequestMapping or WebFlux.inboundGateway()), you don't effect those non-blocking thread. The JMS MessageListenerContainer spawns its own threads and perform pulling from the queue and message processing.
What you are explaining with your JMS configuration and service looks more like this:
#Bean
public IntegrationFlow mqReactiveFlow() {
return IntegrationFlows
.from(Jms.messageDrivenChannelAdapter(this.connectionFactory)
.destination("testQueue"))
.handle(this.eventProcessingService)
.nullChannel();
}
There is really no reason to shift messages just after JMS into a QueueChannel since JMS listening is already an async operation.
We need that nullChannel in the end of your flow just because your service method returns Mono and framework knows nothing what to do with that. Starting with version 5.4.3 the NullChannel is able to subscribe to the Publisher payload of the message produced to it.
You could have though a FluxMessageChannel in between to really simulate a back-pressure for JMS listener, but that won't make to much different for your next service.
I think you are going to have to bypass #JmsListener as that is registering an on message, which although asynchronous isn't going to be reactive. JMS is essentially blocking, so patching a reactive layer on top, is going to be just a patch.
You will need to use the Publisher that you have created to generate the back pressure. I think you are going to have to define and instantiate your own listener bean which does something on the lines of :
public Flux<String> mqReactiveListener() {
return Flux.from(mqReactiveFlow())
.map(Message::getPayload);
}
I have a problem with #TransactionalEventListener.
Given:
Kotlin 1.3.50
Spring Webflux
Spring Data
Hibernate 5
Postgres JDBC (not RDBC!)
What happens:
Request to controller was made.
Service did some job within method marked as #Transactional an event and published event.
Response returned to client
Event listener marked as #Async, #TransactionalEventListener and #Transactional doesn't kick in.
What I did:
I have debugged into service method which publishes an event to find out whether an actual transaction is in progress with the help of some static utility method I don't remember, which told me what I suspected - it is in progress.
Then I enabled debug output to see log saying: No transaction is in progress. Discarding event.
I mean, both publishing and consuming sides do have annotation #Transactional on them. No alterations to default params of annotations were made.
I have found similar situation in this question question but no one answered that.
Also, I have read an article at dzone but there seems to be no mentioning of my case.
Workaround:
I can avoid this by either replacing #TransactionalEventListener to #EventListener (alas, remove transaction bound processing) or by changing param fallbackExecute to true which effectively is the same as first solution.
Suspected parties:
Possibly absence of reactive transaction manager in an webflux environment could be the case? And some weird bug not accounting JpaTransactionManager?
I intend to write integration tests to check whether my listener/handlers work properly.
I am using Spring Boot 2.1.9.RELEASE with the according amqp dependency.
I have written a custom sender (publisher) and listener (receiver). When conducting the publishing test I am able to debug into my publisher but I am not able to debug into my receiver which I tried to annotate with #RabbitListener(queues = "myQueue") on method level and with #RabbitListener(queues = "myQueue") on class level in conjunction with #RabbitHandler on method level.
What do I have to prepare to run such integration tests successfully?
I have already googled a lot finding some suggestions using RabbitListenerTestHarness and so on but this never worked. For RabbitListenerTestHarness I always get the error that no such bean could be found.
I have a simple Spring Boot application which reads from Kafka and writes to Kafka. I wrote a SpringBootTest using an EmbeddedKafka to test all that.
The main problem is: Sometimes the test fails because the test sends the Kafka message too early. That way, the message is already written to Kafka before the Spring application (or its KafkaListener to be precise) is ready. Since the listener reads from the latest offset (I do not want to change any config for my test - except bootstrap.servers), it will not receive all messages in that test.
Does anyone have an idea how I could know inside the test, that the KafkaListener is ready to receive messages?
Only way I could think of is waiting until /health comes available but I have no idea whether I can be sure that this implies the KafkaListener to be ready at all.
Any help is greatly appreciated!
Best regards.
If you have a KafkaMessageListenerContainer instance, then it is very easy to use org.springframework.kafka.test.utils.ContainerTestUtils.waitForAssignment(Object container, int partitions).
https://docs.spring.io/spring-kafka/api/org/springframework/kafka/test/utils/ContainerTestUtils.html
e.g. calling ContainerTestUtils.waitForAssignment(container, 1); in your Test setup will block until the container has gotten 1 partition assigned.
So, I just read about #PostConstruct and it turns out that you can easily use this also within the test:
#PostConstruct
public void checkApplicationReady() {
applicationReady = true;
}
Now I added an #Before method to wait until that flag is set to true.
So far this seems to work very nicely!