I have a Spring Cloud Stream application that processes (credit card) events -- some of which are processed synchronously and some asynchronously. I came up with roughly the following in Kotlin:
spring-cloud-starter-stream-rabbit = 3.1.0
#Service
class CardEventProcessor(
private val streamBridge: StreamBridge,
) : Consumer<AsyncCardEvent> {
fun process(cardEvent: SyncCardEvent): Result { return businessLogic() }
fun processAsynchronously(cardEvent: AsyncCardEvent) {
streamBridge.send("cardEventProcessor-out-0", cardEvent)
}
override fun accept(cardEvent: AsyncCardEvent) { businessLogic() }
}
and have configured it like so:
rabbitmq: ...
cloud:
stream:
bindings:
cardEventProcessor-in-0:
destination: cardevents
group: CardEventProcessor
cardEventProcessor-out-0:
destination: cardevents
It all seems to work fine, except in integration tests the processing fails after the second async card event. I was able to debug / reduce the issue down to two handlers being registered in UnicastingDispatcher, which has a round robin strategy: one for TestChannelBinder and another for OutputDestination$lamdba.
This is what my integration test class looks like:
#SpringBootTest
#Transactional
#AutoConfigureMockMvc
#AutoConfigureEmbeddedDatabase
#Import(TestChannelBinderConfiguration::class)
class IntegrationTests {
#Test
fun `Use case one`() {
sendFirstAsyncRequest() // processed correctly in CardEventProcessor.accept()
sendSecondAsyncRequest() // message never arrives in CardEventProcessor.accept()
}
}
I was following the Testing section in Spring cloud stream docs and can't figure out what I'm missing to get this working. The example there is a Function<> not a Consumer<> and I produce within the same #Service class as the consumer (because the queue is just an implementation detail to solve async, not the typical use case of queues between micro-services) but as far as I understand that should work, and does in fact work when not running as integration test.
I saw Disable Spring Cloud Stream Rabbit for tests but didn't want to depend on the deprecated spring-cloud-stream-test-support and the other two suggestions didn't work either. Any ideas?
Related
I was reading "Spring Microservices In Action (2021)" because I wanted to brush up on Microservices.
Now with Spring Boot 3 a few things changed. In the book, an easy example of how to push messages to a topic and how to consume messages to a topic were presented.
The Problem is: The examples presented do just not work with Spring Boot 3. Sending Messages from a Spring Boot 2 Project works. The underlying project can be found here:
https://github.com/ihuaylupo/manning-smia/tree/master/chapter10
Example 1 (organization-service):
Consider this Config:
spring.cloud.stream.bindings.output.destination=orgChangeTopic
spring.cloud.stream.bindings.output.content-type=application/json
spring.cloud.stream.kafka.binder.zkNodes=kafka #kafka is used as a network alias in docker-compose
spring.cloud.stream.kafka.binder.brokers=kafka
And this Component(Class) which can is injected in a service in this project
#Component
public class SimpleSourceBean {
private Source source;
private static final Logger logger = LoggerFactory.getLogger(SimpleSourceBean.class);
#Autowired
public SimpleSourceBean(Source source){
this.source = source;
}
public void publishOrganizationChange(String action, String organizationId){
logger.debug("Sending Kafka message {} for Organization Id: {}", action, organizationId);
OrganizationChangeModel change = new OrganizationChangeModel(
OrganizationChangeModel.class.getTypeName(),
action,
organizationId,
UserContext.getCorrelationId());
source.output().send(MessageBuilder.withPayload(change).build());
}
}
This code fires a message to the topic (destination) orgChangeTopic. The way I understand it, the firsttime a message is fired, the topic is created.
Question 1: How do I do this Spring Boot 3? Config-Wise and "Code-Wise"?
Example 2:
Consider this config:
spring.cloud.stream.bindings.input.destination=orgChangeTopic
spring.cloud.stream.bindings.input.content-type=application/json
spring.cloud.stream.bindings.input.group=licensingGroup
spring.cloud.stream.kafka.binder.zkNodes=kafka
spring.cloud.stream.kafka.binder.brokers=kafka
And this code:
#SpringBootApplication
#RefreshScope
#EnableDiscoveryClient
#EnableFeignClients
#EnableEurekaClient
#EnableBinding(Sink.class)
public class LicenseServiceApplication {
public static void main(String[] args) {
SpringApplication.run(LicenseServiceApplication.class, args);
}
#StreamListener(Sink.INPUT)
public void loggerSink(OrganizationChangeModel orgChange) {
log.info("Received an {} event for organization id {}",
orgChange.getAction(), orgChange.getOrganizationId());
}
What this method is supposed to do is to fire whenever a message is fired in orgChangeTopic, we want the method loggerSink to fire.
How do I do this in Spring Boot 3?
In Spring Cloud Stream 4.0.0 (the version used if you are using Boot 3), a few things are removed - such as the EnableBinding, StreamListener, etc. We deprecated them before in 3.x and finally removed them in the 4.0.0 version. The annotation-based programming model is removed in favor of the functional programming style enabled through the Spring Cloud Function project. You essentially express your business logic as java.util.function.Funciton|Consumer|Supplier etc. for a processor, sink, and source, respectively. For ad-hoc source situations, as in your first example, Spring Cloud Stream provides a StreamBridge API for custom sends.
Your example #1 can be re-written like this:
#Component
public class SimpleSourceBean {
#Autowired
StreamBridge streamBridge
public void publishOrganizationChange(String action, String organizationId){
logger.debug("Sending Kafka message {} for Organization Id: {}", action, organizationId);
OrganizationChangeModel change = new OrganizationChangeModel(
OrganizationChangeModel.class.getTypeName(),
action,
organizationId,
UserContext.getCorrelationId());
streamBridge.send("output-out-0", MessageBuilder.withPayload(change).build());
}
}
Config
spring.cloud.stream.bindings.output-out-0.destination=orgChangeTopic
spring.cloud.stream.kafka.binder.brokers=kafka
Just so you know, you no longer need that zkNode property. Neither the content type since the framework auto-converts that for you.
StreamBridge send takes a binding name and the payload. The binding name can be anything - but for consistency reasons, we used output-out-0 here. Please read the reference docs for more context around the reasoning for this binding name.
If you have a simple source that runs on a timer, you can express this simply as a supplier as below (instead of using a StreamBrdige).
#Bean
public Supplier<OrganizationChangeModel> ouput() {
return () -> {
// return the payload
};
}
spring.cloud.function.definition=output
spring.cloud.bindings.output-out-0.destination=...
Example #2
#Bean
public Consumer<OrganizationChangeModel> loggerSink() {
return model -> {
log.info("Received an {} event for organization id {}",
orgChange.getAction(), orgChange.getOrganizationId());
};
}
Config:
spring.cloud.function.definition=loggerSink
spring.cloud.stream.bindings.loggerSink-in-0.destination=orgChangeTopic
spring.cloud.stream.bindings.loggerSinnk-in-0.group=licensingGroup
spring.cloud.stream.kafka.binder.brokers=kafka
If you want the input/output binding names to be specifically input or output rather than with in-0, out-0 etc., there are ways to make that happen. Details for this are in the reference docs.
I'm using Quarkus to build a simple Kafka message processor. The processor consumes messages from a Kafka topic and produces messages to another topic; it's pretty straightforward.
I have quarkus-opentelemetry-exporter-otlp enabled in my application, and I was hoping to see traces generated by smallrye-reactive-messaging-kafka and my own traces correctly nested. This doesn't seem to be working, though.
My processor looks like this:
#ApplicationScoped
class MessageProcessor #Inject constructor(private val service: MyService) {
#Blocking
#Incoming("requests")
#Outgoing("events")
#WithSpan("email.process")
fun process(incoming: MessageEnvelope): MessageEnvelope {
val span = Span.current()
.setAttribute(Keys.TYPE, incoming.type)
.setAttribute(Keys.TO, incoming.to)
return service.send(request)
}
}
In Jaeger, I can see the spans produced by Smallrye Reactive Messaging and my own email.process span, but they're not nested as I expected.
What do I need to do to propagate the trace context correctly?
I'm using Kafka embedded broker with spring boot and junit 5.I have been able to wire up successfully and see that the embedded broker is running.
In my setup method I pump in a few messages to the queue that my actual code listens on
#BeforeAll
public void setup() {
// code to play down some messages to topic X
}
My consumer/listener is never trigerred despite there being no errors encountered in the setup method
My Consumer is setup like
class Consumer() {
#KafkaListener(topics="X",
groupId ="...",
containerFactory="my-container-factory"
)
public void consume(ConsumerRecord<String,byte[] rec) {
//logic to handle
logger.info("Print rec : "+rec)
}
}
else where I've set up my ListenerContainerFactory with a name like
#Bean(name="my-container-factory")
public KafkaContainerListenerFactory<String,byte[]> factory() {
}
What could be wrong with this?My assertions in the test case fail and additionally I don't see my log statements that should be printed if my consume method were ever called.
I've a feeling,that auto configuration due to #SpringBootTest and #EmbeddedKafka is setting up some other listener container factory and so maybe my #KafkaListener annotation is wrong.
I know,its a bit vague but could you please tell me what/where to look at?If I run as a #SpringBootApplication my Consumer is pulling in messages from the actual queue.So no problems with my actual app.Its the test that's not executing as per expectation.
Please help.
Edit 1:
I have spring.kafka.consumer.auto-offset-reset=earliest set in my yml file.
maybe someone has an idea to my following problem:
I am currently on a project, where i want to use the AWS SQS with Spring Cloud integration. For the receiver part i want to provide a API, where a user can register a "message handler" on a queue, which is an interface and will contain the user's business logic, e.g.
MyAwsSqsReceiver receiver = new MyAwsSqsReceiver();
receiver.register("a-queue-name", new MessageHandler(){
#Override
public void handle(String message){
//... business logic for the received message
}
});
I found examples, e.g.
https://codemason.me/2016/03/12/amazon-aws-sqs-with-spring-cloud/
and read the docu
http://cloud.spring.io/spring-cloud-aws/spring-cloud-aws.html#_sqs_support
But the only thing i found there to "connect" a functionality for processing a incoming message is a annotation on a method, e.g. #SqsListener or #MessageMapping.
These annotations are fixed to a certain queue-name, though. So now i am at a loss, how to dynamically "connect" my provided "MessageHandler" (from my API) to the incoming message for the specified queuename.
In the Config the example there is a SimpleMessageListenerContainer, which gets a QueueMessageHandler set, but this QueueMessageHandler does not seem
to be the right place to set my handler or to override its methods and provide my own subclass of QueueMessageHandler.
I already did something like this with the Spring Amqp integration and RabbitMq and thought, that it would be also similar here with AWS SQS.
Does anyone have an idea, how to accomplish this?
thx + bye,
Ximon
EDIT:
I found, that Spring JMS could actually do that, e.g. www.javacodegeeks.com/2016/02/aws-sqs-spring-jms-integration.html. Does anybody know, what consequences using JMS protocol has here, good or bad?
I am facing the same issue.
I am trying to go in an unusual way where I set up an Aws client bean at build time and then instead of using sqslistener annotation to consume from the specific queue I use the scheduled annotation which I can programmatically pool (each 10 secs in my case) from which queue I want to consume.
I did the example that iterates over queues defined in properties and then consumes from each one.
Client Bean:
#Bean
#Primary
public AmazonSQSAsync awsSqsClient() {
return AmazonSQSAsyncClientBuilder
.standard()
.withRegion(Regions.EU_WEST_1.getName())
.build();
}
Consumer:
// injected in the constructor
private final AmazonSQSAsync awsSqsClient;
#Scheduled(fixedDelay = 10000)
public void pool() {
properties.getSqsQueues()
.forEach(queue -> {
val receiveMessageRequest = new ReceiveMessageRequest(queue)
.withWaitTimeSeconds(10)
.withMaxNumberOfMessages(10);
// reading the messages
val result = awsSqsClient.receiveMessage(receiveMessageRequest);
val sqsMessages = result.getMessages();
log.info("Received Message on queue {}: message = {}", queue, sqsMessages.toString());
// deleting the messages
sqsMessages.forEach(message -> {
val deleteMessageRequest = new DeleteMessageRequest(queue, message.getReceiptHandle());
awsSqsClient.deleteMessage(deleteMessageRequest);
});
});
}
Just to clarify, in my case, I need multiple queues, one for each tenant, with the queue URL for each one passed in a property file. Of course, in your case, you could get the queue names from another source, maybe a ThreadLocal which has the queues you have created in runtime.
If you wish, you can also try the JMS approach where you create message consumers and add a listener to each one you wish (See the doc Aws Jms documentation).
When we do Spring and SQS we use the spring-cloud-starter-aws-messaging.
Then just create a Listener class
#Component
public class MyListener {
#SQSListener(value="myqueue")
public void listen(MyMessageType message) {
//process the message
}
}
I'm trying to configure a simple Spring Cloud Stream application with RabbitMQ. The code I use is mostly taken from spring-cloud-stream-samples.
I have an entry point:
#SpringBootApplication
public class DemoApplication {
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
}
and a simple messages producer from the example:
#EnableBinding(Source.class)
public class SourceModuleDefinition {
private String format = "yyyy-MM-dd HH:mm:ss";
#Bean
#InboundChannelAdapter(value = Source.OUTPUT, poller = #Poller(fixedDelay = "${fixedDelay}", maxMessagesPerPoll = "1"))
public MessageSource<String> timerMessageSource() {
return () -> new GenericMessage<>(new SimpleDateFormat(this.format).format(new Date()));
}
}
Additionally, here is application.yml configuration:
fixedDelay: 5000
spring:
cloud:
stream:
bindings:
output:
destination: test
When I run the example, it connects to Rabbit and creates an exchange called test. But my problem is, it doesn't create a queue and binding automatically. I can see traffic going in Rabbit, but all my messages are then gone. While I need them to stay in some queue unless they are read by consumer.
Maybe I misunderstand something, but from all the topics I read, it seems like Spring Cloud Stream should create a queue and a binding automatically. If not, how do I configure it so my messages are persisted?
I'm using Spring Cloud Brixton.SR5 and Spring Boot 1.4.0.RELEASE.
A queue would be created as soon as you start a consumer application.
In the case of Rabbit MQ, where we have separate queues for each consumer group and we cannot know all groups beforehand, if you want to have the queues created automatically for consumer groups that are known in advance, you can use the requiredGroups property of the producers. This will ensure that messages are persisted until a consumer from that group is started.
See details here: http://docs.spring.io/spring-cloud-stream/docs/Brooklyn.BUILD-SNAPSHOT/reference/htmlsingle/#_producer_properties
You will need a consumer in order to have a queue created.
Here you can find an example of a producer and a consumer using rabbitMq:
http://ignaciosuay.com/how-to-implement-asyncronous-communication-between-microservices-using-spring-cloud-stream/