How can i send a message with the new Spring Cloud Stream Kafka Functional Model?
The deprecated way looked like this.
public interface OutputTopic {
#Output("output")
MessageChannel output();
}
#Autowired
OutputTopic outputTopic;
public someMethod() {
outputTopic.output().send(MessageBuilder.build());
}
But how can i send a message in the functional style?
application.yml
spring:
cloud:
function:
definition: process
stream:
bindings:
process-out-0:
destination: output
binder: kafka
#Configuration
public class Configuration {
#Bean
Supplier<Message<String>> process() {
return () -> {
return MessageBuilder.withPayload("foo")
.setHeader(KafkaHeaders.MESSAGE_KEY, "bar".getBytes()).build();
};
}
I would Autowire a MessageChannel but there is no MessageChannel-Bean for process, process-out-0, output or something like that. Or can i send a message with a Supplier-Bean?
Could someone please give me an example?
Thanks a lot!
You can either use the StreamBridge or the reactor API - see Sending arbitrary data to an output (e.g. Foreign event-driven sources)
Related
I wrote a Spring Cloud Stream Producer according new functional model introduced with 3.1 version.
#EnableAutoConfiguration
#Component
public class Producer {
private final BlockingQueue<Message<Object>> messageQueue = new LinkedBlockingQueue<>();
public void produce(int messageId, Object message) {
Message<Object> toProduce = MessageBuilder
.withPayload(message)
.setHeader(PARTITION_KEY, messageId)
.build();
messageQueue.offer(toProduce);
}
#Bean
public Supplier<Message<Object>> produceMessage() {
return () -> messageQueue.poll();
}
}
I'm able to call from a REST controller the produce(int, Object) method that put data into the BlockingQueue.
The Supplier, annotated with #Bean, annotation is polled by default every second.
This is a snippet of the application.yml:
spring:
cloud:
function:
definition: produceMessage
stream:
bindings:
produceMessage-out-0:
destination: test-topic
contentType: application/json
producer:
partitionKeyExpression: headers['partitionKey']
partitionCount: 1
errorChannelEnabled: true
...
kafka:
bindings:
produceMessage-out-0:
producer:
configuration:
retries: 10
max.in.flight.requests.per.connection: 1
request.timeout.ms: 20000
Finally I wrote this class in order to test my code:
#SpringBootTest
class ProducerTest {
#Test
void producerTest() {
try (ConfigurableApplicationContext context = new SpringApplicationBuilder
(TestChannelBinderConfiguration.getCompleteConfiguration(Producer.class))
.web(WebApplicationType.NONE)
.run("--spring.jmx.enabled=false")) {
OutputDestination output = context.getBean(OutputDestination.class);
Producer producer= context.getBean(Producer.class);
producer.produce(1, new MyMessage(1, "Hello Message"));
Message<byte[]> received = output.receive();
Assertions.assertNotNull(received);
}
}
}
When I run the test, it fails because received is null.
I read a lot of examples that show this is the way to test this type of Producer.
What am I doing wrong? Can you help me please?
Thanks
in this page tells that you can't make just one Outputs
but I need to make just one Outputs by using Spring Cloud Stream Kafka Binder
what should I do?
some articles says that using org.springframework.cloud.stream.function.StreamBridge but it's not works for me
I made StreamBridge to send topics to Kafka but Kafka doesn't produce topics to my Spring boot application
and this is my application.yml and produce topic code
// producer Springboot application
spring.cloud.stream:
kafka:
binder:
brokers: {AWS.IP}:9092
zkNodes: {AWS.IP}:2181
bindings:
deliveryIncoming:
destination : deliveryIncomingtopic
#Bean
public CommandLineRunner commandLineRunner(ApplicationContext ctx) {
// wanna produce deliveryIncomingtopic and send to Spring's Consumer
}
// Consumer Springboot application
spring :
cloud:
stream:
kafka:
binder:
brokers: {AWS.IP}:9092
zkNodes: {AWS.IP}:2181
function:
definition : deliveryIncoming;
bindings:
deliveryIncoming-in-0:
destination : deliveryIncomingtopic
#Bean
public Consumer<KStream<String, String>> deliveryIncoming() {
return input ->
input.foreach((key, value) -> {
System.out.println("deliveryIncoming is playing");
System.out.println("Key: " + key + " Value: " + value);
});
}
EDIT
sorry I think I made kind of unclear
I just want to do like below
produce(deliveryIncomingtopic) -> Kafka -> consumer(deliveryIncomingtopic)
If that's the case, then you need to change your bean function definition to return java.util.Function instead of java.util.Consumer.
#Bean
public Function<KStream<String, String>, KStream<String, String>> deliveryIncoming() {
return input ->
input.foreach((key, value) -> {
System.out.println("deliveryIncoming is playing");
System.out.println("Key: " + key + " Value: " + value);
});
}
However, AFAIK.. you still need to define the output channel in your application.yml. You can use the same name with different suffix. Something like below :
deliveryIncoming-in-0:
destination: <your_topic_name>
deliveryIncoming-out-0:
destination: <your_topic_name>
Just want to make things clear here. Are you looking to consume a message from your inbound topic deliveryIncomingtopic and then generate/produce another message to another output topic ?
If that's your question about, then I believe you were missing something there within your application.yml.
You need to have another configuration for your output topic. e.g. :
Since you're using Spring Cloud Function (based on what i see in your application.yml), then you should add more configuration for your output topic as follow:
spring :
cloud:
stream:
kafka:
binder:
brokers: {AWS.IP}:9092
zkNodes: {AWS.IP}:2181
function:
definition: deliveryIncoming;deliveryOutput
bindings:
deliveryIncoming-in-0:
destination: deliveryIncomingtopic
deliveryOutput-out-0:
destination: deliveryOutput
And also define another bean for your producer function :
#Bean
public Producer<KStream<String, String>> deliveryOutput() {
// do your necessary logic here to outbound your message
}
Hope this will match your expectation.
I have two microservices written in Java, using Spring Boot.
I use Kafka, through Spring Cloud Stream Kafka, to send messages between them.
I need to send a custom header, but with no success until now.
I have read and tried most of the things I have found on internet and Spring Cloud Stream documentation...
... still I have been unable to make it work.
Which means I never receive a message in the receiver because the header cannot be found and cannot be null.
I suspect the header is never written in the message. Right now I am trying to verify this with Kafkacat.
Any help will be wellcome
Thanks in advance.
------ information --------------------
Here it is the sender code:
#SendTo("notifications")
public void send(NotificationPayload payload, String eventId) {
var headerMap = Collections.singletonMap("EVENT_ID",
eventId.getBytes(StandardCharsets.UTF_8));
MessageHeaders headers = new MessageHeaders(headerMap);
var message = MessageBuilder.createMessage(payload, headers);
notifications.send(message);
}
Where notifications is a MessageChannel
Here is the related configuration for message sender.
spring:
cloud:
stream:
defaultBinder: kafka
bindings:
notifications:
binder: kafka
destination: notifications
contentType: application/x-java-object;type=com.types.NotificationPayload
producer:
partitionCount: 1
headerMode: headers
kafka:
binder:
headers: EVENT_ID
I have also tried with headers: "EVENT_ID"
Here is the code for the receiver part:
#StreamListener("notifications")
public void receiveNotif(#Header("EVENT_ID") byte[] eventId,
#Payload NotificationPayload payload) {
var eventIdS = new String((byte[]) eventId, StandardCharsets.UTF_8);
...
// do something with the payload
}
And the configuration for the receiving part:
spring:
cloud:
stream:
kafka:
bindings:
notifications:
consumer:
headerMode: headers
Versions
<spring-cloud-stream-dependencies.version>Horsham.SR4</spring-cloud-stream-dependencies.version>
<spring-cloud-stream-binder-kafka.version>3.0.4.RELEASE</spring-cloud-stream-binder-kafka.version>
<spring-cloud-schema-registry.version>1.0.4.RELEASE</spring-cloud-schema-registry.version>
<spring-cloud-stream.version>3.0.4.RELEASE</spring-cloud-stream.version>
What version are you using? Describe "can't get it to work" in more detail.
This works fine...
#SpringBootApplication
#EnableBinding(Source.class)
public class So64586916Application {
public static void main(String[] args) {
SpringApplication.run(So64586916Application.class, args);
}
#InboundChannelAdapter(channel = Source.OUTPUT)
Message<String> source() {
return MessageBuilder.withPayload("foo")
.setHeader("myHeader", "someValue")
.build();
}
#KafkaListener(id = "in", topics = "output")
void listen(Message<?> in) {
System.out.println(in);
}
}
spring.kafka.consumer.auto-offset-reset=earliest
GenericMessage [payload=byte[3], headers={myHeader=someValue, kafka_offset=0, ...
GenericMessage [payload=byte[3], headers={myHeader=someValue, kafka_offset=1, ...
EDIT
I also tested it by sending to the channel directly; again with no problems:
#Autowired
MessageChannel output;
#Bean
public ApplicationRunner runner() {
return args -> {
this.output.send(MessageBuilder.withPayload("foo")
.setHeader("myHeader", "someValue")
.build());
};
}
There was a change request on my system, which currently listens to multiple channels and send messages to multiple channels as well, but now the destination names will be in the database and change any time.
I'm having trouble believing I'm the first one to come across this, but I see limited information out there.
All I found is these 2...
Dynamic sink destination:
https://github.com/spring-cloud-stream-app-starters/router/tree/master/spring-cloud-starter-stream-sink-router, but how would that work to active listening to those channels the way it's done by #StreamListener?
Dynamic source destinations:
https://github.com/spring-cloud/spring-cloud-stream-samples/blob/master/source-samples/dynamic-destination-source/, which does this
#Bean
#ServiceActivator(inputChannel = "sourceChannel")
public ExpressionEvaluatingRouter router() {
ExpressionEvaluatingRouter router = new ExpressionEvaluatingRouter(new SpelExpressionParser().parseExpression("payload.id"));
router.setDefaultOutputChannelName("default-output");
router.setChannelResolver(resolver);
return router;
}
But what's that "payload.id"? And where are the destinations specified there??
Feel free to improve my answer, I hope it will help others.
Now the code (It worked in my debugger). This is an example, not production ready!
This is how to send a message to dynamic destination
import org.springframework.messaging.MessageChannel;
import org.springframework.cloud.stream.annotation.EnableBinding;
import org.springframework.cloud.stream.binding.BinderAwareChannelResolver;
#Service
#EnableBinding
public class MessageSenderService {
#Autowired
private BinderAwareChannelResolver resolver;
#Transactional
public void sendMessage(final String topicName, final String payload) {
final MessageChannel messageChannel = resolver.resolveDestination(topicName);
messageChannel.send(new GenericMessage<String>(payload));
}
}
And configuration for Spring Cloud Stream.
spring:
cloud:
stream:
dynamicDestinations: output.topic.1,output.topic2,output.topic.3
I found here
https://docs.spring.io/spring-cloud-stream/docs/Elmhurst.RELEASE/reference/htmlsingle/index.html#dynamicdestination
It will work in spring Cloud Stream version 2+. I use 2.1.2
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream</artifactId>
<version>2.1.2.RELEASE</version>
</dependency>
This is how to consume a message from dynamic destination
https://stackoverflow.com/a/56148190/4587961
Configuration
spring:
cloud:
stream:
default:
consumer:
concurrency: 2
partitioned: true
bindings:
# inputs
input:
group: application_name_group
destination: topic-1,topic-2
content-type: application/json;charset=UTF-8
Java consumer.
#Component
#EnableBinding(Sink.class)
public class CommonConsumer {
private final static Logger logger = LoggerFactory.getLogger(CommonConsumer.class);
#StreamListener(target = Sink.INPUT)
public void consumeMessage(final Message<Object> message) {
logger.info("Received a message: \nmessage:\n{}", message.getPayload());
final String topic = message.getHeaders().get("kafka_receivedTopic");
// Here I define logic which handles messages depending on message headers and topic.
// In my case I have configuration which forwards these messages to webhooks, so I need to have mapping topic name -> webhook URI.
}
}
I have been attempting to get an inbound SubscribableChannel and outbound MessageChannel working in my spring boot application.
I have successfully setup the kafka channel and tested it successfully.
Furthermore I have create a basic spring boot application that tests adding and receiving things from the channel.
The issue I am having is when I put the equivalent code in the application it belongs in, it appears that the messages never get sent or received. By debugging it's hard to ascertain what's going on but the only thing that looks different to me is the channel-name. In the working impl the channel name is like application.channel in the non working app its localhost:8080/channel.
I was wondering if there is some spring boot configuration blocking or altering the creation of the channels into a different channel source?
Anyone had any similar issues?
application.yml
spring:
datasource:
url: jdbc:h2:mem:dpemail;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE
platform: h2
username: hello
password:
driverClassName: org.h2.Driver
jpa:
properties:
hibernate:
show_sql: true
use_sql_comments: true
format_sql: true
cloud:
stream:
kafka:
binder:
brokers: localhost:9092
bindings:
email-in:
destination: email
contentType: application/json
email-out:
destination: email
contentType: application/json
Email
public class Email {
private long timestamp;
private String message;
public long getTimestamp() {
return timestamp;
}
public void setTimestamp(long timestamp) {
this.timestamp = timestamp;
}
public String getMessage() {
return message;
}
public void setMessage(String message) {
this.message = message;
}
}
Binding Config
#EnableBinding(EmailQueues.class)
public class EmailQueueConfiguration {
}
Interface
public interface EmailQueues {
String INPUT = "email-in";
String OUTPUT = "email-out";
#Input(INPUT)
SubscribableChannel inboundEmails();
#Output(OUTPUT)
MessageChannel outboundEmails();
}
Controller
#RestController
#RequestMapping("/queue")
public class EmailQueueController {
private EmailQueues emailQueues;
#Autowired
public EmailQueueController(EmailQueues emailQueues) {
this.emailQueues = emailQueues;
}
#RequestMapping(value = "sendEmail", method = POST)
#ResponseStatus(ACCEPTED)
public void sendToQueue() {
MessageChannel messageChannel = emailQueues.outboundEmails();
Email email = new Email();
email.setMessage("hello world: " + System.currentTimeMillis());
email.setTimestamp(System.currentTimeMillis());
messageChannel.send(MessageBuilder.withPayload(email).setHeader(MessageHeaders.CONTENT_TYPE, MimeTypeUtils.APPLICATION_JSON).build());
}
#StreamListener(EmailQueues.INPUT)
public void handleEmail(#Payload Email email) {
System.out.println("received: " + email.getMessage());
}
}
I'm not sure if one of the inherited configuration projects using Spring-Cloud, Spring-Cloud-Sleuth might be preventing it from working, but even when I remove it still doesnt. But unlike my application that does work with the above code I never see the ConsumeConfig being configured, eg:
o.a.k.clients.consumer.ConsumerConfig : ConsumerConfig values:
auto.commit.interval.ms = 100
auto.offset.reset = latest
bootstrap.servers = [localhost:9092]
check.crcs = true
client.id = consumer-2
connections.max.idle.ms = 540000
enable.auto.commit = false
exclude.internal.topics = true
(This configuration is what I see in my basic Spring Boot application when running the above code and the code works writing and reading from the kafka channel)....
I assume there is some over spring boot configuration from one of the libraries I'm using creating a different type of channel I just cannot find what that configuration is.
What you posted contains a lot of unrelated configuration, so hard to determine if anything gets in the way. Also, when you say "..it appears that the messages never get sent or received.." are there any exceptions in the logs? Also, please state the version of Kafka you're using as well as Spring Cloud Stream.
Now, I did try to reproduce it based on your code (after cleaning up a bit to only leave relevant parts) and was able to successfully send/receive.
My Kafka version is 0.11 and Spring Cloud Stream 2.0.0.
Here is the relevant code:
spring:
cloud:
stream:
kafka:
binder:
brokers: localhost:9092
bindings:
email-in:
destination: email
email-out:
destination: email
#SpringBootApplication
#EnableBinding(KafkaQuestionSoApplication.EmailQueues.class)
public class KafkaQuestionSoApplication {
public static void main(String[] args) {
SpringApplication.run(KafkaQuestionSoApplication.class, args);
}
#Bean
public ApplicationRunner runner(EmailQueues emailQueues) {
return new ApplicationRunner() {
#Override
public void run(ApplicationArguments args) throws Exception {
emailQueues.outboundEmails().send(new GenericMessage<String>("Hello"));
}
};
}
#StreamListener(EmailQueues.INPUT)
public void handleEmail(String payload) {
System.out.println("received: " + payload);
}
public interface EmailQueues {
String INPUT = "email-in";
String OUTPUT = "email-out";
#Input(INPUT)
SubscribableChannel inboundEmails();
#Output(OUTPUT)
MessageChannel outboundEmails();
}
}
Okay so after a lot of debugging... I discovered that something is creating a Test Support Binder (how don't know yet) so obviously this is used to not impact add messages to a real channel.
After adding
#SpringBootApplication(exclude = TestSupportBinderAutoConfiguration.class)
The kafka channel configurations have worked and messages are adding.. would be interesting to know what on earth is setting up this test support binder.. I'll find that sucker eventually.