I want to write an AWS kinesis stream consumer in a Spring boot application. And I'm not sure if Spring has a native support of kinesis, or I have to use the kinesis client library.
According to this blog post org.springframework.integration:spring-integration-aws has it (RELEASE is available in maven repo). However, this example on GitHub uses org.springframework.cloud:spring-cloud-starter-stream-kinesis, which is available only on Spring snapshots repo under 1.0.0.BUILD-SNAPSHOT.
EDIT: The question is, where can I find an example of KinesisMessageDrivenChannelAdapter?
Not clear what is the question though.
If you are looking for a sample, there is indeed no one. Right the solution we have in Spring is definitely a Channel Adapter for Spring Integration. And that KinesisMessageDrivenChannelAdapter is exactly consumer implementation for AWS Kinesis:
#SpringBootApplication
public static class MyConfiguration {
#Bean
public KinesisMessageDrivenChannelAdapter kinesisInboundChannelChannel(AmazonKinesis amazonKinesis) {
KinesisMessageDrivenChannelAdapter adapter =
new KinesisMessageDrivenChannelAdapter(amazonKinesis, "MY_STREAM");
adapter.setOutputChannel(kinesisReceiveChannel());
return adapter;
}
}
The sample you found on GitHub is for Spring Cloud Stream and based on the Kinesis Binder which indeed is still under development.
Related
I'm building a set of microservices in the Spring Boot framework, each of them integrating with Kafka for messaging. There appears to be 3 separate but related Spring libraries offering Kafka integration:
Spring Kafka
Spring Integration Kafka
Spring Cloud Stream
My goal is to abstract away the details of the underlying messaging system and provide a simple messaging service layer to my microservices to send and receive messages. I would like this service layer to work with my domain classes (POJOs) and not have the microservices being concerned with building Message instances. For example:
public interface MyMessagingService {
void send(MyPojo obj);
MyPojo receive();
}
Secondly, I would like to add Avro support, but first I will get it working with JSON.
To cut to the chase, there seems to be multiple ways of achieving this, which is very confusing, especially with the various Spring libraries available. What is the most straightforward way I can provide such a shared messaging layer to my microservices, where they only have to be concerned with domain classes?
I've come across #MessagingGateway from Spring Integration which looked promising, but this seems to tie in to send and reply semantics, and my services won't be expecting a reply message from Kafka.
The examples I have looked at, some linked below, still seem to have to construct Message instances themselves. Is there a simpler way of doing this?
https://codenotfound.com/spring-kafka-spring-integration-example.html
https://www.baeldung.com/spring-cloud-stream
If your ". . goal is to abstract away the details of the underlying messaging system and provide a simple messaging service layer to my microservices to send and receive messages", then why not just use spring-cloud-stream?
The code developer doesn't even have to know that the code he/she writes will be part of some message system. For example,
#SpringBootApplication
public class SampleStreamApplication {
public static void main(String[] args) throws Exception {
SpringApplication.run(SampleStreamApplication.class);
}
#Bean
public Function<String, String> uppercase() {
return value -> value.toUpperCase();
}
}
The above is a complete and fully functioning spring cloud stream application that (in the context of Kafka binder) will receive from "input" topic and send to "output" topic the value that was passed through uppercase(..) function.
Yes the type conversion is transparently handled for you for both JSON, Avro etc.
Obviously there are some details, but we can certainly discuss them when you have a more concrete questions. Fo now I would suggest going through some reference documentation first.
According to Spring release notes, spring-integration-aws.1.1.0.M1 does not include DynamoDB MetaDataStore implementation. There is still ConcurrentMetadataStore class which is a key-value based store and based on implementation I suppose it maps streams with latest sequence number read. But it does not use any data store as to retrieve checkpoints.
I am using spring integration for kinesis consuming and need to implement checkpointing. I am wondering if I need to do it manually by connecting to DynamoDB and always update checkpoints or there is another way of doing it using spring framework?
P.S: I can't use Spring Cloud KinesisBinderConfiguration as I dynamically consume events from a list of configurable streams.
Thank you
If you are not talking about Spring Cloud Stream and the AWS Kinesis Binder implementation, then I don't see any blockers for you to upgrade your solution to the Spring Integration AWS 2.0 and go ahead with already provided DynamoDbMetaDataStore.
Or if that is so hard for you to move to the Spring Integration 5.0, then you simply can consider to copy/paste an implementation to your own class and inject it into the KinesisMessageDrivenChannelAdapter: https://github.com/spring-projects/spring-integration-aws/blob/master/src/main/java/org/springframework/integration/aws/metadata/DynamoDbMetaDataStore.java
Although it is really available in the 1.1.0.RELEASE - I don't see reason for your to stick with the 1.1.0.M1: https://spring.io/blog/2017/11/27/spring-integration-for-aws-1-1-ga-available
I am thinking of using a chain of Akka workers to model a workflow inside a DeferredResult based Spring MVC web application. Essentially the controller will return a DeferredResult and the actors in the chain will work to populate a CompletableFuture which feeds the DeferredResult when completed.
What I am not able to figure out is:
* Will Akka exert back-pressure if this setup takes on too much load.
* If so, how can I detect that this is happening?
Consider using Alpakka's Spring Web connector, which allows integration of Akka Streams in a Spring Web application. Akka Streams provides backpressure as part of its adherence to the reactive streams specification, and the connector allows the exposure of streams as HTTP endpoints in a Spring application. An example from the Alpakka documentation:
#RestController
public class SampleController {
#RequestMapping("/")
public Source<String, NotUsed> index() {
return
Source.repeat("Hello world!")
.intersperse("\n")
.take(10);
}
}
In your case, you could model your workflow as a stream.
The Akka team recently published a blog post about this connector.
I am seeking a simple workable example which use Spring Cloud Stream Kafka with Confluent Schema Registry (producer & consumer). I followed the spring cloud stream reference guide by adding the following code but it didn't work. Can anyone guide me how to achieve it? Many thanks!
#Bean
public SchemaRegistryClient schemaRegistryClient(#Value("${spring.cloud.stream.schemaRegistryClient.endpoint}") String endpoint){
ConfluentSchemaRegistryClient client = new ConfluentSchemaRegistryClient();
client.setEndpoint(endpoint);
return client;
}
I tried this schema registry sample and it worked for me. All instructions are mentioned in README file.
Here is a simple example that shows spring cloud stream with Avro serialization. This example stubs and mocks out the schema registry.
Your bean looks correct, you just need a running schema registry and configuration for "spring.cloud.stream.schemaRegistryClient.endpoint" in application.properties (or application.yml)
Using Kafka as a messaging system in a microservice architecture what are the benefits of using spring-kafka vs. spring-cloud-stream + spring-cloud-starter-stream-kafka ?
The spring cloud stream framework supports more messaging systems and has therefore a more modular design. But what about the functionality ? Is there a gap between the functionality of spring-kafka and spring-cloud-stream + spring-cloud-starter-stream-kafka ?
Which API is better designed?
Looking forward to read about your opinions
Spring Cloud Stream with kafka binder rely on Spring-kafka. So the former has all functionalities supported by later, but the former will be more heavyweight. Below are some points help you make the choice:
If you might change kafka into another message middleware in the future, then Spring Cloud stream should be your choice since it hides implementation details of kafka.
If you want to integrate other message middle with kafka, then you should go for Spring Cloud stream, since its selling point is to make such integration easy.
If you want to enjoy the simplicity and not accept performance overhead, then choose spring-kafka
If you plan to migrate to public cloud service such as AWS Kensis, Azure EventHub, then use spring cloud stream which is part of spring cloud family.
Use Spring Cloud Stream when you are creating a system where one channel is used for input does some processing and sends it to one output channel. In other words it is more of an RPC system to replace say RESTful API calls.
If you plan to do an event sourcing system, use Spring-Kafka where you can publish and subscribe to the same stream. This is something that Spring Cloud Stream does not allow you do do easily as it disallows the following
public interface EventStream {
String STREAM = "event_stream";
#Output(EventStream.STREAM)
MessageChannel publisher();
#Input(EventStream.STREAM)
SubscribableChannel stream();
}
A few things that Spring Cloud Stream helps you avoid doing are:
setting up the serializers and deserializers