I have a micro service that consumes data from Kinesis stream , I want to convert it into reactive stream consumer since application is already written using spring-boot 2.x. I was going through support of Project Rector for Kinesis and found few example like aws sdk example.
I want to know if there is better way to do this. There are dedicated projects for streaming with Kafka and project reactor , is there any such thing for Kinesis ?
Related
I have 2 Spring Cloud Stream applications to publish and consume messages to/from a AWS Kinesis data stream using the spring-cloud-stream-binder-kinesis dependency.
Upon an event consumption from Kinesis, I need to save the information in DynamoDB. My understanding is that there is a Spring module to take kinesis -> DynamoDB but have not been able to find it.
I am a junior programmer in banking. I want to make a microservice system that get data from kafka and processes it. after that, save to database and send final data to client app. What technology can i use? I plan to use spring bacth and kafka. Can the technology be implemented in my project or is there a better alternative?
To process data from a Kafka topic I recommend you to use Kafka Streams API, especially Spring Kafka Streams.
Kafka Streams and Spring
And to store the data in a database, you should use a Kafka Sink Connector.
Kafka Connect
This approach is very common and easy if your company has a Kafka ecosystem.
In terms of alternatives, here you will find an interesting comparison:
https://scramjet.org/blog/welcome-to-the-family
3 in 1 serverless
Scramjet takes a slightly different approach - 3 platforms in one.
Both the free product https://hub.scramjet.org/ for installation on your server and the cloud platform are available - currently also free in the beta version https://scramjet.org/#join-beta
I want to use Spring Cloud Stream for consuming and processing Apache Kafka queues and writing them to MongoDB. I saw that there is an option of using the library so that functions will be Reactive, or Imperative. In most Spring projects the imperative way is the default, but as for my understanding, in spring cloud stream the reactive paradigm is the default.
I wonder what is considered the most “stable” api e.g. what is recommended to use for enterprise?
Reactive API is stable and yes we provide support for it. In other words you can write functions using reactive API (e.g., Function<Flux, Flux>).
However, i want to be very clear that support for API does not mean support for the full stack of reactive capabilities since those actually depend on source and targets which are not reactive.
That said, with Kafka you can rely on native reactive support provided by Kafka itself and Spring Cloud Stream using Kafka Streams binder - https://docs.spring.io/spring-cloud-stream-binder-kafka/docs/3.1.5/reference/html/spring-cloud-stream-binder-kafka.html#_kafka_streams_binder
I am using AWS as a Cloud provider. I have a Microservice that is in Frankfurt Region and will publish events to a Kinesis Data stream in the same region using Spring Cloud Stream (SCDF) Kinesis Adapter. I have multiple Microservices in different regions (Oregon, Ohio, Singapore, Mumbai etc) which is consuming events from respective Kinesis Streams in the respective regions using Spring Cloud Stream (SCDF) Kinesis Adapter. Now I have to route the Events which are there in the Frankfurt Kinesis to different Data Streams in different regions (only related to respective Kinesis).
Can I do this using any of the Spring provided functionality? Can I use Spring Cloud Stream or SCDF to do cross-region routing? if yes, please point to some examples.
If #1 is not possible what are the best ways to do this?
I read about AWS EventBridge, is it a correct choice for the above use case?
Spring Cloud Stream Binder for AWS Kinesis is fully based on a standard AWS Client or KCL. Both of them are require particular region to be configured staticaly or resolved from the EC2 environment. So, to be able to consume from one region and relay stream records to another, you have to code some "replicator" stream application.
Luckily Spring Cloud Stream application can be configured for several binders. Right, in our case both of them are going to be the same Kinesis binder, but we are going to configure them for different credentials and different regions.
See Spring Cloud Stream docs for multi-binder configuration: https://docs.spring.io/spring-cloud-stream/docs/3.1.2/reference/html/spring-cloud-stream.html#multiple-binders.
Probably the code of your Stream Application could be just a plain identity function:
#Bean
public Function<byte[], byte[]> kinesisStreamRelay() {
return Function.identity();
}
And you bind it for an in destination from one Kinesis binder and out destination in the other.
Also see other ways to do that in this article: https://engineering.opsgenie.com/cross-region-replication-of-kinesis-streams-4a62f3bb269d
See Spring Cloud Function support for AWS Lambda: https://docs.spring.io/spring-cloud-function/docs/3.1.1/reference/html/aws.html. Spring Cloud Stream does not provides binder implementation for AWS Lambda.
Using Kafka as a messaging system in a microservice architecture what are the benefits of using spring-kafka vs. spring-cloud-stream + spring-cloud-starter-stream-kafka ?
The spring cloud stream framework supports more messaging systems and has therefore a more modular design. But what about the functionality ? Is there a gap between the functionality of spring-kafka and spring-cloud-stream + spring-cloud-starter-stream-kafka ?
Which API is better designed?
Looking forward to read about your opinions
Spring Cloud Stream with kafka binder rely on Spring-kafka. So the former has all functionalities supported by later, but the former will be more heavyweight. Below are some points help you make the choice:
If you might change kafka into another message middleware in the future, then Spring Cloud stream should be your choice since it hides implementation details of kafka.
If you want to integrate other message middle with kafka, then you should go for Spring Cloud stream, since its selling point is to make such integration easy.
If you want to enjoy the simplicity and not accept performance overhead, then choose spring-kafka
If you plan to migrate to public cloud service such as AWS Kensis, Azure EventHub, then use spring cloud stream which is part of spring cloud family.
Use Spring Cloud Stream when you are creating a system where one channel is used for input does some processing and sends it to one output channel. In other words it is more of an RPC system to replace say RESTful API calls.
If you plan to do an event sourcing system, use Spring-Kafka where you can publish and subscribe to the same stream. This is something that Spring Cloud Stream does not allow you do do easily as it disallows the following
public interface EventStream {
String STREAM = "event_stream";
#Output(EventStream.STREAM)
MessageChannel publisher();
#Input(EventStream.STREAM)
SubscribableChannel stream();
}
A few things that Spring Cloud Stream helps you avoid doing are:
setting up the serializers and deserializers