I have 2 Spring Cloud Stream applications to publish and consume messages to/from a AWS Kinesis data stream using the spring-cloud-stream-binder-kinesis dependency.
Upon an event consumption from Kinesis, I need to save the information in DynamoDB. My understanding is that there is a Spring module to take kinesis -> DynamoDB but have not been able to find it.
Related
We created a streams application in which we are consuming the data and pushing it to database. But it is creating a dummy topic to produce the data and throwing a error like "Not authorized to access topic".
Is there any configuration to restrict the streams app to consume alone.
We could have used a Consumer application but due to performance consideration we switched to streams.
I am using AWS as a Cloud provider. I have a Microservice that is in Frankfurt Region and will publish events to a Kinesis Data stream in the same region using Spring Cloud Stream (SCDF) Kinesis Adapter. I have multiple Microservices in different regions (Oregon, Ohio, Singapore, Mumbai etc) which is consuming events from respective Kinesis Streams in the respective regions using Spring Cloud Stream (SCDF) Kinesis Adapter. Now I have to route the Events which are there in the Frankfurt Kinesis to different Data Streams in different regions (only related to respective Kinesis).
Can I do this using any of the Spring provided functionality? Can I use Spring Cloud Stream or SCDF to do cross-region routing? if yes, please point to some examples.
If #1 is not possible what are the best ways to do this?
I read about AWS EventBridge, is it a correct choice for the above use case?
Spring Cloud Stream Binder for AWS Kinesis is fully based on a standard AWS Client or KCL. Both of them are require particular region to be configured staticaly or resolved from the EC2 environment. So, to be able to consume from one region and relay stream records to another, you have to code some "replicator" stream application.
Luckily Spring Cloud Stream application can be configured for several binders. Right, in our case both of them are going to be the same Kinesis binder, but we are going to configure them for different credentials and different regions.
See Spring Cloud Stream docs for multi-binder configuration: https://docs.spring.io/spring-cloud-stream/docs/3.1.2/reference/html/spring-cloud-stream.html#multiple-binders.
Probably the code of your Stream Application could be just a plain identity function:
#Bean
public Function<byte[], byte[]> kinesisStreamRelay() {
return Function.identity();
}
And you bind it for an in destination from one Kinesis binder and out destination in the other.
Also see other ways to do that in this article: https://engineering.opsgenie.com/cross-region-replication-of-kinesis-streams-4a62f3bb269d
See Spring Cloud Function support for AWS Lambda: https://docs.spring.io/spring-cloud-function/docs/3.1.1/reference/html/aws.html. Spring Cloud Stream does not provides binder implementation for AWS Lambda.
I have a micro service that consumes data from Kinesis stream , I want to convert it into reactive stream consumer since application is already written using spring-boot 2.x. I was going through support of Project Rector for Kinesis and found few example like aws sdk example.
I want to know if there is better way to do this. There are dedicated projects for streaming with Kafka and project reactor , is there any such thing for Kinesis ?
Using Kafka as a messaging system in a microservice architecture what are the benefits of using spring-kafka vs. spring-cloud-stream + spring-cloud-starter-stream-kafka ?
The spring cloud stream framework supports more messaging systems and has therefore a more modular design. But what about the functionality ? Is there a gap between the functionality of spring-kafka and spring-cloud-stream + spring-cloud-starter-stream-kafka ?
Which API is better designed?
Looking forward to read about your opinions
Spring Cloud Stream with kafka binder rely on Spring-kafka. So the former has all functionalities supported by later, but the former will be more heavyweight. Below are some points help you make the choice:
If you might change kafka into another message middleware in the future, then Spring Cloud stream should be your choice since it hides implementation details of kafka.
If you want to integrate other message middle with kafka, then you should go for Spring Cloud stream, since its selling point is to make such integration easy.
If you want to enjoy the simplicity and not accept performance overhead, then choose spring-kafka
If you plan to migrate to public cloud service such as AWS Kensis, Azure EventHub, then use spring cloud stream which is part of spring cloud family.
Use Spring Cloud Stream when you are creating a system where one channel is used for input does some processing and sends it to one output channel. In other words it is more of an RPC system to replace say RESTful API calls.
If you plan to do an event sourcing system, use Spring-Kafka where you can publish and subscribe to the same stream. This is something that Spring Cloud Stream does not allow you do do easily as it disallows the following
public interface EventStream {
String STREAM = "event_stream";
#Output(EventStream.STREAM)
MessageChannel publisher();
#Input(EventStream.STREAM)
SubscribableChannel stream();
}
A few things that Spring Cloud Stream helps you avoid doing are:
setting up the serializers and deserializers
I am migrating from Spring XD to Spring Cloud Data Flow. When I am looking for module list I realised that some of the sources are not listed in Spring Cloud Flow - One of them is KAFKA source.
My question is why KAFKA source is removed from standard sources list in spring cloud data flow ?
When I am looking for module list I realised that some of the sources are not listed in Spring Cloud Flow
Majority of the applications are ported over and the remaining are incrementally prioritized - you can keep track of the remaining subset in the backlog.
My question is why KAFKA source is removed from standard sources list in spring cloud data flow ?
Kafka is not removed and in fact, we are highly opinionated about Kafka in the context of streaming use-cases so much so that it is baked into the DSL directly. More details here.
For instance,
(i) if you've to consume from a Kafka topic (as a source), your stream definition would be:
stream create --definition ":someAwesomeTopic > log" --name subscribe_to_broker --deploy
(ii) if you've to write to a Kafka topic (as a sink), your stream definition would be:
stream create --definition "http --server.port=9001 > :someAwesomeTopic" --name publish_to_broker --deploy
(where *someAwesomeTopic* is the named destination, a topic name)