I've been trying to do some searching (Google, Slack, Stack) and have yet to find an answer. We have some applications that are written using Spring Cloud Streams and were interested in swapping the back end from Kafka to Pulsar. Spring currently doesn't have any native support for Kafka, however it appears that pulsar provides the ability to use the Kafka API to communicate with pulsar directly (https://pulsar.apache.org/docs/en/adaptors-kafka).
I'm wondering if anybody has gone down the route of trying to use this replacement for the Kafka-clients library in the context of Spring cloud messaging.
An alternative valid approach, of course, is to just re-write the code - but I wanted to turn to the community to see if anybody has gone down this road as of yet.
Thanks
While not exactly what you're asking for, I've tried to integrate Pulsar with Spring-Kafka using the shaded library org.apache.pulsar:pulsar-client-kafka:2.5.0 and it blows-up with this stack trace:
Caused by: java.lang.UnsupportedOperationException: null
at org.apache.kafka.clients.consumer.KafkaConsumer.partitionsFor(KafkaConsumer.java:650) ~[pulsar-client-kafka-2.5.0.jar:2.5.0]
at org.springframework.kafka.listener.AbstractMessageListenerContainer.checkTopics(AbstractMessageListenerContainer.java:312) ~[spring-kafka-2.2.10.RELEASE.jar:2.2.10.RELEASE]
at org.springframework.kafka.listener.ConcurrentMessageListenerContainer.doStart(ConcurrentMessageListenerContainer.java:136) ~[spring-kafka-2.2.10.RELEASE.jar:2.2.10.RELEASE]
at org.springframework.kafka.listener.AbstractMessageListenerContainer.start(AbstractMessageListenerContainer.java:292) ~[spring-kafka-2.2.10.RELEASE.jar:2.2.10.RELEASE]
at org.springframework.kafka.config.KafkaListenerEndpointRegistry.startIfNecessary(KafkaListenerEndpointRegistry.java:311) ~[spring-kafka-2.2.10.RELEASE.jar:2.2.10.RELEASE]
at org.springframework.kafka.config.KafkaListenerEndpointRegistry.start(KafkaListenerEndpointRegistry.java:255) ~[spring-kafka-2.2.10.RELEASE.jar:2.2.10.RELEASE]
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:182) ~[spring-context-5.1.10.RELEASE.jar:5.1.10.RELEASE]
This specific error can be bypassed (for these specific library versions) using a custom implementation of AbstractMessageListenerContainer.checkTopics but you'll likely run into more issues.
Related
I am working on a project that is trying to use the polled consumer API. However, existing documentation, blog posts and sample code seems to use deprecated annotations (such as org.springframework.cloud.stream.annotation.Input). This seems to be because they are relying on the older style of Spring Cloud stream applications rather than using Java functional api (e.g., java.util.function.Function), as shown in other examples such as this one, given in the same repo.
Is there a way to use functional style with polled consumers in Spring Cloud Stream?
You are using outdated documentation. The most current is available from the project site - https://spring.io/projects/spring-cloud-stream#learn.
The section you are looking for is - https://docs.spring.io/spring-cloud-stream/docs/3.1.5/reference/html/spring-cloud-stream.html#spring-cloud-streams-overview-using-polled-consumers
I think I'm not grasping some basic concepts of Kafka here, so I'm hoping Stack maybe able to the help.
I've been trying to learn Kafka with Spring boot by following this GIT repo here:
I understand how to without avro take a Java class from one Microservice, send it to Kafka and consume / serialise it on another Microservice...however I hate that idea. As it means I must have an identical class on the other Microservice in terms of package location / name etc
So overall I've two questions here I guess.
I want to understand how I can share message across my spring boot microservices and map them to classes without copying said classes from one service to the other
I want to be able to consume from my Spring Kafka listeners messages created from another language say C#
Where I'm currently at is, I have the avro example from the repo above up and running along with my local kafka and Schema registry instance.
However if I create a duplicate class and call it UserTest (For example) and have it identical to the User class consumed here I get stacktraces like the following:
Caused by: org.springframework.messaging.converter.MessageConversionException: Cannot convert from [io.confluent.developer.User] to [io.confluent.developer.kafkaworkshop.streams.User] for GenericMessage [payload={"name": "vik", "age": 33}, headers={kafka_offset=6, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer#54200a0e, kafka_timestampType=CREATE_TIME, kafka_receivedMessageKey=vik, kafka_receivedPartitionId=1, kafka_receivedTopic=users12, kafka_receivedTimestamp=1611278455840, kafka_groupId=simple-consumer}]
Am I missing something exceptionally basic here? I thought that once the message was send in Avro format that it could be consume and mapped to another object which had the same fields...that way if the object was created in c#, the spring service would be able to interpret it no?
If anyone can help me that would be great....
Thanks!
After some researching, I have discovered that there is a spring actuator project which is used to monitor and log information about the application. Here is the link that suggests how to put log in to my own custom repository for post processing. [http://www.sedooe.com/2017/08/tracing-requests-and-responses-with-spring-boot-actuator/ ]
However, the total in/out byte counts are not included. I know that I can try to wrap HttpServeletRequest and HttpServeletResponse (in the filter, pass them to the filter chain so that it can count in/out throughput whenever they are consumed or written out) which will return custom SerlvetInputStream and ServletOutputStream that can count the number of bytes.
This seems to be a common problem in business application and there should already be a solution so I wonder if there is any easier way to achieve this.
From my understanding the Spring-Boot 1.5 actuator implementation doesn't provide such a functionality. (Assuming you are talking about Spring-Boot 1.5.)
What you could do is having a look at the Micrometer project. It's the new actuator metrics "backend" for Spring-Boot 2.0. In case you are still using Spring-Boot 1.5 you can use the micrometer-spring-legacy library in your app.
Key thing to understand here is that for Spring-Boot 1.5 this is a metric backend which is living next to the "old" actuator.
What the project provides is instrumentation for embedded Jetty and Tomcat server. Allthough only the latter does provide in and out traffic metrics.
Assuming Tomcat, you'll get tomcat_global_received_bytes_total and tomcat_global_sent_bytes_total metrics over which you can reason about in your favorite monitoring system.
I am trying to get the easiest spring cloud stream example to work, so I decided to implement the one from the reference guide reference guide.
I want it to work with kafka so I made too application both with
spring-cloud-starter-stream-kafka
as a dependency. I also tried with rabbitMq by replacint it with
spring-cloud-starter-stream-rabbit
I can't get it to work however. I don't get any exceptions and i can see that the source puts works, the sink however isn't printing the message. I am sure that it connects correctly to kafka/rabbitMq, because i don't get any exceptions and if i don't run kafka/rabbitMq I do get exceptions. I am also using the destination parameter when i run the apps (like it says in the guide), so it should be using the same destination.
I am using spring boot 1.4.2.RELEASE and spring cloud Camden.SR2
Anyone know what i missed?
Spring XD on YARN: ver 1.2.1 direct binding support for kafka source.
1.I know this is not supported yet(as of ver 1.3.0), any definite date/ver would help our project schedule ?
2.This direct binding for kafka source support is very critical for our project. We are in a situation to totally abandon Spring XD YARN in our project just because of this.
Trying to do
stream create --name directkafkatohdfs --definition "kafka | hdfs"
stream deploy directkafkatohdfs --properties "module.*.count=0"
Hitting the exception "must be a positive number. 0-count kafka sources are not currently supported"
I just want to eliminate the use of message bus/transport(redis/kafka/rabbitMQ) and want to have a direct binding of source(kafka) and sink(sink) in the same YARN container.
1.I know this is not supported yet(as of ver 1.3.0), any definite date/ver would help our project schedule.
2.This direct binding for kafka source support is very critical for our project. We are in a situation to totally abandon Spring XD YARN in our project just because of this.
Thanks
Satish Srinivasan
satsrinister#gmail.com
Thanks for the interest in Spring XD :).
For Spring XD 1.x, we suggest using composition instead of direct binding with the Kafka bus - or, in your case, the Kafka source. However, apart from that, in Spring XD 1.x it is not possible to create an entire stream without at least one hop over the bus (regardless of the type of bus or modules being used).
We are addressing direct binding (including support for entire directly bound streams) as part of Spring Cloud Data Flow (http://cloud.spring.io/spring-cloud-dataflow/) - which is the next evolution of Spring XD. We are intending to support it as a specific configuration option, rather than as a side-effect of zero-count modules. From an end-user perspective, SCDF supports the same DSL as Spring XD (with minor variations) and has the same administration UI, and definitely supports YARN, so it should be a fairly seamless transition. I would suggest starting to take a look at that. The upcoming 1.0.0.M2 release of Spring Cloud Data Flow will not support direct binding via DSL yet, but the intent is to support it in the final release which is currently planned for Q1 2016.