Spring XD on YARN: ver 1.2.1 direct binding support for kafka source.
1.I know this is not supported yet(as of ver 1.3.0), any definite date/ver would help our project schedule ?
2.This direct binding for kafka source support is very critical for our project. We are in a situation to totally abandon Spring XD YARN in our project just because of this.
Trying to do
stream create --name directkafkatohdfs --definition "kafka | hdfs"
stream deploy directkafkatohdfs --properties "module.*.count=0"
Hitting the exception "must be a positive number. 0-count kafka sources are not currently supported"
I just want to eliminate the use of message bus/transport(redis/kafka/rabbitMQ) and want to have a direct binding of source(kafka) and sink(sink) in the same YARN container.
1.I know this is not supported yet(as of ver 1.3.0), any definite date/ver would help our project schedule.
2.This direct binding for kafka source support is very critical for our project. We are in a situation to totally abandon Spring XD YARN in our project just because of this.
Thanks
Satish Srinivasan
satsrinister#gmail.com
Thanks for the interest in Spring XD :).
For Spring XD 1.x, we suggest using composition instead of direct binding with the Kafka bus - or, in your case, the Kafka source. However, apart from that, in Spring XD 1.x it is not possible to create an entire stream without at least one hop over the bus (regardless of the type of bus or modules being used).
We are addressing direct binding (including support for entire directly bound streams) as part of Spring Cloud Data Flow (http://cloud.spring.io/spring-cloud-dataflow/) - which is the next evolution of Spring XD. We are intending to support it as a specific configuration option, rather than as a side-effect of zero-count modules. From an end-user perspective, SCDF supports the same DSL as Spring XD (with minor variations) and has the same administration UI, and definitely supports YARN, so it should be a fairly seamless transition. I would suggest starting to take a look at that. The upcoming 1.0.0.M2 release of Spring Cloud Data Flow will not support direct binding via DSL yet, but the intent is to support it in the final release which is currently planned for Q1 2016.
Related
I am working on a project that is trying to use the polled consumer API. However, existing documentation, blog posts and sample code seems to use deprecated annotations (such as org.springframework.cloud.stream.annotation.Input). This seems to be because they are relying on the older style of Spring Cloud stream applications rather than using Java functional api (e.g., java.util.function.Function), as shown in other examples such as this one, given in the same repo.
Is there a way to use functional style with polled consumers in Spring Cloud Stream?
You are using outdated documentation. The most current is available from the project site - https://spring.io/projects/spring-cloud-stream#learn.
The section you are looking for is - https://docs.spring.io/spring-cloud-stream/docs/3.1.5/reference/html/spring-cloud-stream.html#spring-cloud-streams-overview-using-polled-consumers
When creating a new Spring Boot project using Spring Initializr and adding Azure Support, it adds a dependency to com.microsoft.azure:azure-spring-boot-starter.
implementation 'com.microsoft.azure:azure-spring-boot-starter'
The spring cloud documentation says:
The Azure Support entry contains auto-configuration support for Azure
managed services [...]
Now I'd like to send and receive messages from Azure Service Bus and the documentation wants me to add a dependency to azure-servicebus.
implementation 'com.microsoft.azure:azure-spring-boot-starter'
implementation 'com.microsoft.azure:azure-servicebus'
Okay fine. When I now switch over to Microsoft and read the documentation about Service Bus there, it mentions two libraries, where Microsoft states the second one is dated and legacy.
azure-messaging-servicebus (latest) implementation 'com.azure:azure-messaging-servicebus:7.0.0'
azure-servicebus (legacy) 'com.microsoft.azure:azure-servicebus'
Question
Is the Spring Boot documentation just outdated?
I have read somewhere that libraries in com.microsoft.com are for managing the resource itself, while libraries in com.azure are for managing the data. Is this true?
What is meant by auto-configuration support?
As you can see I am very confused which dependencies I need to add. Also I don't get the difference between packages from com.azure and com.microsoft.azure.
Can someone please shed some light on this?
I've been trying to do some searching (Google, Slack, Stack) and have yet to find an answer. We have some applications that are written using Spring Cloud Streams and were interested in swapping the back end from Kafka to Pulsar. Spring currently doesn't have any native support for Kafka, however it appears that pulsar provides the ability to use the Kafka API to communicate with pulsar directly (https://pulsar.apache.org/docs/en/adaptors-kafka).
I'm wondering if anybody has gone down the route of trying to use this replacement for the Kafka-clients library in the context of Spring cloud messaging.
An alternative valid approach, of course, is to just re-write the code - but I wanted to turn to the community to see if anybody has gone down this road as of yet.
Thanks
While not exactly what you're asking for, I've tried to integrate Pulsar with Spring-Kafka using the shaded library org.apache.pulsar:pulsar-client-kafka:2.5.0 and it blows-up with this stack trace:
Caused by: java.lang.UnsupportedOperationException: null
at org.apache.kafka.clients.consumer.KafkaConsumer.partitionsFor(KafkaConsumer.java:650) ~[pulsar-client-kafka-2.5.0.jar:2.5.0]
at org.springframework.kafka.listener.AbstractMessageListenerContainer.checkTopics(AbstractMessageListenerContainer.java:312) ~[spring-kafka-2.2.10.RELEASE.jar:2.2.10.RELEASE]
at org.springframework.kafka.listener.ConcurrentMessageListenerContainer.doStart(ConcurrentMessageListenerContainer.java:136) ~[spring-kafka-2.2.10.RELEASE.jar:2.2.10.RELEASE]
at org.springframework.kafka.listener.AbstractMessageListenerContainer.start(AbstractMessageListenerContainer.java:292) ~[spring-kafka-2.2.10.RELEASE.jar:2.2.10.RELEASE]
at org.springframework.kafka.config.KafkaListenerEndpointRegistry.startIfNecessary(KafkaListenerEndpointRegistry.java:311) ~[spring-kafka-2.2.10.RELEASE.jar:2.2.10.RELEASE]
at org.springframework.kafka.config.KafkaListenerEndpointRegistry.start(KafkaListenerEndpointRegistry.java:255) ~[spring-kafka-2.2.10.RELEASE.jar:2.2.10.RELEASE]
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:182) ~[spring-context-5.1.10.RELEASE.jar:5.1.10.RELEASE]
This specific error can be bypassed (for these specific library versions) using a custom implementation of AbstractMessageListenerContainer.checkTopics but you'll likely run into more issues.
I'm tasked with the requirement of using Spring Cloud Stream (with Kafka bindings). Going through Spring Cloud Stream's Quick Start and judging from the Config file it uses, LogSinkConfiguration, it seems the recommendation here is to use Spring Integration patterns, e.g. #ServiceActivator(inputChannel = Sink.INPUT) to connect to the input channel. However all the tutorials I've found are using a different set of annotations from the Spring Cloud Stream library (vs Spring Integration), i.e. #StreamListener(Processor.INPUT) in this walk-through.
So which is better/newer/preferred (i.e. what's the "best practice"). Should I use the S.I. way of configuring the sink using #ServiceActivator(inputChannel = Sink.INPUT) or the alternative #StreamListener(Processor.INPUT). Are these two methods of hooking up the sink virtually the same, or are there advantages to using one over the other?
With 2.0 we introduced spring-cloud-function support and with the imminent release of 3.0 we are now fully committed to functional support over any annotation driven (Spring Integration included).
So, to answer your question, functional support is the preferred way. Here is the link to more info and feel free to follow up with more concrete questions.
Is there a Spring Integration endpoint which connects to Gigaspaces?
As a general point, I am also interested to know what is the best documentation for using Spring together with Gigaspaces. Am surprised that there does not appear to be a lot of material written on this. Is Gigaspaces still the preferred option for scaling Spring applications, or are there better solutions?
GigaSpaces XAP using Spring as its native configuration. XAP container running Spring container internally.
Every XAP component (data grid node , data grid proxy , event handler ....) exposed via Spring. IDE integration and Unit tests done via Spring application context.
You can deploy spring app as is into XAP. XAP will scale it and make it HA.
See more:
http://docs.gigaspaces.com/xap102tut/spring-integration.html
http://docs.gigaspaces.com/sbp/spring-data.html
Is the question is "How my application based on spring-integration can easily push objects into GigaSpaces XAP?" or "How can I use the spring-integration framework from my code deployed in GigaSpaces XAP (so collocated with the data)?"
For the first question, I am unaware of any off-the-shelf end-points. But it is very easy to program: you will have to decorate your pojo (by annotation for example, to say where are indexes etc.)
You can use JMS integration (use GS XAP as a JMS broker), but I don't think it is the best way here...
For the second question, a GigaSpaces XAP application is mainly a Spring context. By default, there is no use of Spring integration, but it is very very easy to integrate as we have already in a Spring stack.