I have worked with spring cloud data flow and rabbitmq and kafka but i would like to know if it would be possible to install scdf with google pub/sub.
I don't want to create a stream (new app spring cloud stream) with source or sink to gcp i want google pub/sub over spring cloud data flow server to use as an intermediate messaging broker.
Any suggestions?
Yes, you can use Spring dataflow and GCP Pub Sub link.
Through the use of one of many Spring Cloud Stream Binders, many
different messaging middleware products can be used. The following
popular platforms are among the list supported by Spring Cloud Data
Flow:
1.Kafka Streams
2.Amazon Kinesis
3.Google Pub/Sub
4.Solace PubSub+
5.Azure Event
Hubs You can find more information like the the current list of Spring
Cloud Stream Binders here library as a dependency to the application.
Spring Cloud Stream supports Google PubSub (partner maintained) binder implementation link.
Here you can find a related SO question Spring dataflow and GCP Pub Sub.
Related
Following documentation, there are multiple ways to integrate Google Cloud PubSub events with Spring application:
Spring Cloud GCP has several modules for sending messages to Pub/Sub
topics and receiving messages from Pub/Sub subscriptions using the
Spring Framework. You can use these modules independently or combine
them for different use cases:
Spring Cloud GCP Pub/Sub Starter lets you send and receive messages
using helper classes and call the Pub/Sub Java client library for more
advanced scenarios.
Spring Integration Channel Adapters for Pub/Sub
let you connect Spring Integration Message Channels with Pub/Sub.
Spring Cloud Stream Binder for Pub/Sub lets you use Pub/Sub as
messaging middleware in Spring Cloud Stream applications.
I don't have full understanding - what are those different use cases mentioned, and how to determine, which module is best for which use case?
Application (Dockerized Spring Boot app, deployed to Kubernetes in GCP) I am working on is rather simple, it is expected to act upon received PubSub event, it is not going to publish any events itself.
Spring Cloud GCP Pub/Sub Starter module contains the java client classes for pub sub which will be used by your spring application to perform administrative and functional operations (ie. sending and receiving messages).
Spring Integration Channel Adapters for Pub/Sub module is utilized when your spring application uses Message Channels. This module will help routing message between message channel and pub/sub using channel adapters.
Spring Cloud Stream Binder for Pub/Sub module is used in Spring Cloud Stream Applications in order to utilize cloud Pub/Sub API.
Since, your application requirements are basic you can easily go for Spring Cloud GCP Pub/Sub Starter module. For more information you can refer to this Google documentation.
I found these two docs in Spring Azure Documentation
https://learn.microsoft.com/en-us/azure/developer/java/spring-framework/configure-spring-cloud-stream-binder-java-app-with-service-bus
https://learn.microsoft.com/en-us/azure/developer/java/spring-framework/configure-spring-boot-starter-java-app-with-azure-service-bus
What is the difference between them? If there are other options, please suggest them!
I am trying to find some examples of Spring Cloud Data Flow and Azure Service Bus setup. .
I have found https://github.com/microsoft/spring-cloud-azure/tree/master/spring-cloud-azure-stream-binder/spring-cloud-azure-servicebus-topic-stream-binder but it is still in RC and I do not see any examples (which cover Spring Cloud Data Flow) there.
Could you please help me to understand if I can use Spring Cloud Data Flow and Azure Service Bus together?
I was able to run examples with Kafka and RabbitMQ, but I cannot find anything about Azure Service Bus that can be used as the integration solution for Spring Cloud Data Flow
Spring Cloud Data Flow doesn't necessarily need to know what messaging layer you choose to run your applications on. Hence, your question is more likely related to how one can run the application with Azure Service Bus using Spring Cloud Stream.
This is one such a sample: https://learn.microsoft.com/en-us/java/azure/spring-framework/configure-spring-cloud-stream-binder-java-app-azure-event-hub?view=azure-java-stable
Once the application is built using Spring Cloud Stream (using the Azure event hub binder), you can then manage these applications using Spring Cloud Data Flow as Streaming applications.
I have created topic and subscription in google pubsub. How to read the google pub sub message from spring boot application(Subscriber? Also how to make always listening that subscription from my application to pull/read the message.
I would strongly recommend you to take a look at Spring Cloud GCP. It features a Pub/Sub integration that should be just what you're looking for.
There's a sample application here.
I have a spring webapp that communicates with external service over kafka. IS it possible to somehow test contract between those services?
Yes you can. Spring Cloud Contract supports CDC with messaging. If you're using Spring Cloud Stream - the work to be done is trivial. If not then you'll have to implement your own as presented in this issue - Spring Cloud Contract and plain Spring AMQP . Summing it up it's enough for both consumer and producer to implement a custom org.springframework.cloud.contract.verifier.messaging.MessageVerifier bean that will be responsible for receiving and sending of messages via Kafka