I am planning to implement a data processing pipeline using the Microservice Architecture in Java. Since I have been using Spring boot to implement some of the Microservice already, I am investigating the rest of Spring components to see which one can help me to address my requirements. I have found out Spring Cloud Stream helps me to implement my solution using the event-driven approach, but I am not quite sure whether Spring Data Flow acts as a Single Point of Failure and does the orchestration of the microservices or it actually acts as a tool to help with visualizing the pipeline and deployment to Kubernetes, Cloud Foundry, etc. and still work passively outside the pipeline. Therefore, if it is failed for any reason, the pipeline can work without any interruptions.
To clarify what I am asking for, does the combination of Spring Cloud Stream and Spring Cloud Data Flow provide an almost similar approach to what this article refers to it as a best of both worlds of Orchestration and Choreography?
https://www.sapientglobalmarkets.com/blog/big-data-pipeline-orchestration-choreography
Related
I have to implement 4 microservice in spring boot , spring cloud and there is a central microservices that will handle the workflow between these 4 services like first service call then on response back to central one and then central one will make call to second service and so on . my question is how to implement (best possible way)central microservice avoiding if/else ?
Since you're already using Spring I suggest you have a look at Spring Statemachine.
You don't give much details but I think it sounds like it'd be a good fit.
I'm going to begin a new project of microservice and want know about patterns that I might wan't to implement on my new microservice for that I could scale, don't have problems with load balance, have an good service with resilience.
I looking to know about patterns that might microservices should implement
Also could you provide some indications for Java microservices application?
For solve some problems that was said: [1]
Architecture: You could build small microservices as it can be, so that the scale be right focused on need to be.
You could set a Discovery service for registering the applications instances, with that making easier to the client access the right application when are lots of instances of the same service (Netflix Eureka / Spring Cloud).
You could set a Configuration outside the code source of the code, so that all the instances can be configured without human intervention right on the running code (Spring Cloud Config).
Set a way the check the health of the microservice with ease.
I circuit breaker for clear the anomalies microservices instances (Spring Cloud / Netflix Hystrix).
and following as say the Twelve-Factor App.
With lots of another things that you need to that care when building a microservice, suggesting the reading the book, if you'll use Java for building microservices Spring Microservices in Action By John Carnell , if you i'll implement in another programming language this other could guide you, Microservices in Action by Morgan Bruce and Paulo Pereira.
[1] (https://www.amazon.com.br/Spring-Microservices-Action-John-Carnell/dp/1617293989
I am trying to find some examples of Spring Cloud Data Flow and Azure Service Bus setup. .
I have found https://github.com/microsoft/spring-cloud-azure/tree/master/spring-cloud-azure-stream-binder/spring-cloud-azure-servicebus-topic-stream-binder but it is still in RC and I do not see any examples (which cover Spring Cloud Data Flow) there.
Could you please help me to understand if I can use Spring Cloud Data Flow and Azure Service Bus together?
I was able to run examples with Kafka and RabbitMQ, but I cannot find anything about Azure Service Bus that can be used as the integration solution for Spring Cloud Data Flow
Spring Cloud Data Flow doesn't necessarily need to know what messaging layer you choose to run your applications on. Hence, your question is more likely related to how one can run the application with Azure Service Bus using Spring Cloud Stream.
This is one such a sample: https://learn.microsoft.com/en-us/java/azure/spring-framework/configure-spring-cloud-stream-binder-java-app-azure-event-hub?view=azure-java-stable
Once the application is built using Spring Cloud Stream (using the Azure event hub binder), you can then manage these applications using Spring Cloud Data Flow as Streaming applications.
I am new to Spring cloud data flow. Trying to figure out what it mean by registering, creating a task on spring cloud data flow server vs running a spring class with annotation #EnableTask
Any clarification to understand what these two are, will be helpful!
With Spring Cloud Task, you'd build and test standalone Task microservices in isolation. With the programming model, you can address use-cases ranging from ETL/ELT, data migration, or predictive-model-training etc.
Once you have an "n" no. of such applications, you could use Spring Cloud Data Flow's DSL to compose them into coherent Task/Batch pipelines. To use them in the DSL, you'd have to first register the coordinates of the standalone Task Apps.
Refer to the SCDF's Task overview and as well as the Task Developer Guide for more details - you could repeat it locally to follow along and learn the mechanics.
I'm trying to learn SI (Spring Integration) but i'm a bit confused on the real usage of this.
As first example i would like to interact with a WebService but i dont understand what could be the difference from
Invoke a WebService Using SI
Invoke a Webservice using RestTemplate
So, what is the benefit of using SI (in Webservice context, or in other context)?
Looking the web, i havent find an article that explain:
Usually you will do in this way....
With SI you can do better - in this another way - and the benefit are....
To be more explicit on what i have to realize, this is an example:
1) I have to write an application (Standalone application) that have to collect some data in the system periodically and then invoke a Web Service that will store it.
2) The Web Service receive the call from the "clients" and store in the database.
My webservice will use REST.
The reason because i've think to use SI is that the Standalone Application should interact with different system
Webservice in first instance
A Web Mail, if the webservice is not achievable
File system if Web mail is not achievable too
If you only need to pull some data in a simple way and push it onwards to a REST service this does not "justify" the use of Spring Integration. A simple Spring (Boot) application combined with a scheduler will be sufficient.
But if you want to use a more complex data source for which an endpoint is available, you need transformations, complex and flexible routing is a high priority or even Enterprise Integration Patterns (EIP) then Spring Integration is for you. Have a look at the Overview and decide if it mentions something you consider as valuable to you.
Perhaps you will create additional value by mixing in Spring Batch if you need to process a lot of data.
But as I understand your current demand starting with just a RESTTemplate should do for the moment. Starting small will not prevent you from switching to Spring Integration later on.
Have a look at the various tutorials and guides provided by the Spring Boot project. There is even an example for Spring Integration.