in my current project we are using Spring Boot and RabbitMq for some of the internal microservice Communication.
We are currently defining Queue Properties in both services, that publish/listen to this queue. Additionally, we define the exchange only in the publisher service.
However, to make it more maintainable, I would like to find a setup/best practice to define the queue once and all relevant services can rely on it.
So far, i checked the AsyncAPI Project and considered creating an extra library to outsource the configs there.
What is the best practice here or how do you do it in your projects?
So far, i checked the AsyncAPI Project and considered creating an extra library to outsource the configs there.
In my opinion, there isn't a general good practice. you can try to define YOUR practice with your team; you can try a strategy and fail fast.
If your system is "static" (i hope not) you can define the configs with definitions file. I dont like this, but is a possible approach. You could try this solution for DEV ENV with docker
In our team, each workload define its configs and rabbitmq team create VHOST and its configs with devops approach.
Related
I have an application with file-supplier Srping Cloud module included. My workflow is like track file creating/modifying in particular directory->push events to Kafka topic if there are such events. FileSupplierConfiguration is used to configure this file supplier. But now I have to track one more directory and push events to another relevant Kafka topic. So, there is an issue, because there is no possibility to include multiple FileSupplierConfiguration in project for configuration another file supplier. I remember that one of the main principles of microservices for which spring-cloud-stream was designed for is you do one thing and do it well without affecting others, but it still the same microservice with same tracking/pushing functionality but for another directory and topic. Is there any possibility to add one more file supplier with relevant configuration with file-supplier module? Or is the best solution for this issue to run one more application instance with another configuration?
Yes, the best way is to have another instance of this application, but with its own specific configuration properties. This is really how these functions have been designed: microservice based on the convention on configuration. What you are asking really contradicts with Spring Boot expectations. Imaging you'd need to connect to several data bases. So, you only can have a single JdbcTemplate auto-configured. The rest is only possible manually. But better to have the same code base which relies on the auto-configuration and may apply different props.
I have a few microservices where each microservice has REST endpoints for CRUD operations.
I have to create a workflow that will start from one microservice with some initial input, but later outputs from a microservice can be used as input to other microservices. There can be some synchronous and asynchronous calls to these REST APIs.
I have looked for some of the workflow-engines but I do not think that I can create my workflow without writing any java code.
Should I write a separate microservice just for microservices orchestration? This orchestration microservice will know the exact workflow and can be configurable for inputs required to start the workflow, and it can also use some third-party workflow engines like Camunda to store the definition of the workflow.
Is this correct thinking to have a separate microservice just for microservices orchestration? Till now the existing microservices have no idea about the other microservices. There could be a chance that output from one microservice needs to be massaged before using as input for other microservice.
I have looked for some of the workflow-engines but I do not think that
I can create my workflow without writing any java code.
This depends on your business processes and the complexity of your workflow. Usually yes you will need to write some code to achieve it.
Should I write a separate microservice just for microservices
orchestration? This orchestration microservice will know the exact
workflow and can be configurable for inputs required to start the
workflow, and it can also use some third-party workflow engines like
Camunda to store the definition of the workflow.
Yes you can do that. I did something similar on a system using micro-services. This would be a very good Idea on the long run as you could configure your workflow based on environments as well. For example on your development machine you would have a little different workflow/configuration. This is practical for Developers or QA's testing their solutions. On the other hand on Staging/Production you can pre-define Customer setups/orchestration which you can reuse any time if you get new customers or users.
Is this correct thinking to have a separate microservice just for
microservices orchestration? Till now the existing microservices have
no idea about the other microservices. There could be a chance that
output from one microservice needs to be massaged before using as
input for other microservice.
Yes you can do that without problems although I would be careful with the name orchestration as this has another meaning in context in micro-service architecture(Docker, Docker-Swarm, Kubernetes). Similar examples would be some kind of EndToEndTest or Cross micro-service testing-micro-service. That would test cross micro-service business operations and assert the results. Usually business operations involve more then 1 micro-service so in order to test that you can use this approach. This micro-service would call APIs from multiple micro-services and test the results and scenarios based on your Business rules. Another example would be something like seeder-micro-service(which seems to be very similar to what you are trying to do here). This seeder-micro-service would be responsible for seeding(creating) test data to your micro-services. This test data is some basic setup/configuration data which you need in order to have your micro-service business processes to work. This is very handy for development machines or some test environments where you need to quickly setup an environment. Using this seeder-micro-service you can easily setup do your work or tests and dispose the environment(data) as you need it. This is especially useful for development machines setups but it can also be used on shared test environments and etc. Both of those examples are micro-services which server your needs and make your life easier to work with your system.
One final note regarding this:
Till now the existing microservices have no idea about the other
microservices.
They should be abstracted from each other in a way that they are not aware of internal implementation or data(separate databases) but they should communicate between each other in order to perform business operations which sometimes are cross micro-services. Like the typical example of payment-micro-service and order-micro-service from an online shop example. So it is fine that they know about each other and communicate but this communication has to be very carefully designed in order to avoid some common pitfalls.
They usually communicate with each other with direct calls over HTTP or some other protocol or through some message queue like Apache Kafka or RabbitMq or others. You can read more about it in this answer.
Yes, you should cover the orchestration part in a separate service. And, yes, go with a BPMN 2 process engine as orchestrator, as you already suspected. Yes, this may include writing a little code mostly for data mapping or connectors.
Benefits include for instance ootb support for:
state management and long running processes / persistence for data
versioning (!)
retries and error handling
tooling to modify state and date in case something went wrong
timeouts, parallel execution (if necessary)
scalability
graphical process model
audit trail
and end to end visibility in monitoring tools based on BPMN 2 model
ability to include business rules tasks (DMN) for more complex rules
combination of push and pull communication pattern and a/sync communication
business-IT alignment via BPMN 2
support for the various BPMN 2 events
standardization (skills, security, software quality, features)
...
This is a great related article about the WHY using an airline ticket booking as an example:
https://blog.bernd-ruecker.com/3-common-pitfalls-in-microservice-integration-and-how-to-avoid-them-3f27a442cd07
This is about the important design consideration in case you go with a process engine:
https://blog.bernd-ruecker.com/the-microservice-workflow-automation-cheat-sheet-fc0a80dc25aa
I faced a similar problem as in the original question. I'd a few microservices with simple dependencies that needed to be managed and did go down the path of writing my own microservice https://github.com/pedro-r-marques/workflow to manage the dependencies and do the orchestration. It uses a yaml definition file to describe the dependencies and rabbitmq for message passing. One could also replace the usage of rabbitmq by REST API calls front-ended by a load-balancer.
Let me preface this with the fact that I am still very new to Apache Camel. I'm still trying to understand how it all works, and what needs to be done (and HOW to do it) to achieve a particular effect.
I am trying to develop a Spring Boot application that will use Apache Camel to handle the transmission (and possibly also receipt) of data to/from a number of possible sources and destinations. The purpose of the application is to provide a means to produce/generate network traffic, at the network application level, that will be fed into another Spring Boot application - let's call this the target. We are trying to observe and measure the effects various network loads have on the target.
We would like to be able to transmit data via a number of protocols, including: ftp, http/s, file systems (nfs), various mail protocols (smtp, pop) and data streaming protocols for voice and video. There may be other protocols added at a later time. The data itself is irrelevant, we just need to be able to transmit data via various protocols with various loads.
These applications/services will be running in a containerized environment (Docker) that will be run within our local development and test environment, as well as possibly in a cloud environment, such as AWS. We have used Docker, Ansible, Terraform and are currently working towards using Kubernetes and Istio to manage the configuration, deployment, and operation of these applications.
We need to be able to provide specific configurations of Camel routes for particular deployments.
It would appear that the preferred method to configure Camel routes is via Java DSL, rather than XML DSL. The Camel documentation and nearly every other source of information I've found have a strong bias towards using Java DSL. Examples of XML DSL route configuration are far and few.
My initial impression is that going the Java DSL route (excuse the pun), would not work well with our need to be able to deploy a Camel application with a specific route configuration. It seems like you are required to have Java DSL defined route configurations hardwired into the code.
We think that it will be easier to provide a specific route configuration via an XML file that can be included in a deployment, hence why I've been trying to investigate and experiment with XML DSL. Perhaps we are mistaken in this regard.
My question to the community is: Considering what I've described above, can the Java DSL approach be used to meet the requirements as I've described them? Can we use Java DSL in a way that allows for dynamic route configuration? Keep in mind we would not be attempting to change configuration during operation, just in the course of performing a deployment.
If Java DSL could be used for this purpose, it would be very much appreciated if pointers to documentation, examples, etc. could be provided.
For your use cases you could use XML DSL also. Anyhow below book covers most aspects Camel development with examples. In this book authors describes XML DSL use for most of java DSL examples.
https://www.manning.com/books/camel-in-action-second-edition
In below github repository you can find the source code for all the examples listed in above book.
https://github.com/camelinaction/camelinaction2
Simple tutorial and github repository for Apache Camel using Spring boot.
https://www.baeldung.com/apache-camel-spring-boot
https://github.com/eugenp/tutorials/tree/master/spring-boot-modules/spring-boot-camel
Maven Plugin for build and deployment of spring boot container application into Kubernetes cluster
https://maven.fabric8.io/
In case if your company can afford some funding for your effort look at below link which provides commercial offerings around Camel.
https://camel.apache.org/manual/latest/commercial-camel-offerings.html
Thanks
Madhu Gupta
Our team has a few projects which use the Java DSL for building routes. In order to make them dynamic, there are control structures for iterating and setting endpoints based off configurations. That works for us because the routes are basically all the same, just with different sources and sinks.
If you could dynamically add/change the XML DSL files in a way that doesn't involve redeploying your application, that might be a viable route to follow. One might, for example, change the camel.springboot.xml-routes property to point to a folder which changes as needed.
It's well understood that in a microservices architecture, configuration must be externalized.
Tools like zookeeper, etcd or consul are excellent options to store that configuration. However a new layer on top of those services is required in order to provide new functionalities that are fundamental in a configuration server. Ex. versioning; change history; "draft" / published configuration, etc...
I've found spring config server, which is an interesting project and addresses all these concerns using git for handling the above mentioned requirements. However, I'd like avoid using git due to additional required setup. ex. replication, etc...
Do you know any other options other then spring config server?
Our findings have been the same - configuration must be externalized and the notion of "code as config" through the lessons learned from implementing time consuming Puppet/Chef systems.
We're building a microservices and API orchestration system at LunchBadger. We also uses git - but it's encapsulated into our system that we provide as a service, because we want configuration to be externalized AND dovetail into whatever CI/CD pipeline infrastructure you may have or are looking to adopt. We also provide visualization on top of microservices and APIs so that you can get an idea of the topology of your once monolithic app in the form of many microservices.
You can look into Microconfig.IO and it's Microconfig Server. It's deployed via docker and configured via 3 env vars. It still use git repo for config storage, but that's actually a proper way to go. As a bonus you get a powerful templating functionality designed explicitly to handle app configuration
We are writing a new set of services and have decided to make them share a common interface... calling it a BaseService. The idea is that whenever anyone wants to develop a new service in our organization, they should be just able to extend and use this BaseService.
We have written a few other classes which also form a part of this base jar, it does things like handle transactions and connect to database using hibernate etc.
Right now all the services that extend the BaseService are a part of the same project (Eclipse + Maven), and some of the services are dependent on each other, but because they are in the same project we don't have a problem with dependencies.However, we expect 40-50 services to be written which would extend base service and would also be interdependent.
I am worried that the size of the base project would be huge and that just because when someone has to use one service they might have to depend on my base jar which has 50 services.
Is there a way that we can make some projects dynamically dependent on others?
Lets say I have a service A which depends on service B, when I build/compile Service A,it should be able to realize that it has a dependency on service B and automatically use the Service B jar.
I have heard of OSGi, will it solve my problem or is there a way I can do it with Maven or is there a simpler solution ?
Sorry about the long post !
Thanks in advance for your suggestions
It doesn't make any sense to "dynamically" manage project dependencies, since build dependencies are by definition not dynamic.
Your question, at least for the moment, seems to be entirely about how to build your code rather than about how to run it. If you are interested in creating a runtime system in which new service implementations can be dynamically added then OSGi may be the solution to look at. An extra advantage here would be that you could enforce the separation of API from implementation, and prevent the implementing services from invalidly depending on parts of your core module that you do not want them to have visibility of.
You could also use OSGi to manage evolution of your core service API through versioning; for example how do you communicate the fact that a non-breaking change has been made to the API versus a breaking change etc.
I would say there are two options depending if i understand your question correct. First one. You have already defined an interface (java term) and now you have different implementations of that. The simple solution for Maven would be to a have a module which is called for example: service-api and than this will be released and can be used by others as dependencies. On their side they simply implement the interface. No problem with the dependencies. If you are more talking about OSGi than you should take a look to maven-tycho.