It's well understood that in a microservices architecture, configuration must be externalized.
Tools like zookeeper, etcd or consul are excellent options to store that configuration. However a new layer on top of those services is required in order to provide new functionalities that are fundamental in a configuration server. Ex. versioning; change history; "draft" / published configuration, etc...
I've found spring config server, which is an interesting project and addresses all these concerns using git for handling the above mentioned requirements. However, I'd like avoid using git due to additional required setup. ex. replication, etc...
Do you know any other options other then spring config server?
Our findings have been the same - configuration must be externalized and the notion of "code as config" through the lessons learned from implementing time consuming Puppet/Chef systems.
We're building a microservices and API orchestration system at LunchBadger. We also uses git - but it's encapsulated into our system that we provide as a service, because we want configuration to be externalized AND dovetail into whatever CI/CD pipeline infrastructure you may have or are looking to adopt. We also provide visualization on top of microservices and APIs so that you can get an idea of the topology of your once monolithic app in the form of many microservices.
You can look into Microconfig.IO and it's Microconfig Server. It's deployed via docker and configured via 3 env vars. It still use git repo for config storage, but that's actually a proper way to go. As a bonus you get a powerful templating functionality designed explicitly to handle app configuration
Related
in my current project we are using Spring Boot and RabbitMq for some of the internal microservice Communication.
We are currently defining Queue Properties in both services, that publish/listen to this queue. Additionally, we define the exchange only in the publisher service.
However, to make it more maintainable, I would like to find a setup/best practice to define the queue once and all relevant services can rely on it.
So far, i checked the AsyncAPI Project and considered creating an extra library to outsource the configs there.
What is the best practice here or how do you do it in your projects?
So far, i checked the AsyncAPI Project and considered creating an extra library to outsource the configs there.
In my opinion, there isn't a general good practice. you can try to define YOUR practice with your team; you can try a strategy and fail fast.
If your system is "static" (i hope not) you can define the configs with definitions file. I dont like this, but is a possible approach. You could try this solution for DEV ENV with docker
In our team, each workload define its configs and rabbitmq team create VHOST and its configs with devops approach.
I have some properties in my microservice. They are changed a lot so I want to store them somewhere out of microservice so that I don't have to undeploy it again and again. One solution is to store them in database but it will be less efficient that way. Can you advise me a solution where I can store them?
Basically this microservice is used by a vast number of people. I want that file to be read ONCE when microservice is deployed (unless or until there is some change in the file) to minimize the calls.
You should use Spring Cloud Config which enables to use a centralized server which exposes a Git Repo which you can use to store environment specific configuration files (application.properties)
You should go with Centralized configuration, spring cloud supports config server, It is highly recommended for microservice architecture.
For reference: https://spring.io/guides/gs/centralized-configuration/
Basically, you have a centralized place to store all the configurations and don't need the redeploy the application every time you change the configs.
NOTE: in case you change the db properties then probably you need the restart the service
I am implementing a Spring Boot application which will be providing a multitenant environment. That is achieved in my case by using a database schema for each customer. Example see this project.
Now I am wondering how to implement tenant-specific configurations. I am using #ConfigurationProperties to bundle my property values, but these are getting instantiated once and not for each tenant.
What if I would like to use Spring Cloud Config with multiple tenant specific git repository as an configuration backend. Would it be possible when using a jdbc backend for Spring Cloud Config?
Is there any way with default Spring mechanisms or do I have to implement a database based configuration framework myself?
Edit: For example I have two tenants called Tenant1 and Tenant2. Both are running over the same application in the same context and are writing in the database schemes tenant_1 and tenant_2.
Identification of tenants is happening over keycloak (see Spring Keycloak multi tenant example). So I identify the tenantId from the jwt token and select the database connection like described here.
But now I would need the same mechanism for #Configuration beans. Since #Configuration beans are as far as I know Singletons, so there is always ONE configuration per application scope, and not ONE configuration per tenant.
So using Spring Cloud Config Tenant1 is using https://git-url/tenant1, Tenant2 is using Hashicorp Vault as backend and perhaps Tenant3 will be using a jdbc based configuration backend. And all of that in ONE (of course scalable) application.
In case your application uses tenant specific files (html templates etc), the following can be applied. As I have used the below approach for handling many tenants and works fine and easy to maintain.
I would suggest that you maintain a consistent configuration source (JDBC) for all of your tenant configurations. This helps you have a single source that is cacheable and scalable for your application. Also, you could have your tenants navigate to a configuration page to manage their settings and alter them to suit their needs at any point of time on the fly. (Example Settings: Records Per Page, Theme, Logo, Filters etc...)
Having the tenant configuration in files in git will be a difficult task when you wanted to auto-provision tenant's when they sign-up as it will involve couple of distributed services. Having them in a TenantSettings table with the tenantId as a column could help you get the data in no time and will be easy.
You can use Spring Cloud Config for your scenario and it is adoptable. It is easily configurable and provides out of the box features. For your specific scenario, you can have any number of microservices running yet all controlled by one Spring Cloud Config Server which is connected to one Git Repository. Your all microservices are asking configuration properties from Spring Cloud Config Server and it is directly fetching properties from Git Repository. That repository can have multiple property files. It can hold common properties for all the microservices or specific service based configuration properties. If you want to keep confidential properties more securely, that is also made possible via HashiCorp vault. I will leave an image below for you to get a better idea about this concept.
In the below image, you can see the Git Repository with common configuration property files and specific configuration property files for different services yet in same repository.
I will add another image for you to get a better idea how does this can be arranged with application profiles as well.
Finally I will add something additional to show the power of Spring Cloud Config and out of the box features it allows us to play with. You can automatically refresh configuration properties in running application as well. You can configure Spring Cloud Config to do that. I will add an architectural diagram to achieve that.
References for this answer is taken from Spring in Action, Fifth Edition
Craig Walls
Let me preface this with the fact that I am still very new to Apache Camel. I'm still trying to understand how it all works, and what needs to be done (and HOW to do it) to achieve a particular effect.
I am trying to develop a Spring Boot application that will use Apache Camel to handle the transmission (and possibly also receipt) of data to/from a number of possible sources and destinations. The purpose of the application is to provide a means to produce/generate network traffic, at the network application level, that will be fed into another Spring Boot application - let's call this the target. We are trying to observe and measure the effects various network loads have on the target.
We would like to be able to transmit data via a number of protocols, including: ftp, http/s, file systems (nfs), various mail protocols (smtp, pop) and data streaming protocols for voice and video. There may be other protocols added at a later time. The data itself is irrelevant, we just need to be able to transmit data via various protocols with various loads.
These applications/services will be running in a containerized environment (Docker) that will be run within our local development and test environment, as well as possibly in a cloud environment, such as AWS. We have used Docker, Ansible, Terraform and are currently working towards using Kubernetes and Istio to manage the configuration, deployment, and operation of these applications.
We need to be able to provide specific configurations of Camel routes for particular deployments.
It would appear that the preferred method to configure Camel routes is via Java DSL, rather than XML DSL. The Camel documentation and nearly every other source of information I've found have a strong bias towards using Java DSL. Examples of XML DSL route configuration are far and few.
My initial impression is that going the Java DSL route (excuse the pun), would not work well with our need to be able to deploy a Camel application with a specific route configuration. It seems like you are required to have Java DSL defined route configurations hardwired into the code.
We think that it will be easier to provide a specific route configuration via an XML file that can be included in a deployment, hence why I've been trying to investigate and experiment with XML DSL. Perhaps we are mistaken in this regard.
My question to the community is: Considering what I've described above, can the Java DSL approach be used to meet the requirements as I've described them? Can we use Java DSL in a way that allows for dynamic route configuration? Keep in mind we would not be attempting to change configuration during operation, just in the course of performing a deployment.
If Java DSL could be used for this purpose, it would be very much appreciated if pointers to documentation, examples, etc. could be provided.
For your use cases you could use XML DSL also. Anyhow below book covers most aspects Camel development with examples. In this book authors describes XML DSL use for most of java DSL examples.
https://www.manning.com/books/camel-in-action-second-edition
In below github repository you can find the source code for all the examples listed in above book.
https://github.com/camelinaction/camelinaction2
Simple tutorial and github repository for Apache Camel using Spring boot.
https://www.baeldung.com/apache-camel-spring-boot
https://github.com/eugenp/tutorials/tree/master/spring-boot-modules/spring-boot-camel
Maven Plugin for build and deployment of spring boot container application into Kubernetes cluster
https://maven.fabric8.io/
In case if your company can afford some funding for your effort look at below link which provides commercial offerings around Camel.
https://camel.apache.org/manual/latest/commercial-camel-offerings.html
Thanks
Madhu Gupta
Our team has a few projects which use the Java DSL for building routes. In order to make them dynamic, there are control structures for iterating and setting endpoints based off configurations. That works for us because the routes are basically all the same, just with different sources and sinks.
If you could dynamically add/change the XML DSL files in a way that doesn't involve redeploying your application, that might be a viable route to follow. One might, for example, change the camel.springboot.xml-routes property to point to a folder which changes as needed.
I have been playing with Spring Cloud Config and like many of the ideas I see there. I would like to better understand how its creators intended on it being used though.
Lets say that I have several services that support a larger API. Because these services are independent from each other, their source is managed in separate repositories. This allows us to version them and deploy them separately from one another. Today, their properties are managed individually.
I like the idea of having a single config server provide all of the configuration information for the individual applications/services that support this larger API. Looking at the default implementation of EnvironmentRepository (which is GIT based), I would have to have a single repository with all of my application config files in it. Because they all live within the same repository, they would all be managed/versioned together in a single place.
How do I make both models work with each other? Would it be better to have a repository per application instead of one for all applications? What are your thoughts?
-Joshua
It might just be a detail of the implementation of the EnvironmentRepository. See here for some discussion on how and when that might happen.