I am using spring cloud config server to host a centralized location for all the property files configurations to be used in the project.
I tried using the config files from a local file system using below and it works fine:
spring.profiles.active=native
spring.cloud.config.server.native.searchLocations=file://${HOME}/Documents/test-config/cloud-config-properties/
I also used the git repo using: spring.cloud.config.server.git.uri=ssh://xxxxxx.com:7999/test/cloud-config-properties.git
I would like to try using a combination of this in my project.
Example - for dev/test profile - i would like to use from local filesystem and for the production - I would like to use Git repository.
I enabled both the git uri and native profiles in my application.properties in config server application. But the properties are always picked up from the local file system. Is this possible?
Not supported out of the box, however there is a workaround for this. You can define the basedir for the configuration server, which is where it saves the files it fetches from the remote server, by setting the property (in the config server):
spring.cloud.config.server.git.basedir=<your_dir>
If you are working with docker, you can map this directory to the host filesystem.
Now whatever file you put in there will be picked up by configuration-server if it matches any of the application/profile in the request. For example you could put a file there called application-dynamic.properties, and have all your clients use dynamic as the last profile, for example
spring.profiles.active=systesting,dynamic
This way everything you will put in application-dynamic.properties will override whatever is defined in your config repo.
One thing to notice though is that you need to add the file only after configuartion server starts, because it deletes this folder during startup.
Needles to say, it's not a good practice doing this in production (for example a restart will cause the file to be deleted), but for test/dev this is the best option.
Related
My application.properties files of my Cloud Config Server looks like this.
config.source=Local Cloud Server #just to check where config come from
server.port=8012
encrypt.key=xxxxxxxxxxxxxx
spring.profiles.active=staging
spring.application.name=my-config-server
# Git Backend
spring.cloud.config.server.git.username=MY_USERNAME
spring.cloud.config.server.git.password=ghp_MY-DEV-ACCESS-TOKEN
spring.cloud.config.server.git.uri=https://github.com/my-username/app-config
spring.cloud.config.server.git.clone-on-start=true
spring.cloud.config.server.git.basedir=file://${user.dir}/cloned_configurations
On startup a new folder "cloned_configuration" is created inside of Cloud Config Server folder. I see all of my propertie files cloned form github with correct values.
But for some reason none of these are used. For e.g config.source must have the value "GitHub" because the cloned application.repository has an entry
config.source=GitHub
But on application start I see "Local Cloud Server". All other settings are also not used from cloned properties.
With Postman I can receive all Configs without any issues. But none of theme are used by my Config Server or any of my other webservices.All webservices and the config server using their own application.properties file.
What I do wrong?
You probably need to move your config server properties into your bootstrap.properties file instead of application.properties.
spring.cloud.config.server.git.username=MY_USERNAME
spring.cloud.config.server.git.password=ghp_MY-DEV-ACCESS-TOKEN
spring.cloud.config.server.git.uri=https://github.com/my-username/app-config
spring.cloud.config.server.git.clone-on-start=true
spring.cloud.config.server.git.basedir=file://${user.dir}/cloned_configurations
https://cloud.spring.io/spring-cloud-config/multi/multi__spring_cloud_config_client.html
I have a folder like this in my gitlab
test-project/
DEV/application-testconfig.properties
SYS/application-testconfig.properties
PROD/application-testconfig.properties
appication-commonconfig.properties
These testconfig.properties file contents differ per environment
How do i access the files based on the environment in my spring config server?
Ex:
I would like to use the url like the below for prod, dev:
https://my.configserver.com/my-project/prod/testconfig/master,
https://my.configserver.com/my-project/dev/testconfig/master
also I should be able to access appication-commonconfig.properties
The problem is i am migrating around 30 config files per each environment. the filename in one environment is similar in the other. I can't rename all the files to have "-dev" or "-prod" in it
I've tried with different paths but no luck
I am trying to setup spring config cloud using local file system.
Below is my config on cloud server.
server.port=8888
spring.profiles.active=native
spring.cloud.config.server.native.searchLocations=classpath:/
Bootstrap.properties on client app is as shown below
spring.application.name=hello-world
spring.cloud.config.uri=http://localhost:8888
management.endpoints.web.exposure.include=*
I have also created hello-world.yml on class path for the hello-world spring boot application with property, test: Hello World
Followed below steps to make use of config server.
Step 1: Update the config file and start cloud config server. Able to
see config http://localhost:8888/hello-world/default
Step 2: Start client app hello-world, client app able to read the
test property file from cloud config server.
Step 3: Make changes to config by updating test: Good Bye on
hello-world.yaml.
At this moment, if I check
http://localhost:8888/hello-world/default ,it still shows old value.
Step 4: Run /actuator/refresh on client app. But it won't detect
any change on config server.
The new changes are reflected only if I restart the cloud config server.
Is there any configuration issue causing the cloud config server to unable to listen to changes ?
I could see o.s.cloud.commons.util.InetUtils : Cannot determine local hostname INFO log on cloud config app.
First of all I followed the same exact steps you followed and got the same issue, after almost day of search and study on the matter found out the followings,
we shouldn't use classpath:/<whatever> for
spring.cloud.config.server.native.searchLocations
because when we use so and build the project and run the location refers to the directory inside the generated .jar file, so we will not be able to update it in runtime.
To confirm this you can stop config server, open you .jar archive and delete hello-world.yml file then try http://localhost:8888/hello-world/default you will get default null responses
So we have to use some other locations for spring.cloud.config.server.native.searchLocations either with full directory path or just directory from app running location
Examples
For full path in windows use file:///full-path
spring.cloud.config.server.native.searchLocations: file:///E:\configs
Just for a directory (which will search project root directory if you running from IDE, if running jar then target directory or jar location spring.cloud.config.server.native.searchLocations: configs
spring.cloud.config.server.native.searchLocations: configs\whatever
we can configure multiple locations too as follows, spring.cloud.config.server.native.searchLocations: file:///E:\configs, configs
I am using Spring Config to share in a git server the configuration for some Spring Boot microservices.
It works great but when I am traveling I have to work offline sometimes.
I have configured Spring Config microservice local profile to get the config from my local git, (file:) and not to be HTTP git server, so I can change config and test without needing access to main git server.
The problem is that as I am not able to do a "git push" to push the change to main repository, Spring Confgig notes it and shows this message:
The local repository is dirty or ahead of origin. Resetting it to origin/master.
and resets it deleting my last local commit with last config changes.
How can I make Spring Config just to get the last committed configuration in my local git ignoring if it is pushed or not to the main server?
I had a similar problem.
(the difference is that I am not using Spring Config microservice, but runnging spring configuration server as a stand alone application)
But this should work for you as well:
Instead of launching local configuration server with a local git repository (file:) I have launched it with native spring profile.
spring:
profiles:
active: native
cloud:
config:
server:
native:
searchLocations: file:///path/to/local/git/repository
So now content of local repository is served and no reset is done on access
Actually going through spring-cloud-config source code you will see:
if (!isClean(git, label)) {
this.logger.warn(
"The local repository is dirty or ahead of origin. Resetting"
+ " it to origin/" + label + ".");
resetHard(git, label, LOCAL_BRANCH_REF_PREFIX + label);
}
Solution
So it will always try to do this and the only work around will be similar to this answer:
How spring cloud config use local property override remote property
You will need to commit:
spring:
cloud:
config:
allowOverride: true
overrideNone: false
To the configuration properties you are using. e.g.:local/my-app.properties. If you do have a remote repo for storing all the properties, make sure it is merged to master.
Then you could change any application property as you wish in application.yml/application.properties in your Spring Boot app. It won't get overriden by the remote properties file.
Alternative
Alternatively, you could just remove the config file for the environment you are working on as dev environment. In my example above, you could just remove local/my-app.properties from remote repo and commit master. So that it will never bother overriding local application properties from there because no properties files exist for cloud-config.
Please comment if anything is unclear so that I could improve the instruction.
You can use an identical directory as not only a file system backend and also a git repository for properties files. You should make 2 bootstrap.properties files for config server. One is for the file backend and the other is for the git repository. For example, I want to use a file system backend for develop and a git repository for production. I made bootstarp-dev.properties and bootstrap-prd.properties.
## bootstarp-dev.properties
server.port=8887 // If you want to launch both a config server for dev and a config server for prd, you must assign different ports.
spring.cloud.config.server.native.searchLocations=file://PATH_OF_YOUR_DIRECTORY
## bootstarp-prd.properties
server.port=8888
spring.cloud.config.server.git.uri=YOUR_GIT_URL
If you launch your config server with -Dspring.profiles.active=dev,native parameter, the config sever access your properties files as local files.
If you launch your config server with -Dspring.profiles.active=prd parameter, the config sever access your properties files as git files.
Done.
How can I make Spring Config just to get the last committed configuration in my local git ignoring if it is pushed or not to the main server?
This is what --assume-unchaged is for.
--assume-unchaged
Raise the --assume-unchaged flag on this file so it will stop tracking changes on this file
--[no-]assume-unchanged
When this flag is specified, the object names recorded for the paths are not updated.
Instead, this option sets/unsets the assume unchanged bit for the paths.
When the assume unchanged bit is on, the user promises not to change the file and allows Git to assume that the working tree file matches what is recorded in the index. If you want to change the working tree file, you need to unset the bit to tell Git. This is sometimes helpful when working with a big project on a filesystem that has very slow lstat(2) system call (e.g. cifs).
Git will fail (gracefully) in case it needs to modify this file in the index e.g. when merging in a commit; thus, in case the assumed-untracked file is changed upstream, you will need to handle the situation manually.
We are planning to utilize Spring Cloud Config for our service. Our biggest concern is that when the container starts up, it relies on github to be available all the time so that it can pull the config files. In case github is down, what is the best practice to mitigate the issue?
I was thinking of storing a local folder of the configs as a backup and configuring the application.yml to fallback to it (I do not know how).
I was going to use a Composite Environment Repositories
Please see here: Section 2.1.8
However it states:
Any type of failure when retrieving values from an environment repository results in a failure for the entire composite environment.
This means if the git retrieve fails, it does not fall back to the local component of the composite. I wish it did. Have any of you handled a similar problem? How did you solve it?
Here is a good article about best practices. However, I need a workaround for case 1: Best practices on handling GIT repository inavailability
Spring-Cloud has a configuration property to handle this issue;
spring.cloud.config.server.git.basedir = /your/config/local/fallback/directory
NOTE - If you're using a .yml file, then define the above property as
per yaml conventions.
To have a background knowledge, look at the documentation: http://cloud.spring.io/spring-cloud-static/Finchley.RC1/single/spring-cloud.html#_version_control_backend_filesystem_use
So essentially what happens here is that - as long as your application was initially able to connect to the git repository which you set up in spring.cloud.config.server.git.uri = https://your-git/config-repo.git, then on config-server/container startup, the directory you have defined in your spring.cloud.config.server.git.basedir gets created locally and by default spring-cloud clones your configurations into this directory to be available as fallback.
So whenever your git repository is unreachable, spring-cloud will pick up your configurations from this base directory.
Important things to note:
Unless you really want to re-clone the git configurations only on config-server startup alone, ensure that the property spring.cloud.config.server.git.clone-on-start is NOT set to true or is entirely not set at all - Otherwise, every time you restart your cloud-config service the configurations will be deleted and freshly cloned again and if the repository is not available at the time, application startup will fail - and you perhaps don't want that.
However, if spring.cloud.config.server.git.clone-on-start is set to false or is not even set at all (in which case the default is false), then the git repository will only be cloned on demand - hence if the repository is unreachable, spring-cloud will fallback gracefully to pick up configurations from the spring.cloud.config.server.git.basedir
Even when the application config-server (or its container) is restarted and the git repository is not reachable, you will see something like below;
No custom http config found for URL: https://your-git/config-repo.git/info/refs?service=git-upload-pack
... s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext#3a26f314: startup date [Mon Oct 15 22:01:34 EDT 2018]; root of context hierarchy
... o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: file:/your/config/local/fallback/directory/application.properties
Notice the line:
Adding property source:file:/your/config/local/fallback/directory/application.properties
That's where the magic happens.
So if you want the spring.cloud.config.server.git.basedir to be available as a fallback even before the first startup of your config-server (and whether or not your git repo is unreachable during the startup), you can carry out the following steps;
Manually create the spring.cloud.config.server.git.basedir
From your terminal cd /your/config/local/fallback/directory
git clone https://your-git/config-repo.git while the repo is available
Ensure that all your config files/folders/sub-folders including the .git folder are cloned directly to the root of the fallback directory.
For instance, there's a tendency that git clone https://your-git/config-repo.git will clone the repo into the fall back directory as /your/config/local/fallback/directory/config-repo. You will have to copy every darn content of config-repo - including the .git folder too - out and directly into /your/config/local/fallback/directory
Start the config-server (or its container) for the first time or whenever! ......... Voila!!