We are planning to utilize Spring Cloud Config for our service. Our biggest concern is that when the container starts up, it relies on github to be available all the time so that it can pull the config files. In case github is down, what is the best practice to mitigate the issue?
I was thinking of storing a local folder of the configs as a backup and configuring the application.yml to fallback to it (I do not know how).
I was going to use a Composite Environment Repositories
Please see here: Section 2.1.8
However it states:
Any type of failure when retrieving values from an environment repository results in a failure for the entire composite environment.
This means if the git retrieve fails, it does not fall back to the local component of the composite. I wish it did. Have any of you handled a similar problem? How did you solve it?
Here is a good article about best practices. However, I need a workaround for case 1: Best practices on handling GIT repository inavailability
Spring-Cloud has a configuration property to handle this issue;
spring.cloud.config.server.git.basedir = /your/config/local/fallback/directory
NOTE - If you're using a .yml file, then define the above property as
per yaml conventions.
To have a background knowledge, look at the documentation: http://cloud.spring.io/spring-cloud-static/Finchley.RC1/single/spring-cloud.html#_version_control_backend_filesystem_use
So essentially what happens here is that - as long as your application was initially able to connect to the git repository which you set up in spring.cloud.config.server.git.uri = https://your-git/config-repo.git, then on config-server/container startup, the directory you have defined in your spring.cloud.config.server.git.basedir gets created locally and by default spring-cloud clones your configurations into this directory to be available as fallback.
So whenever your git repository is unreachable, spring-cloud will pick up your configurations from this base directory.
Important things to note:
Unless you really want to re-clone the git configurations only on config-server startup alone, ensure that the property spring.cloud.config.server.git.clone-on-start is NOT set to true or is entirely not set at all - Otherwise, every time you restart your cloud-config service the configurations will be deleted and freshly cloned again and if the repository is not available at the time, application startup will fail - and you perhaps don't want that.
However, if spring.cloud.config.server.git.clone-on-start is set to false or is not even set at all (in which case the default is false), then the git repository will only be cloned on demand - hence if the repository is unreachable, spring-cloud will fallback gracefully to pick up configurations from the spring.cloud.config.server.git.basedir
Even when the application config-server (or its container) is restarted and the git repository is not reachable, you will see something like below;
No custom http config found for URL: https://your-git/config-repo.git/info/refs?service=git-upload-pack
... s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext#3a26f314: startup date [Mon Oct 15 22:01:34 EDT 2018]; root of context hierarchy
... o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: file:/your/config/local/fallback/directory/application.properties
Notice the line:
Adding property source:file:/your/config/local/fallback/directory/application.properties
That's where the magic happens.
So if you want the spring.cloud.config.server.git.basedir to be available as a fallback even before the first startup of your config-server (and whether or not your git repo is unreachable during the startup), you can carry out the following steps;
Manually create the spring.cloud.config.server.git.basedir
From your terminal cd /your/config/local/fallback/directory
git clone https://your-git/config-repo.git while the repo is available
Ensure that all your config files/folders/sub-folders including the .git folder are cloned directly to the root of the fallback directory.
For instance, there's a tendency that git clone https://your-git/config-repo.git will clone the repo into the fall back directory as /your/config/local/fallback/directory/config-repo. You will have to copy every darn content of config-repo - including the .git folder too - out and directly into /your/config/local/fallback/directory
Start the config-server (or its container) for the first time or whenever! ......... Voila!!
Related
Upon running a Spring Cloud Config application with a Git backend a clone is performed into the default location of c:/user/AppData/Temp/local/<config-repo-1>. Cloning can take a long time for larger applications (currently one of my teams takes ~15 min).
When the Config Server is stopped and restarted again instead of using the same local repo another clone is performed, c:/user/AppData/Temp/local/<config-repo-2>.
Is there a way to change this behavior so that restarting the Config Server does not do a new clone but instead just does a fetch for the existing clone.
Thanks in advance for any assistance provided.
spring:
config:
server:
git:
clone-on-start: false
I saw this option somewhere, but couldn't figure out where I got it. Unfortunately, the official spring documentation does not refer to this option. This will not clone the rep when starting.
The other Option from the Spring Documentation is:
spring:
config:
server:
git:
force-pull: false
As documented:
... there is a force-pull property that makes Spring Cloud Config Server force pull from the remote repository if the local copy is dirty
I am using Spring Config to share in a git server the configuration for some Spring Boot microservices.
It works great but when I am traveling I have to work offline sometimes.
I have configured Spring Config microservice local profile to get the config from my local git, (file:) and not to be HTTP git server, so I can change config and test without needing access to main git server.
The problem is that as I am not able to do a "git push" to push the change to main repository, Spring Confgig notes it and shows this message:
The local repository is dirty or ahead of origin. Resetting it to origin/master.
and resets it deleting my last local commit with last config changes.
How can I make Spring Config just to get the last committed configuration in my local git ignoring if it is pushed or not to the main server?
I had a similar problem.
(the difference is that I am not using Spring Config microservice, but runnging spring configuration server as a stand alone application)
But this should work for you as well:
Instead of launching local configuration server with a local git repository (file:) I have launched it with native spring profile.
spring:
profiles:
active: native
cloud:
config:
server:
native:
searchLocations: file:///path/to/local/git/repository
So now content of local repository is served and no reset is done on access
Actually going through spring-cloud-config source code you will see:
if (!isClean(git, label)) {
this.logger.warn(
"The local repository is dirty or ahead of origin. Resetting"
+ " it to origin/" + label + ".");
resetHard(git, label, LOCAL_BRANCH_REF_PREFIX + label);
}
Solution
So it will always try to do this and the only work around will be similar to this answer:
How spring cloud config use local property override remote property
You will need to commit:
spring:
cloud:
config:
allowOverride: true
overrideNone: false
To the configuration properties you are using. e.g.:local/my-app.properties. If you do have a remote repo for storing all the properties, make sure it is merged to master.
Then you could change any application property as you wish in application.yml/application.properties in your Spring Boot app. It won't get overriden by the remote properties file.
Alternative
Alternatively, you could just remove the config file for the environment you are working on as dev environment. In my example above, you could just remove local/my-app.properties from remote repo and commit master. So that it will never bother overriding local application properties from there because no properties files exist for cloud-config.
Please comment if anything is unclear so that I could improve the instruction.
You can use an identical directory as not only a file system backend and also a git repository for properties files. You should make 2 bootstrap.properties files for config server. One is for the file backend and the other is for the git repository. For example, I want to use a file system backend for develop and a git repository for production. I made bootstarp-dev.properties and bootstrap-prd.properties.
## bootstarp-dev.properties
server.port=8887 // If you want to launch both a config server for dev and a config server for prd, you must assign different ports.
spring.cloud.config.server.native.searchLocations=file://PATH_OF_YOUR_DIRECTORY
## bootstarp-prd.properties
server.port=8888
spring.cloud.config.server.git.uri=YOUR_GIT_URL
If you launch your config server with -Dspring.profiles.active=dev,native parameter, the config sever access your properties files as local files.
If you launch your config server with -Dspring.profiles.active=prd parameter, the config sever access your properties files as git files.
Done.
How can I make Spring Config just to get the last committed configuration in my local git ignoring if it is pushed or not to the main server?
This is what --assume-unchaged is for.
--assume-unchaged
Raise the --assume-unchaged flag on this file so it will stop tracking changes on this file
--[no-]assume-unchanged
When this flag is specified, the object names recorded for the paths are not updated.
Instead, this option sets/unsets the assume unchanged bit for the paths.
When the assume unchanged bit is on, the user promises not to change the file and allows Git to assume that the working tree file matches what is recorded in the index. If you want to change the working tree file, you need to unset the bit to tell Git. This is sometimes helpful when working with a big project on a filesystem that has very slow lstat(2) system call (e.g. cifs).
Git will fail (gracefully) in case it needs to modify this file in the index e.g. when merging in a commit; thus, in case the assumed-untracked file is changed upstream, you will need to handle the situation manually.
Until now I have always used FileZilla for transferring my local files to my host for changing a site Im working on. I want to change that and learn how to git gud! So basically the ideal situation would be to be able to work in VSCode on a directory which is mirrored from my web host and from there be able to make changes locally and then commit and change files on my web host when I want to - this has to be possible right?
I have been able to use Git Bash to connect to my host and files using ssh. I have created both an init --bare and init since Im not sure which one to use, but I do have the .git folder created there. I can also using the Bash and the command git status see all the files waiting for to be committed (?).
As I understand I have to initialize the repository, then commit them to "track" these files and have them visible in VSCode to work with, is this right?
But when I try to git commit following error message is displayed.
Waiting for your editor to close the file... code --wait: code: command not found.
This points to that the relative pathing to vscode and/or git is not working, but it is. When I start-up the bash I can use for example code --help and git config --global -e to launch a window of VSCode. So my git config --global core.editor "code --wait" is probably working as it should.
Although AFTER I have connected to my web host using ssh, neither of these command work anymore. Why is this?
And also, am Im on the right way right now in thinking on how to make this "connection" between VSCode, git and my web host (one.com)?
Im thinking that I have to create a local repository in the folder where I today have a duplicate of my web host-files and a remote repository at the actual web host and then make some kind of connection between them two and VSCode. Im not quite sure how yet.
Thanks in advance
I think you might be a little confused with what Git is and how you should be using it for what you want to do. Let's clear some stuff up.
Simply put, Git is a version control system for tracking changes to files over time. You create or edit your files, git add them to a "staging area" and then git commit them with a commit message. If you edit the files after committing them, then git can detect changes to the files and you can add and commit them again, or discard them depending on what you want. However the most important part to understand is that these changes are local at this point. If you want to share them with anyone (or have another system pull them down), then you will need to establish a remote repository.
This is what Github/GitLab are for. Log in/create an account on either site (gitLab provides free private repos) and create a repository named appropriately. Then once you have created a remote repo, follow the instructions to add it as a "remote" to your local repo, then git push your changes up to that remote one. Now, on your server, you will git clone the remote repo and that will pull down your changes. From then on, if you push new changes to the remote repo, you can pull the changes down to the server by doing git pull. This is a very basic and barebones approach to deploying code on the server and there are more sophisticated ways of doing it but I will keep it simple in this answer.
Git is completely separate from VSCode (although VSCode has some git integration and plugins). I would not recommend changing the core.editor to VSCode. What that setting controls is the editor that is used to author commit messages. Loading up VSCode takes too long for that...I recommend that you stick with the default Vim or use nano. Or, in most cases, specify a message when committing: git commit -m "added foobar".
So, the git repo that is on your server (the one that you init'd with --bare) is junk and should be instead created by git cloneing from a remote repo. Hopefully this makes sense!
I have a simple Spring Cloud Config Server which consume configuration from git server.
ConfigServer bootstrap.yml :
spring:
application:
name: config-service
cloud:
config:
server:
git:
uri: ssh://git#mydomain:myport/myrepo.git
searchPaths: "configurations/{application}/{profile}"
server:
port: 8888
When I deploy ConfigServer on local, I can retrieve configuration from http://localhost:8888/myapp/test. But when I deploy ConfigServer on test server, it throws No such label: master when I hit http://testserverip:8888/myapp/test.
Any help would be most appreciated!
I know that I'm posting this answer very very late. But, still posting it so that other folks can find this helpful. Now a days, default branch name of GitHub is "main", but Spring Cloud Config still looks for "master" as a label name. So, if you want to change the default behavior, you can always change the label name look up using below property in application.yml or application.properties:
spring.cloud.config.server.git.default-label=main
This worked very well for me. I'm using Spring Boot 2.3.6.RELEASE
GitHub created default branch "main" , however Spring cloud Config was looking for "master branch". I have created a "master" branch and it worked!
I faced this same annoying scenario.
I'll first state how I solved this scenario and will later emphasize my mistakes which I did in my previous approaches.
In my context, I'm using application.properties
My application overview looks like below:
I have a config-server which is centralized & provides the respective configuration data to the respective micro-services.
For instance, a micro-service 'limits-service', requires some configuration data, it gets it from the central configuration server('spring-cloud-config-server').
Hence to achieve this, 'limits-service' queries the central config server which in-turn fetches the requested data from a remote git branch('spring-cloud-samples').
┌---------------------- <--> [currency-exchange-service]
[git] <--> [spring-cloud-config-server] ------- <--> [limits-service]
└---------------------- <--> [currency-conversion-service]
Solution:
I simply created a new Git Repository (spring-cloud-samples) for all the configuration files which would be consumed by several micro-services through a central configuration server.
In the application.properties file of central configuration server('spring-cloud-config-server'), I have provided a property as:
spring.cloud.config.server.git.uri=https://github.com/{username}/{git-reponame.git}
which might look like
spring.cloud.config.server.git.uri=https://github.com/alice123/spring-cloud-samples.git
That's it!
On starting my central-config-server & observe the log which is as follows:
2020-02-17 05:25:07.867 INFO 15000 --- [nio-8888-exec-9] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: file:/D:/Users/{userName}/AppData/Local/Temp/config-repo-3453413414/limits-service.properties
Moreover, I also consumed the config-data through my 'limits-service' & it worked like a charm!
I tried the following things that resulted in failures and a lot of screaming & yelling. XD
I'm posting it, just so that someone who's trying this same concept might save a night's work or maybe less :p
1) Initially, I was using my Git Repositories SSH URL in my application.properties file of central-config-server like follows:
spring.cloud.config.server.git.uri=git#github.com:alice123/spring-cloud-samples.git
spring.cloud.config.server.git.username=alice123
spring.cloud.config.server.git.password=alice123Pwd
which resulted in the following error:
2020-02-17 05:22:45.091 WARN 15000 --- [nio-8888-exec-1] .c.s.e.MultipleJGitEnvironmentRepository : Error occured cloning to base directory.
org.eclipse.jgit.api.errors.TransportException: git#github.com:aniketrb-github/spring-cloud-samples.git: Auth fail
2) Later, I tried to read the config data from a native/local git directory, which was pointing to my Git remote branch(spring-cloud-samples)
Then I provided the following in my application.properties file:
spring.cloud.config.server.git.uri=file://D:\aniket-workspace\github-ws\microservices\udemy-microservices\git-localconfig-repo
This crashed with:
java.lang.IllegalStateException: Failed to load property source from location 'classpath:/application.properties'
3) Later after enough googling, I changed the above property to below which worked!!:
spring.cloud.config.server.git.uri=D:\\\\alice-workspace\\\\github-ws\\\\microservices\\\\git-localconfig-repo
The 'limits-service' eventually failed while it tried to get the config-data from 'spring.cloud.config.server' with following error:
404: Not Found - "org.springframework.cloud.config.server.environment.NoSuchLabelException: No such label: master"
The solution I've stated worked for me due to all the failures I came across which I mentioned above.
Please correct me if I'm wrong & improvise if necessary. I hope this helps and saves some crucial time.
References: spring-cloud-config, https-or-ssh, git-ssh-windows, git-ssh-windows-troubleshooting
This happens when your service cannot access to Git repository.
Since you can start your application, I assume test server can reach to Git. You should set up SSH authentication to your Git server. You can check this site for SSH configuration.
In my case nothing of given answers worked. only thing that worked is setting default-label to master
spring.cloud.config.server.git.default-label=master
When you create a new repository, nowdays git makes "main" as the default branch instead of master. This issue is all about this.
Create "master" branch manually. It will solve all these issues.
for me this caused due to my default branch 'main'. whenever you create a new branch the default branch is 'main' not 'master' anymore. but spring server config look for master branch.
the quick solution is to checkout a new branch called 'master' from 'main' branch. and try from spring config server. thats it.
to create a new branch from main branch. first point to main branch and from there
git checkout -b master
git push --set-upstream origin master
restart your config server and test
I hope you are doing well.
I am using STS 4 and added source link folder (local git repo) for spring cloud configuration from local git repo.
I had facing same issue,
I did just change back slashes with forward slashes.
when i was using like this
spring.cloud.config.server.git.uri=file://C:\Users\coder\Desktop\spring material\in28microservies\git-localconfig-repo
I got error
2020-04-04 14:57:37.874 ERROR 6952 --- [nio-8888-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.IllegalStateException: Cannot load environment] with root cause
java.lang.IllegalStateException: Cannot convert uri to file: file://C:UserscoderDesktopspring materialin28microserviesgit-localconfig-repo
Then i just replaced all back slashes with forward slashes in config server properties file, and it worked for me
spring.cloud.config.server.git.uri=file://C:/Users/coder/Desktop/spring material/in28microservies/git-localconfig-repo
spring.profiles.active=native
spring.cloud.config.server.native.searchLocations=file://<location>
Try this as mentioned by our friend below:
https://stackoverflow.com/a/33179106/1908827
This will work for sure !!! :)
1 ) You have to commit your changes in following way to show up the master if you initialize the repo for first time in that particular folder
> git add -A .
> git commit -m "added prop..."
2) re-run the project and hit same url.
Njoy :)
This is what I found when I too had a similar issue.
Issue/Mistake#1 >> I created the properties file using notepad(I am on Windows OS).
The file was created with .txt extension. Hence the cloud config was not picking up the changes.
Fix: changed the extension from .txt to .properties
Issue/Mistake#2 >> The .properties file name was different from what is mentioned.
The file should be named:
servicename.properties
matching the servicename defined in the *application.properties* as below:
spring.application.name=<<servicename>>
In my example:
spring.application.name=limits-service
Hence the file should be named **limits-service.properties**,
but I created with the name limits-service-service.properties
enter code here
Fix: renamed the file to: limits-service.properties
Now I am able to see the configurations from the spring-cloud-config and it is picking the values from the configured GIT repository.
To those who may have a 'delayed(>2mins) but correct response' when visiting: localhost:8888/limits-service/default, One simple way that works for me is:
In limits-service.properties, change the configserver port to 8887 instead of sticking with 8888:
spring.config.import=optional:configserver:http://localhost:8887
It appears that there is a timeout when Spring is trying to visit port 8888.
In application.properties of spring-cloud-config-server, change spring.cloud.config.server.git.uri directly to a REAL git that is stored on Github. I am using my own public git so that you guys can copy-paste it:
spring.cloud.config.server.git.uri=https://github.com/SimonSongCA/git-localconfig-repo.git
It significantly reduces the delay from X minutes to seconds, and there are no exceptions at the console anymore.
The below solution worked for me.
Add below property in properties file.
spring.cloud.config.server.git.default-label=master
Also check to see if you are pointing to right git location
spring.cloud.config.server.git.uri=file:///C://Work//my_project//app-props//repo//test
The above error comes when you refer to old tutorial because GitHub switched to the default branch as "main" then your application tries to access the master branch of your config-server repository but it is not available so above error comes and when you access though browser it will display white label error page.
solution already mentioned in above comments
I am using spring cloud config server to host a centralized location for all the property files configurations to be used in the project.
I tried using the config files from a local file system using below and it works fine:
spring.profiles.active=native
spring.cloud.config.server.native.searchLocations=file://${HOME}/Documents/test-config/cloud-config-properties/
I also used the git repo using: spring.cloud.config.server.git.uri=ssh://xxxxxx.com:7999/test/cloud-config-properties.git
I would like to try using a combination of this in my project.
Example - for dev/test profile - i would like to use from local filesystem and for the production - I would like to use Git repository.
I enabled both the git uri and native profiles in my application.properties in config server application. But the properties are always picked up from the local file system. Is this possible?
Not supported out of the box, however there is a workaround for this. You can define the basedir for the configuration server, which is where it saves the files it fetches from the remote server, by setting the property (in the config server):
spring.cloud.config.server.git.basedir=<your_dir>
If you are working with docker, you can map this directory to the host filesystem.
Now whatever file you put in there will be picked up by configuration-server if it matches any of the application/profile in the request. For example you could put a file there called application-dynamic.properties, and have all your clients use dynamic as the last profile, for example
spring.profiles.active=systesting,dynamic
This way everything you will put in application-dynamic.properties will override whatever is defined in your config repo.
One thing to notice though is that you need to add the file only after configuartion server starts, because it deletes this folder during startup.
Needles to say, it's not a good practice doing this in production (for example a restart will cause the file to be deleted), but for test/dev this is the best option.