I have configuration files which are in HOCON (.conf) format. I would like to use Spring Cloud Config to fetch them from a BitBucket repo, and send them to an application (which is NOT a Spring application) when it boots.
I am using the "Serving Plain Text" feature of Spring Cloud Config to fetch and serve a single HOCON file, which defines a particular environment. For example, the client application requests the development.conf environment configuration file, and by specifiying it's exact path in the BitBucket repo, say myApp/development/master/development.conf, the client is served that plaintext file.
However, as is usually desired, I would like base properties from the base.conf to be included in development.conf and have values overridden by the dev enviroment property values if there are common properties in both files.
Previously there was no config server, and so by simply writing the include "base" command at the top of the development.conf file, a HOCON parser (part of the TypeSafe library) in the client app requested the base.conf file (both files being in local directories), and did the overriding when necessary.
When including a Spring Config server, development.conf can be fetched in plain text as mentioned, but it is only (as before) at the client side that the application parses the include "base" command and then refetches the base.conf file. This makes the boot phase of the app very slow (probably) because fetching the additional files requires additional authorization on BitBucket.
Question: how to make it faster?
I have tried spring.cloud.config.server.git.cloneOnStart: true, but this has apparently no improvement (maybe I'm using it incorrectly?)
Should I consider converting the .conf format to .properties format, and using spring.profiles.include: base property at the top of the new developement.properties file?
Perhaps embed the Spring server in another Spring Boot application?
I like the property overriding and inclusion feature of Spring Cloud. I just don't know how to make it work with HOCON formatted files.
Related
Spring cloud-config-server has a built in mechanism to communicate with git repository and read files which are stored there. Then cloud-config-server also has built in mechanism to expose endpoints to clients (normally spring boot apps) which can read those files and use them as configurations.
This is well documented in spring documentation as can be seen from bellow.
According to doc
Spring Cloud Config Server
Spring Cloud Config Server provides an HTTP
resource-based API for external configuration (name-value pairs or
equivalent YAML content).
Also as documented about the serving format
Serving Alternative Formats
The default JSON format from the
environment endpoints is perfect for consumption by Spring
applications, because it maps directly onto the Environment
abstraction. If you prefer, you can consume the same data as YAML or
Java properties by adding a suffix (".yml", ".yaml" or ".properties")
to the resource path. This can be useful for consumption by
applications that do not care about the structure of the JSON
endpoints or the extra metadata they provide (for example, an
application that is not using Spring might benefit from the simplicity
of this approach).
It can also support txt format
Serving Plain Text
Instead of using the Environment abstraction (or
one of the alternative representations of it in YAML or properties
format), your applications might need generic plain-text configuration
files that are tailored to their environment.
But considering that spring cloud config server has the built in mechanism to communicate with a git repository and also exposes endpoints to the clients to consume the delivered files, it would make sense for other type of files to be able to be served from those endpoints as well.
It could be for example .pdf , .xslx , or even .zip
For example let's assume that the configured git repository contains the file myFile.zip in featureA branch. Then the call under the exposed path of type /{application}/{profile}[/{label}] for example as
serverUrl:serverPort/myApp/default/featureA/myFile.zip is able to deliver the file but is always delivered as raw .txt file which then corrupts the content of the original file existing in git.
I have already found the solution, but invested many hours on it and it was strange that it was not documented in spring documentation. So it is probably good to exist here as well to spare some time from others having the same issue.
As discussed under this issue, spring-cloud-config-server runs under the hood with the help of a normal spring-boot app. Considering that spring-boot has built in content negotiation mechanism it is able to consume and produce different content as well.
As for spring-cloud-config-server it is possible to fetch binary files from git as well as other files (ex zip, pdf, word, xlsx ...) if the call is made with the header Accept: application/octet-stream . This way the call to serverUrl:serverPort/myApp/default/featureA/myFile.zip is able to deliver a copy of the original file myFile.zip without any corruption.
I'm trying to serve up config for multiple applications, while providing reasonable defaults - I (apparently incorrectly) assumed that spring cloud config would serve the following property sources if I put in git the following files (using spring.app.name=foo, spring.profiles.active=bar & all on the server/git):
application.properties
application-bar.properties
foo.properties
foo-bar.properties
with each lower file overwriting/appending to the upper ones. The bottom 3 return - but is there no way to make a server-side 'application.properties' for all applications (to enforce consistent logging location standards, for instance)?
I found the cause (which will happen 90% of the time you post a question publicly) the application.properties file was .gitignored!
I am relatively new to spring cloud config and I am able to pull the config file with the application name. There is a need for me to pull all the config files from Git via Spring Cloud Config without having to know or pass the application name in the context.
Currently I have used "spring.cloud.config.server.git.searchPaths= *" in my bootstrap file to search my entire git project and using "/{application}-{profile}.properties" to pull the properties files matching the application file. Would want to pull all properties files without passing this application name, is this possible?
I would require this logic to know how many properties files are checked in and how many are duplicates and this logic will be handled by my custom REST service that I am planning to write.
I am using Springboot and developing web services based on those. In my services, I am keeping the configuration at git instead of with project, so that I will be able to change the properties at run-time without restarting the application.
I am using a ConfigServer to fetch the configuartion and all my services are Config Client.
But the issue is my config files are fetched from git based on
-.properties file, so for ex. if my application name is demo and active profile is testing, then the config file which is getting fetched from git is demo-testing.properties.
But the issue I want to have more than one config file. So how should I mention second config file?
Also the second config file is common to other services also?
So actually I want to use same config file for many services which is located at git?
Does anybody knows if it is possible to expose more than 1 property file per application in Spring Cloud config server?
For example I would like to have defined in my git repo properties for the same app, but in different files:
myapp-customer-services.yml
myapp-products-services.yml
and have all those properties defined inside the files, exposed under "myapp".
No that's not possible currently. I'm not sure it really makes much sense to be honest, since you can easily clearly delineate different sets of properties within a YAML file using separate documents.
Yes it is possible to expose more than 1 property file per application in Spring Cloud config server
You can access it in you client by using following properties
first specify profile which you want
example
myapp-customer.yml
myapp-products.yml
spring.profiles.active=customer,products
spring.cloud.config.name=myapp