Spring cloud-config-server has a built in mechanism to communicate with git repository and read files which are stored there. Then cloud-config-server also has built in mechanism to expose endpoints to clients (normally spring boot apps) which can read those files and use them as configurations.
This is well documented in spring documentation as can be seen from bellow.
According to doc
Spring Cloud Config Server
Spring Cloud Config Server provides an HTTP
resource-based API for external configuration (name-value pairs or
equivalent YAML content).
Also as documented about the serving format
Serving Alternative Formats
The default JSON format from the
environment endpoints is perfect for consumption by Spring
applications, because it maps directly onto the Environment
abstraction. If you prefer, you can consume the same data as YAML or
Java properties by adding a suffix (".yml", ".yaml" or ".properties")
to the resource path. This can be useful for consumption by
applications that do not care about the structure of the JSON
endpoints or the extra metadata they provide (for example, an
application that is not using Spring might benefit from the simplicity
of this approach).
It can also support txt format
Serving Plain Text
Instead of using the Environment abstraction (or
one of the alternative representations of it in YAML or properties
format), your applications might need generic plain-text configuration
files that are tailored to their environment.
But considering that spring cloud config server has the built in mechanism to communicate with a git repository and also exposes endpoints to the clients to consume the delivered files, it would make sense for other type of files to be able to be served from those endpoints as well.
It could be for example .pdf , .xslx , or even .zip
For example let's assume that the configured git repository contains the file myFile.zip in featureA branch. Then the call under the exposed path of type /{application}/{profile}[/{label}] for example as
serverUrl:serverPort/myApp/default/featureA/myFile.zip is able to deliver the file but is always delivered as raw .txt file which then corrupts the content of the original file existing in git.
I have already found the solution, but invested many hours on it and it was strange that it was not documented in spring documentation. So it is probably good to exist here as well to spare some time from others having the same issue.
As discussed under this issue, spring-cloud-config-server runs under the hood with the help of a normal spring-boot app. Considering that spring-boot has built in content negotiation mechanism it is able to consume and produce different content as well.
As for spring-cloud-config-server it is possible to fetch binary files from git as well as other files (ex zip, pdf, word, xlsx ...) if the call is made with the header Accept: application/octet-stream . This way the call to serverUrl:serverPort/myApp/default/featureA/myFile.zip is able to deliver a copy of the original file myFile.zip without any corruption.
Related
I'm trying to serve up config for multiple applications, while providing reasonable defaults - I (apparently incorrectly) assumed that spring cloud config would serve the following property sources if I put in git the following files (using spring.app.name=foo, spring.profiles.active=bar & all on the server/git):
application.properties
application-bar.properties
foo.properties
foo-bar.properties
with each lower file overwriting/appending to the upper ones. The bottom 3 return - but is there no way to make a server-side 'application.properties' for all applications (to enforce consistent logging location standards, for instance)?
I found the cause (which will happen 90% of the time you post a question publicly) the application.properties file was .gitignored!
Let me preface this with the fact that I am still very new to Apache Camel. I'm still trying to understand how it all works, and what needs to be done (and HOW to do it) to achieve a particular effect.
I am trying to develop a Spring Boot application that will use Apache Camel to handle the transmission (and possibly also receipt) of data to/from a number of possible sources and destinations. The purpose of the application is to provide a means to produce/generate network traffic, at the network application level, that will be fed into another Spring Boot application - let's call this the target. We are trying to observe and measure the effects various network loads have on the target.
We would like to be able to transmit data via a number of protocols, including: ftp, http/s, file systems (nfs), various mail protocols (smtp, pop) and data streaming protocols for voice and video. There may be other protocols added at a later time. The data itself is irrelevant, we just need to be able to transmit data via various protocols with various loads.
These applications/services will be running in a containerized environment (Docker) that will be run within our local development and test environment, as well as possibly in a cloud environment, such as AWS. We have used Docker, Ansible, Terraform and are currently working towards using Kubernetes and Istio to manage the configuration, deployment, and operation of these applications.
We need to be able to provide specific configurations of Camel routes for particular deployments.
It would appear that the preferred method to configure Camel routes is via Java DSL, rather than XML DSL. The Camel documentation and nearly every other source of information I've found have a strong bias towards using Java DSL. Examples of XML DSL route configuration are far and few.
My initial impression is that going the Java DSL route (excuse the pun), would not work well with our need to be able to deploy a Camel application with a specific route configuration. It seems like you are required to have Java DSL defined route configurations hardwired into the code.
We think that it will be easier to provide a specific route configuration via an XML file that can be included in a deployment, hence why I've been trying to investigate and experiment with XML DSL. Perhaps we are mistaken in this regard.
My question to the community is: Considering what I've described above, can the Java DSL approach be used to meet the requirements as I've described them? Can we use Java DSL in a way that allows for dynamic route configuration? Keep in mind we would not be attempting to change configuration during operation, just in the course of performing a deployment.
If Java DSL could be used for this purpose, it would be very much appreciated if pointers to documentation, examples, etc. could be provided.
For your use cases you could use XML DSL also. Anyhow below book covers most aspects Camel development with examples. In this book authors describes XML DSL use for most of java DSL examples.
https://www.manning.com/books/camel-in-action-second-edition
In below github repository you can find the source code for all the examples listed in above book.
https://github.com/camelinaction/camelinaction2
Simple tutorial and github repository for Apache Camel using Spring boot.
https://www.baeldung.com/apache-camel-spring-boot
https://github.com/eugenp/tutorials/tree/master/spring-boot-modules/spring-boot-camel
Maven Plugin for build and deployment of spring boot container application into Kubernetes cluster
https://maven.fabric8.io/
In case if your company can afford some funding for your effort look at below link which provides commercial offerings around Camel.
https://camel.apache.org/manual/latest/commercial-camel-offerings.html
Thanks
Madhu Gupta
Our team has a few projects which use the Java DSL for building routes. In order to make them dynamic, there are control structures for iterating and setting endpoints based off configurations. That works for us because the routes are basically all the same, just with different sources and sinks.
If you could dynamically add/change the XML DSL files in a way that doesn't involve redeploying your application, that might be a viable route to follow. One might, for example, change the camel.springboot.xml-routes property to point to a folder which changes as needed.
I have configuration files which are in HOCON (.conf) format. I would like to use Spring Cloud Config to fetch them from a BitBucket repo, and send them to an application (which is NOT a Spring application) when it boots.
I am using the "Serving Plain Text" feature of Spring Cloud Config to fetch and serve a single HOCON file, which defines a particular environment. For example, the client application requests the development.conf environment configuration file, and by specifiying it's exact path in the BitBucket repo, say myApp/development/master/development.conf, the client is served that plaintext file.
However, as is usually desired, I would like base properties from the base.conf to be included in development.conf and have values overridden by the dev enviroment property values if there are common properties in both files.
Previously there was no config server, and so by simply writing the include "base" command at the top of the development.conf file, a HOCON parser (part of the TypeSafe library) in the client app requested the base.conf file (both files being in local directories), and did the overriding when necessary.
When including a Spring Config server, development.conf can be fetched in plain text as mentioned, but it is only (as before) at the client side that the application parses the include "base" command and then refetches the base.conf file. This makes the boot phase of the app very slow (probably) because fetching the additional files requires additional authorization on BitBucket.
Question: how to make it faster?
I have tried spring.cloud.config.server.git.cloneOnStart: true, but this has apparently no improvement (maybe I'm using it incorrectly?)
Should I consider converting the .conf format to .properties format, and using spring.profiles.include: base property at the top of the new developement.properties file?
Perhaps embed the Spring server in another Spring Boot application?
I like the property overriding and inclusion feature of Spring Cloud. I just don't know how to make it work with HOCON formatted files.
I am relatively new to spring cloud config and I am able to pull the config file with the application name. There is a need for me to pull all the config files from Git via Spring Cloud Config without having to know or pass the application name in the context.
Currently I have used "spring.cloud.config.server.git.searchPaths= *" in my bootstrap file to search my entire git project and using "/{application}-{profile}.properties" to pull the properties files matching the application file. Would want to pull all properties files without passing this application name, is this possible?
I would require this logic to know how many properties files are checked in and how many are duplicates and this logic will be handled by my custom REST service that I am planning to write.
I have been playing with Spring Cloud Config and like many of the ideas I see there. I would like to better understand how its creators intended on it being used though.
Lets say that I have several services that support a larger API. Because these services are independent from each other, their source is managed in separate repositories. This allows us to version them and deploy them separately from one another. Today, their properties are managed individually.
I like the idea of having a single config server provide all of the configuration information for the individual applications/services that support this larger API. Looking at the default implementation of EnvironmentRepository (which is GIT based), I would have to have a single repository with all of my application config files in it. Because they all live within the same repository, they would all be managed/versioned together in a single place.
How do I make both models work with each other? Would it be better to have a repository per application instead of one for all applications? What are your thoughts?
-Joshua
It might just be a detail of the implementation of the EnvironmentRepository. See here for some discussion on how and when that might happen.