I am building a Spring Boot Microservices using protobuf and gRPC for communication. However, I realised that I will need to define the entities in all microservices that requires it.
Instead of the straight forward method of copy and paste the raw file (not recommended), I can think of 2 methods:
sharing raw proto file
sharing proto generated files
Which is the proper way of doing? If I am sharing raw proto files, how can I share the proto files properly across microservices?
When sharing across languages, copying raw protos between repositories is typical. Some build systems like Bazel don't need to copy the protos, but most do. When copying protos it is important there there is a single well-known canonical copy of the protos and all other copies are bit-identical to a version of the canonical copy.
But when sharing in a Java-centric collection of projects, creating a canonical Java package for the generated code is superior as it is easier to use and it helps make sure only one copy of the generated code is in the classpath.
A typical Java protobuf Jar will include both the raw protos and the generated code; the raw protos are automatically included by both the Maven and Gradle Protobuf plugins. You then depend on that Jar like normal and it provides the dependency for Java code and Protobuf definitions. The Maven and Gradle Protobuf plugins automatically find .proto files in dependencies and add them to the include path (-I) of protoc when generating code.
Related
Spring cloud-config-server has a built in mechanism to communicate with git repository and read files which are stored there. Then cloud-config-server also has built in mechanism to expose endpoints to clients (normally spring boot apps) which can read those files and use them as configurations.
This is well documented in spring documentation as can be seen from bellow.
According to doc
Spring Cloud Config Server
Spring Cloud Config Server provides an HTTP
resource-based API for external configuration (name-value pairs or
equivalent YAML content).
Also as documented about the serving format
Serving Alternative Formats
The default JSON format from the
environment endpoints is perfect for consumption by Spring
applications, because it maps directly onto the Environment
abstraction. If you prefer, you can consume the same data as YAML or
Java properties by adding a suffix (".yml", ".yaml" or ".properties")
to the resource path. This can be useful for consumption by
applications that do not care about the structure of the JSON
endpoints or the extra metadata they provide (for example, an
application that is not using Spring might benefit from the simplicity
of this approach).
It can also support txt format
Serving Plain Text
Instead of using the Environment abstraction (or
one of the alternative representations of it in YAML or properties
format), your applications might need generic plain-text configuration
files that are tailored to their environment.
But considering that spring cloud config server has the built in mechanism to communicate with a git repository and also exposes endpoints to the clients to consume the delivered files, it would make sense for other type of files to be able to be served from those endpoints as well.
It could be for example .pdf , .xslx , or even .zip
For example let's assume that the configured git repository contains the file myFile.zip in featureA branch. Then the call under the exposed path of type /{application}/{profile}[/{label}] for example as
serverUrl:serverPort/myApp/default/featureA/myFile.zip is able to deliver the file but is always delivered as raw .txt file which then corrupts the content of the original file existing in git.
I have already found the solution, but invested many hours on it and it was strange that it was not documented in spring documentation. So it is probably good to exist here as well to spare some time from others having the same issue.
As discussed under this issue, spring-cloud-config-server runs under the hood with the help of a normal spring-boot app. Considering that spring-boot has built in content negotiation mechanism it is able to consume and produce different content as well.
As for spring-cloud-config-server it is possible to fetch binary files from git as well as other files (ex zip, pdf, word, xlsx ...) if the call is made with the header Accept: application/octet-stream . This way the call to serverUrl:serverPort/myApp/default/featureA/myFile.zip is able to deliver a copy of the original file myFile.zip without any corruption.
I am building a microservice application adopting gRPC as the communication protocol among the microservices. I realised that I have to copy and paste the proto files to all microservices. Futhermore, if there is a change in definition, I will have to C&P again to all relevant microservices.
How should the proto files be shared? Preferably there is a common that we can call to upload and download the updated proto files.
I am creating a multi module maven project and want to keep resources (property files, json) in a separate module. Now to import this resource module in other modules, I have two option - either to use maven-resources-plugin or to use maven-remote-resources-plugin. I already have working code wrt maven-resources-plugin but while searching in internet I found post only on maven-remote-resources-plugin which made me question if I am following maven standards on resources use.
And is it worth to move from maven-resources-plugin to maven-remote-resources-plugin?
Any pointers with respect to this is appreciated.
"...want to keep resources (property files, json) in a separate
module"
You can use Maven Remote Resources Plugin Of course to bundle your resources in a single JAR that can be used across.
This plugin is used to retrieve JARs of resources from remote
repositories, process those resources, and incorporate them into JARs
you build with Maven. A very common use-case is the need to package
certain resources in a consistent way across your organization: at
Apache it is required that every JAR produced contains a copy of the
Apache license and a notice file that references all used software in
a given project.
Maven Resource Plugin has its own usage of copying resources within modules when you don't have them located in the default maven path and want to use them explicitly.
I am working on a java maven project with several modules.
I am facing issues with sharing configuration files from one module to its dependency.
For instance i have a module named utils which holds a log.properties file and i would like to use it in another module named gui. What is the best practice to do this ?
Currently we put the log.properties in a config directory as Maven standard layout suggest it, and it is not included in the jar file. Is it correct ? Should I put it in resources instead ?
I use assembly plugin to copy it to a common config directory, this works well, but when I try to build each module individually the config file cannot be reached. How can i solve this ?
Thanks for your help,
Pierre.
You should put your configuration in src/main/resources/config/. This way it will be included in the jar by default. The maven convention is that only src/main/java and src/main/resources are contained in the final jar by default.
Making property files directly accessible to other modules is not a good practice. You should provide a service in the module owning the configuration that is the only place where those files are accessed. This service will be able to give configurations to other modules. Otherwise you violate the single responsibility principle.
You can configure maven to include the src/main/config directory in the built artifact by specifying it in the resources section of the pom as described here.
I have some questions derived from a problem that I have already solved through this other question. However, I am still wondering about the root cause. My questions are as follows:
What is the purpose of spring.handlers and spring.schemas?
As I understand it's a way of telling the Spring Framework where to locate the xsd so that everything is wired and loaded correctly. But...
Under what circumstances should I have those two files under the META-INF folder?
In my other question linked above, does anybody know why I had to add the maven-shade-plugin to create those two files (based on all my dependencies) under META-INF? In other words, what was the ROOT CAUSE that made me have to use the maven shade plugin?
What is the purpose of spring.handlers and spring.schemas?
well you more or less found it out by yourself, let's add some more details:
some spring libraries contain a spring.schemas and a spring.handlers file inside a META-INF directory
META-INF/spring.schemas
re-maps(*) schemalocation to a xsd inside the library
(abstract) only re-mapped versions are supported by this library
META-INF/spring.handlers
provides namespace handler classes for specific namespaces
the namespace handler class provides the parser logic to parse spring-batch beans, like job,
step, etc.
(*) the actual re-mapping happens during the build of the spring application context
Under what circumstances should I have those two files under the
META-INF folder?
normally the files are inside the spring library jars you use, but you can use the mechanism to implement own namespace bean parsing, then you would have own files
In my other question linked above, does anybody know why I had to add
the maven-shade-plugin to create those two files (based on all my
dependencies) under META-INF? In other words, what was the ROOT CAUSE
that made me have to use the maven shade plugin?
if you use a spring namespace in your spring configuration, you need the appropriate files
the problem arises when you want to run a java application:
with a main class either
the spring libraries need to be on the classpath
or all is merged into one jar, which has to be on the classpath (*)
as war/ear server application, the spring libaries need to be on the classpath, normally inside the war
i guess you did not start the mainclass with the complete classpath and i updated my answer for your first question too
(*) if you merge all into one jar, you have to make sure, that the contents of all spring.schemas/spring.handlers files are merged into one spring.schemas and one spring.handlers file, see this answer for a configuration with maven to create an all-in-one.jar