I'm currently experimenting with golang's plugin system. A problem which I experienced in my testings is that the file size of the plugins is relativly big.
The application loading the plugin will be referenced as "host".
The host application itself is ~50MiB big since it is a web application and should be extended with plugin functionality.
I've implemented a small plugin loader to start the plugins up.
The plugins may use the already existing APIs in the application for example to access the database.
I've prepared a example plugin for this question. The plugin .so file size is ~39MiB. This gives me the reasonable suspicion that the plugin also contains source code from the host application.
Command used to create main.so:
go build -ldflags="-s -w" --buildmode=plugin main.go
Is it possible to "remove" the duplicated source code from the application to reduce file size since it is already loaded on runtime when the plugin gets loaded?
Plugin loader: https://github.com/jonasfranz/gitea/blob/feature/plugin/modules/plugins/loader.go
Example plugin: https://git.jonasfranz.software/JonasFranzDEV/giteaplugin
Source code is not included in plugins. But what is included in them is their dependencies, recursively. This is so because there is no guarantee that the main app that loads the plugin also contains the dependencies, so to ensure the viability of the plugin, its dependencies must be self-contained.
This does not cause problems if the main app also include the same dependencies (with the same version), they will only be "instantiated" once in the go runtime, for details, see How do Go plugin dependencies work?
What to do in order to reduce plugins' sizes? Besides removing the debug information (what you did), you should minimize the dependencies.
This may require redesign and major changes both in the plugin or in the app you wish to create the plugin for. For example, plugins should not refer to "implementation" packages, plugins should only refer to "interface" packages. If interfaces and implementations are not separated, this may not be possible (hence may it be required to change the main app too).
You may also try utilities that try to compress binaries, for details see: Shrink your Go binaries with this one weird trick
Related
I know you can access different modules (included using include) in a project via org.gradle.api.Project#getSubprojects(), and I know you can get the name and directories of separate builds that have been included (using includeBuild) via org.gradle.api.invocation.Gradle#getIncludedBuilds().
But how can my plugin get information such as the locations of Java source files and class files for projects included using includeBuild?
My goal here is to determine which files have changed in the current git branch (which I can do), and then collect their corresponding class files into a jar file that's used for our patching mechanism that inserts the patch jars at the front of the classpath rather than redeploying the whole application.
I don’t think it is a goal of Gradle to provide including builds with detailed information on included builds. Currently, the Gradle docs basically only state two goals for such composite builds:
combine builds that are usually developed independently, […]
decompose a large multi-project build into smaller, more isolated chunks […]
Actually, isolation between the involved builds seems to be an important theme in general:
Included builds do not share any configuration with the composite build, or the other included builds. Each included build is configured and executed in isolation.
For that reason, it also doesn’t seem to be possible or even desired to let an including build consume any build configurations (like task outputs) of an included build. That would only couple the builds and hence thwart the isolation goal.
Included builds interact with other builds only via dependency substitution:
If any build in the composite has a dependency that can be satisfied by the included build, then that dependency will be replaced by a project dependency on the included build.
So, if you’d like to consume specific parts of an included build from the including build, then you have to do multiple things:
Have a configuration in the included build which produces these “specific parts” as an artifact.
Have a configuration in the including build which consumes the artifact as a dependency.
Make sure that both configurations are compatible wrt. their capabilities so that dependency substitution works.
Let some task in the including build use the dependency artifact in whatever way you need.
Those things happen kind of automatically when you have a simple dependency between two Gradle projects, like a Java application depending on a Java library. But you can define your own kinds of dependencies, too.
The question is: would that really be worth the effort? Can’t you maybe solve your goal more easily or at least without relying on programmatically retrieved information on included builds? For example: if you know that your included build produces class files under build/classes/java/main, then maybe just take the classes of interest from there via org.gradle.api.initialization.IncludedBuild#getProjectDir().
I know, this may not be the answer you had hoped to get. I still hope it’s useful.
We have a directory structure like so
java
build/build.gradle (This does NOT exist yet, but we want this)
servers
server1/build.gradle
server2/build.gradle
libraries
lib1/build.gradle
lib2/build.gradle
We have 11 servers and 14 libraries with varying uses of dependencies. EACH server is a composite build ONLY depending on libraries (we don’t allow servers to depend on each other). In this way, as our mono-repo grows, opening up server1 does NOT get slower and slower as more and more gradle code is added(ie. gradle only loads server1 and all it’s libraries and none of the other libraries OR servers are loaded keeping things FAST).
Ok, so one problem we are running into is duplication now which is why we need build/build.gradle file AND we want EVERY module in our mono repo to include that somehow for a few goals(each goal may need a different solution)
GOAL 1: To have an ext { … } section containing a Map of Strings to gradle dependencies much like so
deps = [
'web-webserver': "org.webpieces:http-webserver:${webpiecesVersion}",
'web-webserver-test': "org.webpieces:http-webserver-test:${webpiecesVersion}",
'web-devrouter': "org.webpieces:http-router-dev:${webpiecesVersion}"
]
In this way, we want ALL our projects to them import dependencies like so
compile deps['web-webserver']
GOAL 2: We want to 'include' a standard list of plugins so we are versioning all gradle plugins the same across the repo. While the above configures all jars to avoid jar hell in a mono-repo, we would like to do the same with just this section
plugins {
id 'com.github.sherter.google-java-format' version '0.9'
}
Of course, it each project may also want to add a few more plugins OR even not depend on this section(in case of an emergency and trying to just get the job done).
GOAL 3: We want checkstyle configuration (or any plugin config) to be defined the SAME for all projects (eventually!!!). We would like the checkstyle gradle to live in a common area but have all libraries somehow pull it in. Again, it would be nice for it to be optional in that, I can pull the gradle section into my build.gradle OR can create a new one in case of emergencies so I don't have to fix all projects in the monorepo right away.
IDEALLY, perhaps I kind of want configuration injection where when I run server1/build.gradle, it actually runs java/build/build.grade as it’s parent somehow but with overrides (IF I declare 'extends xxx.gradle' maybe) then all libraries it uses also use java/build/build.gradle as their parent. I am not sure this is possible or feasible. I am pretty sure 'extends xxx' doesn't exist in gradle.
Are any of these GOALS possible?
thanks,
Dean
I have been working on a monorepo with the exact same requirement as you, using gradle composite builds as well. The way we have solved this problem is by using pre compiled plugins
You need to do a new gradle project with only the code you want to share. This will create a plugin, that you can just add as a composite build and apply to the other projects.
I'm a bit confused by why you don't just use a "standard" gradle top level build file and compose the others as subprojects.
This solves all 3 of your goals
If you are concerned by build speed, you can target each server individually simply by running
./gradlew :server1:build
But if you are not able to do this for some reason you can use the apply from: syntax as described here
I found this note in the Maven's documentation:
You can add elements to this classloader by extensions. These are loaded into the same place as ${maven.home}/lib and hence are available to the Maven core and all plugins for the current project and subsequent projects (in future, we plan to remove it from subsequent projects).
I couldn't understand what they mean by "subsequent projects" here. As far as I understand, extensions are enhancements to lifecycle phases of Maven and are not project specific. So it makes sense to work for all the Maven projects.
Question: Can anyone explain what this statement means "in future, we plan to remove it from subsequent projects"
First an extensions can be extensions of a life cycle but not need to. You can implement an extensions also as an EventSpy for example.
This documentation is related to the Core Classloader which is available within such extensions and makes it also possible to enhance it via an extensions. This classloader contains those files from ${maven.home}/lib which is not a good idea and not necessary. It would be better having only the Maven Plugin API there and it's instances which are currently used and not more...
There existing some extensions like Wagon which are using to make a transport in special cases possible which could be project specific.
Starting with Maven 3.3.1 the core extensions mechanism has been improved to make loading project specific extensions more simpler which means they are located into ${maven.projectBasedir}/.mvn/extensions.xml file and also being loading from an repository. Before 3.3.1 you need to do that manually via mvn -Dmaven.ext.class.path=extension.jar.
I would like to create an assembly descriptor defining how my applications are uploaded on Nexus and in particular I would like those final packages to include a couple of startup scripts handy to run applications on different platforms. Untill now I have always copied the assembly descritor and the startup template scripts from project to project, but now I would like to find a clever solution for the problem.
In the documentation page of the maven-assembly-plugin I found an example showing how to share descriptors across multiple projects; unfortunately it does not cover the case where one wants to include a common resource in the the distribution package but and after a couple of experiments I came to the conclusion it is impossible to do that: to include a resource in the distribution file one has to specify the path of the resource (apparently it is not possible to link a resource contained within a jar).
Additional solutions I have found so far try to use the ant or dependency plugins to unpack the jar containing the the assembly descritor and the script templates inside the build folder before packaging the application.
Even if I can stick with the the solutions of the last paragraph, I am wondering if there is a clearer way to achieve that: do I really need to use additional plugins for such a task?
My need is pretty basic but I could not find any clean answer to it: I simply need to be able to distribute a resource in a multi-module project.
Let us consider for example the LICENSE file, which I hereby assume to be the same for all modules. I prefer not to manually copy it into each and every module because the file could change over time. I also prefer not to statically link to resources (even if using relative paths) outside the project folder, because the modular structure can possibly change too.
Is there any plugin that can be used to robustly guarantee that each module is given the required file? It would be equally acceptable for such copy to be obtained by exploiting the POM of the parent project or directly performed by the super project in the modular hierarchy.
you could use the assembly and the dependency plugins.. did you stumble over that link?
http://www.sonatype.com/people/2008/04/how-to-share-resources-across-projects-in-maven/
it describes that option ..its from 2008, but maven is around for quite some time.. so I guess its more or less up to date
edit regarding comment
Another option is the maven-remote-resources-plugin.
For a more detailed example see:
http://maven.apache.org/plugins/maven-remote-resources-plugin/examples/sharing-resources.html
Since their intro speaks actually for itself, I quote (maven.apache.org)
This plugin is used to retrieve JARs of resources from remote repositories, process those resources, and incorporate them into JARs you build with Maven. A very common use-case is the need to package certain resources in a consistent way across your organization: at Apache it is required that every JAR produced contains a copy of the Apache license and a notice file that references all used software in a given project.