I am running Gradle builds in a Docker container and I wanted to create a Docker image that would already contain all most common dependencies, so the build itself does not need to download them.
Is there an easy way to tell Gradle to download particular library (or plugin) with all dependencies without a specific build file? I want to use the image to run different builds that do not share any configuration.
I am looking for something similar to Maven's dependency:get.
I did not find any solutions other than creating a small build script in Gradle with some dependencies defined there and running build command.
However, it was easier to just mount Gradle cache as a volume in the container I was using. That has added benefit of being reusable accross many Docker containers. This is the approach I ended up using, if anyone is interested.
Related
We have a directory structure like so
java
build/build.gradle (This does NOT exist yet, but we want this)
servers
server1/build.gradle
server2/build.gradle
libraries
lib1/build.gradle
lib2/build.gradle
We have 11 servers and 14 libraries with varying uses of dependencies. EACH server is a composite build ONLY depending on libraries (we don’t allow servers to depend on each other). In this way, as our mono-repo grows, opening up server1 does NOT get slower and slower as more and more gradle code is added(ie. gradle only loads server1 and all it’s libraries and none of the other libraries OR servers are loaded keeping things FAST).
Ok, so one problem we are running into is duplication now which is why we need build/build.gradle file AND we want EVERY module in our mono repo to include that somehow for a few goals(each goal may need a different solution)
GOAL 1: To have an ext { … } section containing a Map of Strings to gradle dependencies much like so
deps = [
'web-webserver': "org.webpieces:http-webserver:${webpiecesVersion}",
'web-webserver-test': "org.webpieces:http-webserver-test:${webpiecesVersion}",
'web-devrouter': "org.webpieces:http-router-dev:${webpiecesVersion}"
]
In this way, we want ALL our projects to them import dependencies like so
compile deps['web-webserver']
GOAL 2: We want to 'include' a standard list of plugins so we are versioning all gradle plugins the same across the repo. While the above configures all jars to avoid jar hell in a mono-repo, we would like to do the same with just this section
plugins {
id 'com.github.sherter.google-java-format' version '0.9'
}
Of course, it each project may also want to add a few more plugins OR even not depend on this section(in case of an emergency and trying to just get the job done).
GOAL 3: We want checkstyle configuration (or any plugin config) to be defined the SAME for all projects (eventually!!!). We would like the checkstyle gradle to live in a common area but have all libraries somehow pull it in. Again, it would be nice for it to be optional in that, I can pull the gradle section into my build.gradle OR can create a new one in case of emergencies so I don't have to fix all projects in the monorepo right away.
IDEALLY, perhaps I kind of want configuration injection where when I run server1/build.gradle, it actually runs java/build/build.grade as it’s parent somehow but with overrides (IF I declare 'extends xxx.gradle' maybe) then all libraries it uses also use java/build/build.gradle as their parent. I am not sure this is possible or feasible. I am pretty sure 'extends xxx' doesn't exist in gradle.
Are any of these GOALS possible?
thanks,
Dean
I have been working on a monorepo with the exact same requirement as you, using gradle composite builds as well. The way we have solved this problem is by using pre compiled plugins
You need to do a new gradle project with only the code you want to share. This will create a plugin, that you can just add as a composite build and apply to the other projects.
I'm a bit confused by why you don't just use a "standard" gradle top level build file and compose the others as subprojects.
This solves all 3 of your goals
If you are concerned by build speed, you can target each server individually simply by running
./gradlew :server1:build
But if you are not able to do this for some reason you can use the apply from: syntax as described here
After first attempts of setting up automated docker builds for my personal spring-boot github repos (I'm a newbie) they were constantly failing due to the jar file not being found. While my first impression was that I had the pathing wrong, it dawned on me that docker wasn't building the jars before attempting the builds.
FROM openjdk:8-jdk-alpine
VOLUME /tmp
ARG JAR_FILE=target/myjar*.jar
ADD ${JAR_FILE} myjar.jar
EXPOSE 8080
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/myjar.jar"]
While I have found a couple of resouces that describe how to integrate maven into the Dockerfile, it seemed like my image could easily get too big too easily with that approach. Has anybody tackled this issue before and could recommend recommend a way to integrate maven into the Dockerfile build? An alternative I've given thought to is to learn Jenkins and develop a solution that way for a pipeline.
There are many approaches to achieve this.
Approach 1
Rely on the maven with its docker maven plugin
This indeed allows building the image during the maven build.
This approach will work in general and allows a level of customizations sufficient for
many use cases. Usually, spring boot applications come as a single Jar with all the dependencies packed inside it, so there is no need for multiple layers in docker image (at least in my experience).
The point here is to call the plugin when the jar of spring boot is already built.
The jar is prepared with the help of another plugin and it is usually invoked in package phase, so if you go with this approach, make sure that you invoke the image creation plugin after the spring boot maven plugin that builds the single jar artifact.
The artifact must reside in <you_module>/target folder and it will be pretty big in size. The original module will reside next to it but will have the suffix .original.
Approach 2
Let maven build the artifact but not the image. Maven will end after this step.
Then invoke a script in Jenkins that will run the docker build command and it will build the image.
The result will be the same, you'll work here with docker directly possibly utilizing all its perks if you really need it.
Both approaches can work for you, choosing between them depends on the following factors:
You're more a Java / maven guy rather than devops guy - then go with approach 1 otherwise go with approach 2
You would like to actually locally run the image that you've built, for example, if you have a kubernetes cluster installed locally - in this case, go with approach 1
You need to utilize some latest features of docker not available for use from the plugin - then go with approach 2
I've spent years trying to deploy libraries that use native code to Maven Central. I've run into the following problems:
There weren't any good plugins for building native code using Maven. native-maven-plugin was a very rigid build system that, among other things, made it difficult to debug the resulting binaries. You'd have to manually synchronize the native-maven-plugin build system with the native IDE you use for debugging.
Maven did not replace variables in deployed pom.xml files: MNG-2971, MNG-4223. This meant that libraries had to declare platform-specific dependencies once per Maven profile (as opposed to declaring the dependency once and setting a different classifier per profile); otherwise, anyone who depended on your library had to re-define those same properties in their project file in order to resolve transitive dependencies. See Maven: Using inherited property in dependency classifier causes build failure.
Jenkins had abysmal support for running similar logic across different platforms (e.g. "shell" vs "batch" tasks, and coordinating a build across multiple machines)
Running Windows, Linux and Mac in virtual machines was way too slow and fragile. Even if you got it working, attempting to configure the VMs as Jenkins slaves was a lesson in frustration (you'd get frequent intermittent build errors).
Maven Central requires a main jar for artifacts that are platform-specific: OSSRH-975
Sonatype OSS Repository Hosting and maven-release-plugin assumed that it would be possible to release a project in an atomic manner from a single machine but I need to build the OS-specific bits on separate machines.
I'm going to use this Stackoverflow question to document how I've managed to overcome these limitations.
Here is how I overcame the aforementioned problems:
I used CMake for building native code. The beauty of this system is that it generates project files for your favorite (native) IDE. You use the same project files to compile and debug the code. You no longer need to synchronize the two systems manually.
Maven didn't support CMake, so I built my own plugin: https://github.com/cmake-maven-project/cmake-maven-project
I manually hard-coded platform-specific dependencies into each Maven profile, instead of defining the dependency once with a different classifier per profile. This was more work, but it doesn't look like they will be fixing this bug in Maven anytime soon.
I plan to investigate http://www.mojohaus.org/flatten-maven-plugin/ and https://github.com/mjiderhamn/promote-maven-plugin as alternatives in the near future.
Jenkins pipeline does a good job of orchestrating a build across multiple machines.
Running Jenkins slaves on virtual machines is still very error-prone but I've managed to workaround most of the problems. I've uploaded my VMWare configuration steps and Jenkins job configuration to help others get started.
I now create an empty JAR file for platform-specific artifacts in order to suppress the Sonatype error. This was actually recommended by Sonatype's support staff.
It turns out that maven-release-plugin delegates to other plugins under the hood. Instead of invoking it, I do the following:
Use mvn versions:set to change the version number from SNAPSHOT to a release and back.
Tag and commit the release myself.
Use nexus-staging:rc-open, nexus-staging:deploy -DstagingProfileId=${stagingProfileId} -DstagingRepositoryId=${stagingRepositoryId}, and nexus-staging:rc-close to upload artifacts from different platforms into the same repository. This is called a Staging Workflow (referenced below).
Upon review, release the repository to Maven Central.
Important: do not enable <autoReleaseAfterClose> in the nexus-staging plugin because it closes the staging repository after each deploy instead of waiting for all deploys to complete.
Per https://issues.sonatype.org/browse/NEXUS-18753 it isn't possible to release SNAPSHOT artifacts atomically (there is no workaround). When releasing SNAPSHOTs, you need to skip rc-open, rc-close and invoke nexus-staging:deploy without -DstagingProfileId=${stagingProfileId} -DstagingRepositoryId=${stagingRepositoryId}. Each artifact will be uploaded into a separate repository.
See my Requirements API for a real-life example that works.
Other quirks to watch out for:
skipNexusStagingDeployMojo must be false in last reactor module (otherwise no artifacts will be deployed): https://issues.sonatype.org/browse/NEXUS-12365. The best workaround is to use Maven profiles to omit whatever modules you want when deploying (don't use skipNexusStagingDeployMojo at all)
skipLocalStaging prevents deploying multiple artifacts into the same repository: https://issues.sonatype.org/browse/NEXUS-12351
I have a project, which is written using the Play Framework, say myproject-web. It is mostly a thin HTTP layer over another project, which forms the core of the entire business logic, called myproject-engine. In my build setup, myproject-web is a sbt project, whereas myproject-engine is a Gradle one.
What I want to achieve is that Play recognize myproject-engine as a dependency, and invoke gradle to build it whenever I try to build the play application (either on run, or automatically, as it happens in the dev mode) or when I do play dist. Is it possible? What is most important for me is that it automatically loads any dependencies that myproject-engine has.
Eventually, the state I want it to reach is that I host my Maven repo for these projects and then SBT can simply pull this package from over there and will get all its dependencies. Is this rather easy to setup? Even if it is, is it relatively easy to maintain?
As #Peter-Niederwieser pointed out in his comment, I think the only viable solution is to have a maven/ivy/gradle repository where the myproject-engine Gradle project is published to. With the correct resolvers the project becomes yet another project dependency, regardless of the build tool it uses.
See Resolvers in the official documentation of sbt.
I'm currently working on a project that consists of several services written in Java that are accessed by a Ruby/Rails front-end. In an attempt to simplify local development, I've created a separate project that adds all of our service WAR projects as dependencies, and uses the cargo-maven-plugin to deploy each of these as a deployable inside of a single embedded Jetty instance.
The issue I'm having is that I'd like to be able to tell cargo to re-deploy a single WAR out of the several that are being run at a time. Starting the entire set of services from scratch takes a bit, and is really unnecessary when only one deployable has actually changed. As far as I can tell, the cargo:redeploy goal only works for non-standalone containers, and I also haven't been able to find any documentation that its possible to specify what you want to re-deploy on the command line.
Is there a way to tell cargo to re-deploy a single deployable from the command line? I'm thinking of something along the lines of mvn cargo:redeploy -DgroupId=com.foo.bar -DartifactId=baz
Apologies if this isn't clear, or if there is a different approach that I should be taking entirely - I'm relatively new to Java development and Maven.
Thanks for any help.
Download the latest war file to your local machine, then redeploy using the following pattern:
mvn install:install-file -DgroupId=com.foo.bar -DartifactId=baz -Dversion=1.x -Dpackaging=war -Dfile=C:/cargo.jar