I have a set of applications, all use Maven and the local repository. The applications form a dependency tree using <dependency> in their pom.xml. All of these projects have -SNAPSHOT in their version.
Is it possible for Maven (or some compatible dependency manager) to build an application together with all of its local dependencies whose source changed?
I do not want to create a multi-module project, because:
the projects are exactly libraries, not modules;
I do not want an additional complexity just to have a form of build which is already precisely defined;
I want the process to be dynamic: if a library is mature enough to be put into a remote repository, it would be no more rebuilt with the main project and that's ok.
For now, there is a lot of refactoring, moving code from one library to another etc. and it happens often that substantial parts of the dependency tree need to be rebuilt. I thus need to manually write mvn install in several projects in order to assure that there is no stale code.
No, it doesn't work. Even with a multi-module project, maven does not detect which modules have changed sources in it and which do not.
There was a (flaky) implementation in Maven 2, but it was not continued in 3.x, see How to get maven 3.0 to only build modules with local scm changes
I hoped they would include it again in maven 4, but I didn't see it yet: https://maarten.mulders.it/2020/11/whats-new-in-maven-4/
I once did a similar setup, but had to use shell scripts with some git magic to get it working.
You can also decide to put your libraries in separate repo's from the start, and use the repo tool that google uses for android development: https://github.com/GerritCodeReview/git-repo/blob/main/README.md
Once you run mvn install on the particular Maven project, it will be accessible for all other Maven projects, which are on the same workstation, during dependency collection (before the compile phase).
Official Maven Build Lifecycle description:
install - install the package into the local repository, for use as a dependency in other projects locally
It's not necessary to keep libraries as part of the same project(or have it as a multi-module project). But once you want to share those libraries with your teammates, you would need either to force them installing libraries locally (as you did), or store those libraries at some external repo, like Artifactory or Nexus
Related
Here I read about how to make a Groovy library .jar ... i.e. pretty much the same as making a Groovy (standalone) project. But I'm not clear what you do then with the resultant .jar...
Say I have two Eclipse "proper"/"standalone" projects (I'm using Groovy for everything) and I want them to share a third Gradle library project of mine as a dependency, which is merely a library of classes... how are my standalone projects expected to find the latest .jar version of the library which they're both using...?
My expectation would be that somehow these versions of the library .jar would have to under GRADLE_USER_HOME (i.e. same location as all other dependency .jars).
Then I would assume that in the build.gradle of both standalone projects you'd have a line like
compile 'mylibrary:mylibrarymodule:3.+'
... of course the first part of these compile directives normally involves a "domain name in reverse" ... and this is normally used by a repository like Maven. How does it work with something which doesn't need to be published?
NB at the time of writing I don't have a Maven account as such and have no idea whether "publication" for re-use of a local common library project like this is essential or not.
Naturally, when I distribute versions of my standalone projects they will need to be packaged up with the library .jar in question.
A link to a how-to for a case like this would be more than welcome: I haven't found it under gradle.org.
If you are developing by yourself, you can use maven-publish plugin to publish your artifacts to local maven repository(you don't have to install maven for this) and on your dependent project you can simply say use mavenLocal repository for dependencies.
If you are on a company, I suggest installing a repository manager and deploy your artifacts to this repository so others can use. You can use their respective plugins to deploy easily. (Gradle Artifactory Plugin, Gradle Nexus Plugin, these are just deployment plugins, you have to setup respository manager to. There are other repository management tools also.) Doing the above process from CI server is the preferred way.
To use latest version of a dependency, you can use Gradle Versions Plugin. If the versioning happen often, using snapshot versions also a possibility.
I've spent years trying to deploy libraries that use native code to Maven Central. I've run into the following problems:
There weren't any good plugins for building native code using Maven. native-maven-plugin was a very rigid build system that, among other things, made it difficult to debug the resulting binaries. You'd have to manually synchronize the native-maven-plugin build system with the native IDE you use for debugging.
Maven did not replace variables in deployed pom.xml files: MNG-2971, MNG-4223. This meant that libraries had to declare platform-specific dependencies once per Maven profile (as opposed to declaring the dependency once and setting a different classifier per profile); otherwise, anyone who depended on your library had to re-define those same properties in their project file in order to resolve transitive dependencies. See Maven: Using inherited property in dependency classifier causes build failure.
Jenkins had abysmal support for running similar logic across different platforms (e.g. "shell" vs "batch" tasks, and coordinating a build across multiple machines)
Running Windows, Linux and Mac in virtual machines was way too slow and fragile. Even if you got it working, attempting to configure the VMs as Jenkins slaves was a lesson in frustration (you'd get frequent intermittent build errors).
Maven Central requires a main jar for artifacts that are platform-specific: OSSRH-975
Sonatype OSS Repository Hosting and maven-release-plugin assumed that it would be possible to release a project in an atomic manner from a single machine but I need to build the OS-specific bits on separate machines.
I'm going to use this Stackoverflow question to document how I've managed to overcome these limitations.
Here is how I overcame the aforementioned problems:
I used CMake for building native code. The beauty of this system is that it generates project files for your favorite (native) IDE. You use the same project files to compile and debug the code. You no longer need to synchronize the two systems manually.
Maven didn't support CMake, so I built my own plugin: https://github.com/cmake-maven-project/cmake-maven-project
I manually hard-coded platform-specific dependencies into each Maven profile, instead of defining the dependency once with a different classifier per profile. This was more work, but it doesn't look like they will be fixing this bug in Maven anytime soon.
I plan to investigate http://www.mojohaus.org/flatten-maven-plugin/ and https://github.com/mjiderhamn/promote-maven-plugin as alternatives in the near future.
Jenkins pipeline does a good job of orchestrating a build across multiple machines.
Running Jenkins slaves on virtual machines is still very error-prone but I've managed to workaround most of the problems. I've uploaded my VMWare configuration steps and Jenkins job configuration to help others get started.
I now create an empty JAR file for platform-specific artifacts in order to suppress the Sonatype error. This was actually recommended by Sonatype's support staff.
It turns out that maven-release-plugin delegates to other plugins under the hood. Instead of invoking it, I do the following:
Use mvn versions:set to change the version number from SNAPSHOT to a release and back.
Tag and commit the release myself.
Use nexus-staging:rc-open, nexus-staging:deploy -DstagingProfileId=${stagingProfileId} -DstagingRepositoryId=${stagingRepositoryId}, and nexus-staging:rc-close to upload artifacts from different platforms into the same repository. This is called a Staging Workflow (referenced below).
Upon review, release the repository to Maven Central.
Important: do not enable <autoReleaseAfterClose> in the nexus-staging plugin because it closes the staging repository after each deploy instead of waiting for all deploys to complete.
Per https://issues.sonatype.org/browse/NEXUS-18753 it isn't possible to release SNAPSHOT artifacts atomically (there is no workaround). When releasing SNAPSHOTs, you need to skip rc-open, rc-close and invoke nexus-staging:deploy without -DstagingProfileId=${stagingProfileId} -DstagingRepositoryId=${stagingRepositoryId}. Each artifact will be uploaded into a separate repository.
See my Requirements API for a real-life example that works.
Other quirks to watch out for:
skipNexusStagingDeployMojo must be false in last reactor module (otherwise no artifacts will be deployed): https://issues.sonatype.org/browse/NEXUS-12365. The best workaround is to use Maven profiles to omit whatever modules you want when deploying (don't use skipNexusStagingDeployMojo at all)
skipLocalStaging prevents deploying multiple artifacts into the same repository: https://issues.sonatype.org/browse/NEXUS-12351
The entire java project has an ant build; however couple of module(s) have maven build too.
My new module (maven built, say A) has dependency over an existing module(or simply a folder?, say B) which is being built using ant which just packages the src into jar and drops it inside the project.
Maven build for module A fails (unable to locate moduleB files); Options -
1. Package module B using maven, push to m2_repo
I do not want to go with this option.
Please let me know what are the other options available for the same.
If you have control over the source of all your modules, and if you decided that Maven is your way to go forward, then I recommend to to go all-in as soon as possible. If you do not then you will have continuous problems for the two build strategies to play along. And not only during build time but also at deploy time when it is time to collect all your runtime dependencies. Unless you build one uber-JAR as your only deployable artifact.
If your modules will (almost) always release in sync, then consider using a multi-module project setup.
When I say "all-in" then I do not mean that you have to give up Ant completely. You can use the maven-antrun-plugin to kick of your existing ant builds.
You should also consider running your own repository server, e.g. Nexus, to take full advantage of your maven builds.
My company has a policy that software deployed into production has be be built on a specific machine that has no access to the internet.
We're currently using Maven. When running build on development machines, maven automatically download the dependencies from central Maven repository without problem. Then before go production, we put all files in local Maven repository (.m2/repository) into source control, and then run offline build with
mvn -o -Dmaven.local.repo=<local repo dir> package
this method works, but managing thousands of files in source control is a real pain, particularly the dependencies for Maven plugins. Thus my question, how can I improve the workflow so as to make it easier to maintain the dependencies in the source control?
I'm considering switching to Gradle, mainly because it's more flexible and doesn't depend on plugin downloaded from repository. but then I found out the Gradle local cache directory is not transportable between computers, which means I cannot check it into source control.
Suggestions and recommendations are all appreciated.
Use internal repository manager like Nexus or Artifactory. Always put released artefact to production.
But building project on production machine is not good idea. Better use complete artefact like EAR or WAR with all dependencies included, or something like jar-with-dependencies or other assembled distros. Build project on your CI server and deploy complete package with one click to production server.
I am using a local maven repository to house some code I am using to develop a project. I have cited this repository in my project.clj file, and am now able to rely on local jars in this way (how to do this in a previous question of mine).
Since I am actively developing these projects, I have my project.clj file looking for the LATEST version. But, in order to update a dependency, I still have to increment the version number of that dependency and then run lein install to build it to the maven repository.
Does leiningen have a way to do this where this is automatically done for me when I build the project that depends on things from the maven repo? Can lein just look for those things and rebuild them as needed?
If you want to develop two projects in parallel, with one depending on the other, you can use symlinks in a checkouts directory to avoid having to install snapshots all the time.
To quote from the Leiningen README:
Q: I want to hack two projects in parallel, but it's annoying to switch between them.
A: Use a feature called checkout dependencies. If you create a directory called checkouts in your project root and symlink some other project roots into it, Leiningen will allow you to hack on them in parallel. That means changes in the dependency will be visible in the main project without having to go through the whole install/switch-projects/deps/restart-repl cycle. Note that this is not a replacement for listing the project in :dependencies; it simply supplements that for tighter change cycles.
Are your dependencies version snapshots? Maven should update all *-SNAPSHOT dependencies on build automatically.