I've spent years trying to deploy libraries that use native code to Maven Central. I've run into the following problems:
There weren't any good plugins for building native code using Maven. native-maven-plugin was a very rigid build system that, among other things, made it difficult to debug the resulting binaries. You'd have to manually synchronize the native-maven-plugin build system with the native IDE you use for debugging.
Maven did not replace variables in deployed pom.xml files: MNG-2971, MNG-4223. This meant that libraries had to declare platform-specific dependencies once per Maven profile (as opposed to declaring the dependency once and setting a different classifier per profile); otherwise, anyone who depended on your library had to re-define those same properties in their project file in order to resolve transitive dependencies. See Maven: Using inherited property in dependency classifier causes build failure.
Jenkins had abysmal support for running similar logic across different platforms (e.g. "shell" vs "batch" tasks, and coordinating a build across multiple machines)
Running Windows, Linux and Mac in virtual machines was way too slow and fragile. Even if you got it working, attempting to configure the VMs as Jenkins slaves was a lesson in frustration (you'd get frequent intermittent build errors).
Maven Central requires a main jar for artifacts that are platform-specific: OSSRH-975
Sonatype OSS Repository Hosting and maven-release-plugin assumed that it would be possible to release a project in an atomic manner from a single machine but I need to build the OS-specific bits on separate machines.
I'm going to use this Stackoverflow question to document how I've managed to overcome these limitations.
Here is how I overcame the aforementioned problems:
I used CMake for building native code. The beauty of this system is that it generates project files for your favorite (native) IDE. You use the same project files to compile and debug the code. You no longer need to synchronize the two systems manually.
Maven didn't support CMake, so I built my own plugin: https://github.com/cmake-maven-project/cmake-maven-project
I manually hard-coded platform-specific dependencies into each Maven profile, instead of defining the dependency once with a different classifier per profile. This was more work, but it doesn't look like they will be fixing this bug in Maven anytime soon.
I plan to investigate http://www.mojohaus.org/flatten-maven-plugin/ and https://github.com/mjiderhamn/promote-maven-plugin as alternatives in the near future.
Jenkins pipeline does a good job of orchestrating a build across multiple machines.
Running Jenkins slaves on virtual machines is still very error-prone but I've managed to workaround most of the problems. I've uploaded my VMWare configuration steps and Jenkins job configuration to help others get started.
I now create an empty JAR file for platform-specific artifacts in order to suppress the Sonatype error. This was actually recommended by Sonatype's support staff.
It turns out that maven-release-plugin delegates to other plugins under the hood. Instead of invoking it, I do the following:
Use mvn versions:set to change the version number from SNAPSHOT to a release and back.
Tag and commit the release myself.
Use nexus-staging:rc-open, nexus-staging:deploy -DstagingProfileId=${stagingProfileId} -DstagingRepositoryId=${stagingRepositoryId}, and nexus-staging:rc-close to upload artifacts from different platforms into the same repository. This is called a Staging Workflow (referenced below).
Upon review, release the repository to Maven Central.
Important: do not enable <autoReleaseAfterClose> in the nexus-staging plugin because it closes the staging repository after each deploy instead of waiting for all deploys to complete.
Per https://issues.sonatype.org/browse/NEXUS-18753 it isn't possible to release SNAPSHOT artifacts atomically (there is no workaround). When releasing SNAPSHOTs, you need to skip rc-open, rc-close and invoke nexus-staging:deploy without -DstagingProfileId=${stagingProfileId} -DstagingRepositoryId=${stagingRepositoryId}. Each artifact will be uploaded into a separate repository.
See my Requirements API for a real-life example that works.
Other quirks to watch out for:
skipNexusStagingDeployMojo must be false in last reactor module (otherwise no artifacts will be deployed): https://issues.sonatype.org/browse/NEXUS-12365. The best workaround is to use Maven profiles to omit whatever modules you want when deploying (don't use skipNexusStagingDeployMojo at all)
skipLocalStaging prevents deploying multiple artifacts into the same repository: https://issues.sonatype.org/browse/NEXUS-12351
Related
I have a set of applications, all use Maven and the local repository. The applications form a dependency tree using <dependency> in their pom.xml. All of these projects have -SNAPSHOT in their version.
Is it possible for Maven (or some compatible dependency manager) to build an application together with all of its local dependencies whose source changed?
I do not want to create a multi-module project, because:
the projects are exactly libraries, not modules;
I do not want an additional complexity just to have a form of build which is already precisely defined;
I want the process to be dynamic: if a library is mature enough to be put into a remote repository, it would be no more rebuilt with the main project and that's ok.
For now, there is a lot of refactoring, moving code from one library to another etc. and it happens often that substantial parts of the dependency tree need to be rebuilt. I thus need to manually write mvn install in several projects in order to assure that there is no stale code.
No, it doesn't work. Even with a multi-module project, maven does not detect which modules have changed sources in it and which do not.
There was a (flaky) implementation in Maven 2, but it was not continued in 3.x, see How to get maven 3.0 to only build modules with local scm changes
I hoped they would include it again in maven 4, but I didn't see it yet: https://maarten.mulders.it/2020/11/whats-new-in-maven-4/
I once did a similar setup, but had to use shell scripts with some git magic to get it working.
You can also decide to put your libraries in separate repo's from the start, and use the repo tool that google uses for android development: https://github.com/GerritCodeReview/git-repo/blob/main/README.md
Once you run mvn install on the particular Maven project, it will be accessible for all other Maven projects, which are on the same workstation, during dependency collection (before the compile phase).
Official Maven Build Lifecycle description:
install - install the package into the local repository, for use as a dependency in other projects locally
It's not necessary to keep libraries as part of the same project(or have it as a multi-module project). But once you want to share those libraries with your teammates, you would need either to force them installing libraries locally (as you did), or store those libraries at some external repo, like Artifactory or Nexus
I have only little experience using maven with eclipse. One of the job descriptions which I received has "Workflow management using Maven" as a required skill. What does this mean ? What do they possibly expect?
I think they want you to correct them? :D
I'm not sure what they refer to. I would guess it relates to the developer workflow of creating and delivering software with eclipse (?) and maven.
So setting up a project from scratch is often done from an maven archetype (a project template if you like). A lot of open source frameworks offer archetypes to start with.
For existing projects you would check out the code from version control and import it into eclipse. the m2eclipse plugin is required to do that (but I think its quite common to have it)
Then there is building the software. Which is done through executing maven phases (which will then execute plugins). See maven-phases for more details. Maven phases have default plugins that execute (for example compile will run the compiler plugin).
So your workflow would look like this: you modify the files. compile them, test them, package them, deploy the artifacts into the maven repository. the maven install phase will store the artifacts in you local repository, the maven deploy phase will upload them into the company's repository.
From there the the files are installed. Yet you can use maven plugins to install the software into a application server. That depends on the traditions of the company.
I would not think of workflow as some strict step by step think like BPMN. Development is usually done with huge amounts of personal practices (are tests written in advance or while implementing, and so on).
Hope that will help :)
So I am new to the continuous integration world and of course, like everyone else, have been thrown into the task of setting it up and integrating it.
My company is .NET based and handles MVC applications, regular websites, form applications, as well as windows services. We have a ton of solutions in our repository and most of these solutions have trouble building in our CI that I am testing out (TeamCity). My local PC being the testing server.
Anyways, I have been reviewing a ton of documentation regarding TeamCity on their confluence pages but a lot of it is very technical and I'm getting tired of reading things more than once. So with that being said, there were some terms and concepts that I was a little confused on that I was hoping someone could explain pretty plainly maybe with an example.
Build Parameters -
I understand what parameters are and know what they are used for. Why would I use one in a build?
2.Snapshot dependencies -
What are these and When should I use them? I am having trouble visualizing this.
Artifact Dependencies -
I understand what artifacts are, I don't know why a project would need them to build however.
Any input would be greatly appreciated. Thanks guys.
I could give you some examples.
Build Parameters
It gives you a way to configure your build. For example, you may want to control your build version in Teamcity. You can use build parameters to specify major, minor version (with default value) for your build.
Personally, I try to avoid to use it as we should control configuration in source code or somewhere.
Snapshot dependencies
It is mostly used to build a chain of builds. For example, you have two builds. One is building source and other one is deploying to development environment. If you want to trigger deployment build after each source code build is successful, you need to add snapshot dependency to deployment build.
Artifact Dependencies
The same example as above, you want to deploy the artifact build in other build. In this situation, you need to use artifact dependency to copy the artifact into deployment build to upload somewhere etc.
Or sometimes you may have different projects, some projects are common libraries. If you don't have a artifact repository, you can use artifact dependency when those common libraries are required by other projects.
I'm in the latter stages of setting up a CI environment for my project. I'm using Maven, Jenkins and Artifactory Pro and can successfully build my project and deploy it's artifacts to Artifactory. I have also written a bash script to retrieve the resulting artifacts of a specific build from Artifactory and copy them somewhere.
The main part I'm missing right now is automated versioning. I've looked at enabling Artifactory release management, which is really cool, but involves the rebuilding of the project. I'm really trying to follow the mantra of 'Build Once, Deploy Anywhere', so any rebuilding is a no-no.
My question boils down to: Is there an automated way (either with one of the aforementioned tools, or a plugin) to handle versioning, without rebuilding an artifact?
Artifactory Pro allows you to easily extend Artifactory's behavior with your own plugins written in groovy. (https://www.jfrog.com/confluence/display/RTF/User+Plugins)
You can find here, an example of Promote extension, that will change your artifacts versions without the needs of new build.
You can find more usefully examples in the GitHub "artifactory-user-plugins" repository.
My company has a policy that software deployed into production has be be built on a specific machine that has no access to the internet.
We're currently using Maven. When running build on development machines, maven automatically download the dependencies from central Maven repository without problem. Then before go production, we put all files in local Maven repository (.m2/repository) into source control, and then run offline build with
mvn -o -Dmaven.local.repo=<local repo dir> package
this method works, but managing thousands of files in source control is a real pain, particularly the dependencies for Maven plugins. Thus my question, how can I improve the workflow so as to make it easier to maintain the dependencies in the source control?
I'm considering switching to Gradle, mainly because it's more flexible and doesn't depend on plugin downloaded from repository. but then I found out the Gradle local cache directory is not transportable between computers, which means I cannot check it into source control.
Suggestions and recommendations are all appreciated.
Use internal repository manager like Nexus or Artifactory. Always put released artefact to production.
But building project on production machine is not good idea. Better use complete artefact like EAR or WAR with all dependencies included, or something like jar-with-dependencies or other assembled distros. Build project on your CI server and deploy complete package with one click to production server.