For a project with a large number of Maven artifacts (both internally generated as well as external ones), how does one go about automating the releasing of the internally controlled artifacts as part of an overall product release.
Things to be aware of about this question, we use Jenkins and the Maven release plugin. So the operation of releasing a single artifact is automated (albeit the operation to kick-start the process is manual). However the process of releasing all the changed artefacts over the course of a release is not automated (i.e. one has to manually kick-start the release of each artifact). Part of the problem is that almost nothing is released until the end of the release, prior to that everything remains in SHAPSHOT. We have a huge number of components as well as numerous applications/services (over 30) which rely on the plethora of components. So it is not just the case of picking a component and releasing, there are release dependency hierarchies that must be followed (i.e. start at the bottom releasing components that do not use other components and then work your way up until all the applications/services are released).
It is also worth noting that we use two common parent poms which, for the most part, control the versions of the external artifact dependencies and the internal component dependencies. Some pom files for components and applications may override this, but this is (or should be) an exception and should be for a good, but temporary, reason. So when an internal artifact is released, the version in the corresponding parent dependency pom should also be updated.
The product has a release number (of course), however the various pom files technically do not share this version number. While this is not strictly true, the idea as that when parts of the software are set to end-of-life, they will not be updated in the future, thus while a limited number of artifact versions match the product's version at present, this will eventually not be the case.
Any thoughts on ways to get this process automated would be greatly appreciated. Also if you feel what I have described seems to be a crazy way to manage the software, then please provide a comment. Thank you.
You might be able to make use of the Maven Versions plugin which can help formalise versions for projects.
For example, the use-next-releases goal may allow you to release the lowest level of project and then more rapidly bring those released versions into their dependencies.
There may also be scope to use the use-next-versions goal if you fancy releasing components as necessary and simply bring your projects to the "latest" version thats been formally released.
Related
This is not related to update the pom dependencies using a maven goal. I already have that sorted out.
So I am responsible for creating, packaging and maintaining common utilities. These common utilities are in turn used by all the teams in the org. Problem is that the teams using these utilities do not update the dependencies unless it is the last resort. We would like them to use the latest release version of our common utilities barring very few.
Now I have come across this Maven Versions Plugin by mojohaus which I think serves my need by using two goals - versions:update-properties and versions:use-latest-releases. It serves my purpose except two things:
I do not see a way to exclude certain groupid:artifactid from the update dependency/property
We really want this to be a compulsory thing (maybe part of the Test execution - this is for test automation utilities mainly) rather than a maven goal. Because if it is a maven goal, it needs to be invoked separately and hence becomes optional for teams.
We know that forcefully updating to latest version might cause some issues with defect re-produciblity, but we are willing to take that risk. Our utilities are really test products.
Any direction/help on this is appreciated.
Edit: We run our tests using maven goals clean install already. So they use existing pom. We want dependencies update to happen before the tests run. Also it is desirable to commit the changes to source control (bitbucket) if possible.
We have our tests setup using Jenkins but teams also run multiple test on local machines.
Edit: Found the answer to #1. The plugin provides to exlude regex for group and artifact id. using tags excludes and excludesList
Sorry for insisting. From the Maven point of view I see two main solutions:
Your projects and your utility jar are tightly coupled and every project always needs to use the latest version. Then you can bundle all projects and your utility in one multi-module project. This makes sure that everything is up to date all the time, but it requires that all projects and your utilities are always build together (not separately).
You distribute your utilities to different projects which build and release at different times. Then it is up to the projects to decide when to update. There is unfortunately no standard way to deprecate jars.
If I understand you correctly, you want something "in the middle". This may be hard to achieve.
I recently encountered a difficulty to find back a transitively dependent Maven artifact when building our product, which triggered me to think what is the correct strategy to deal with the situation.
In my product building CI, I need to use a plugin called Maven-Unix-Plugin, which is visible to us. This plugin transitively depends on function-java, which is not visible to us.
We have a Nexus repo, which proxy to the Maven repo. So, in the ideal situation, we don't need to store either maven-unix-plugin or functional-java.
However, I found the expected version of maven-unix-plugin still exists in Maven repository, but its dependent version of functional-java disappeared! More ironically, for all versions of maven-unix-plugin, whose dependent version of functional-java disappeared! In short, the maven-unix-plugin is not usable anymore!
In simple, this sounds an integrity problem of Maven repository. However, my major thinking is how we can avoid this on our side? Now, I managed to find back the old versioned functional-java and put it in our Nexus. However, first of first, it is impossible for us to figure out all of the transitive dependents and put all of them in our Nexus. The ideal picture of Maven is we should NOT be required to do that!
Of course, if we continuously build and maintain our system (and keep updating the dependency version ideally), we should have not faced this problem. However, we are a very small team to maintain a long list of huge legacy systems. The last build for this product is actually 2 years ago!
Any brilliant insights what is good strategies for this kind of headaches to happen in the future?
First of all, MavenCentral very rarely removes artifacts, and only if absolutely necessary. So this problem is likely not to hit you again.
If you want to make sure that this problem will not come again, make sure that you backup the storage of your Nexus repo. All artifacts that are drawn from MavenCentral through your Nexus will be stored in your Nexus storage. So even if MavenCentral is completely wiped out tomorrow, they will still exist in your Nexus.
I have a maven multi-module project with more than 6 children modules in it. If a team of 3 is working in parallel taking 2 sub-modules each as their task, then how to increase the version number.
whereas I am following Major.minor.patch-meta format for version numbering in release cycles.
Project A
-- sub-module-assembler
pom.xml
-- sub-module-1
-sub-sub-module-1 pom.xml
-sub-sub-module-2 pom.xml
pom.xml
-- sub-module-2
-sub-sub-module-1 pom.xml
-sub-sub-module-2 pom.xml
pom.xml
-- sub-module-3
-sub-sub-module-1 pom.xml
-sub-sub-module-2 pom.xml
pom.xml
-- sub-module-4
-sub-sub-module-1 pom.xml
-sub-sub-module-2 pom.xml
pom.xml
-- sub-module-5
-sub-sub-module-1 pom.xml
-sub-sub-module-2 pom.xml
pom.xml
-- sub-module-6
-sub-sub-module-1 pom.xml
-sub-sub-module-2 pom.xml
pom.xml
--pom.xml
Exact usage of incrementing and decrementing version numbers in parallel programming, Whereas the key thing to be notified in this is some sub-modules in that project totally depends upon some other sub-module, if that is the case then how to version number exactly while the development goes parallel as well in certain order ?
I am not clear about the below but is this right way of version numbering
Should i need to do meta release in case of bug fix e.g alpha, beta , gamma etc.,
Should i need to do a patch release in the case of completion of a feature[sub-sub-module-x] in a sub-module whereas i am using second level+ sub-modules e.g 0.1.1-alpha, 0.1.2-alpha etc.,
Should i need to do a minor release in the case of completion of a sub-module e.g 0.2.0-alpha , 0.2.0-RC etc.,.
So after integrating all the RC's 0.2.0-RC + 0.3.0-RC + 0.4.0-RC etc should i need to do a major release as 1.0.0-RTM etc.,
So understanding the above flow is bit cofusing ..
Is there any way to automate the build numbering in a project so as to maintain clear build release numbers. Please provide a solution.
Thanks
I suspect there will be no answer for all versioning strategies. But let me try some hints that may help you choosing a strategy that is appropriate for your situation.
The project looks rather big. The first grouping I would do is by responsibility. Are there modules that are restricted to a certain development team? Or is the whole thing maintained "all-in-one"? Hopefully you are able to split it up a bit.
Once you know about module(s) responsibilities you need to define some life-cycle. How and how often is a release (or bugfix, patch) created and by whom? If everyone follows the same rhythm you can share one version and release the whole tree. But usually this is not the case in larger projects. It also introduces side effects during development if a lot of people share one version.
If you do not plan to release the complete tree all at once you may split it more consequently. You can still use a common parent pom.xml (but one with a released version).
I would define one version for modules in a parent pom.xml and inherit it. So no individual version in submodules. Furthermore every team that depends on other modules should not use SNAPSHOT dependencies to work of other teams. They may only use a released version (be it BETA or RC1). It is important to keep builds reproducible. E.g.: "no snapshot dependencies to artifacts you are not in charge of (you control what changes and when)"
As for the versioning itself: in doubt the simpler option may be better. All the meta information may only confuse about the actual state?
What may also be of some use is to draw a deployment pipeline: what module comes first, which ones depend on it and so forth. The amount of changes in one module define how the version changes (major, minor, patch change). If those changes do not propagate across modules (the API stays stable, which is your goal) the next module may only do patch releases.
If you have not yet released anything, plan ahead. Every iteration is usually an API change (therefore it actually would be a major release). That would lead to version 18.0.0 to be released (after 18 iterations). So usually the minor version is used to indicate iterations and patch versions indicate some fixes to stabilize that release. So major versions are chosen more from a marketing aspect than from a technical one.
It also depends somewhat on the type of software you build (a product, an in house-solution, some additional service for your landscape). Products have a much clearer versioning, they usually use the META part to indicate iterations and the major.minor.patch numbers to indicate what is going on. That strategy ("indicate what changes to expect") may help for what you are doing as well?
So hopefully this did not raise more questions than you had at the beginning :)
We check all of our source code's dependent third-party JARs into source control along with our source code. When needed, we manually download updates to third party JARs and replace those JARs that are in source control with the newer versions. We haven't felt the need to use Maven yet as this process seems simple enough for us. But are we missing something of great value by not using Maven? Or does our scenario not warrant using Maven?
"JARs dont change much", I hear this all the time.....
Storing jars in the SCM is simple in the beginning of the project. Over time the number of jars gets larger and larger.... Wait 2 or 3 years and nobody remembers where the jars came from, what their licensing terms were and most commonly what versions are being used (important to know when analysing security vulnerabilities).....
The best article I've read recently making the case for a repository manager is:
http://www.sonatype.com/people/2012/07/wait-you-dont-have-a-repository-manager/
A little irreverant, but does make a valid point about the kind of technical inertia one encounters all the time.
Switching a project team from ANT to Maven can be scary.... Maven works quite differently, so I find it is best deployed with greenfield or adventurous project teams. For the old-school ANT users, I recommend using the Apache ivy plugin. Ivy allows such teams to outsource the management of their dependencies but keep the build technology they're comfortable with.
Ultimately the biggest benefit of using Maven are not dependency management. It's the standized build process. I've seen several failed attempts to create a "standard" ANT build process. Problem every build engineer has his opinion on what the standard should be.... Maven's approach of forcing users to write build plugins may appear restrictive in the beginning, but just like the iPhone eventually developers discover "there's a Maven plugin for that" :-)
When it comes to dependency management Maven really can be quite valuable. As Mark O'Connor suggests, running a local repository manager would likely be better than checking the artifacts into source control.
There are many tools (like m2e in eclipse) that can help with dependency management and provide valuable feedback on which modules or dependencies require which other dependencies. Maven will also make sure to get the appropriate version of a dependency even if different modules depend on different versions of a given library. That will help prevent duplicate versions of the same jar showing up in your deployed project as long as they have the same group and artifact id.
Even for a very simple project I don't think I would resort to checking dependencies into the source control system.
It's not only about 3rd Party Libraries. Mostly if you have multiple repositories. In our case, we had four repositories with lots of inter- and intra-dependencies.
Actually I started this answer and then I had to go for 15 minutes to talk to some colleague about a problem happened after someone forgot to update the .jar of one project in the other's lib directory.
And it looks more professional :)
What are the best practices for software versioning and multimodules projects with Maven?
I mean, when I create a multimodules project with Maven, what is the best approach for the versioning? To use a single version for all the modules (defined in the top project)? To use a version for each module (defined in the POM of each module)? Is there another approach that I'm missing? What are the pros and cons of each approach?
In general, are the different modules released together (possibly sharing the same version number)?
Thanks
Honestly it depends on what you would like to do. Multimodule projects are created for multiple reasons, one of them being you only need to deploy what has changed instead of all modules.
Think about it this way: if you had a non-multi-module project and you only had to change one line in the services layer, you have to rebuild the entire project and deploy all of the code again...even though only your services layer will change.
With multi-module projects, you can regenerate your project and deploy only what changed...your services. This reduces risk and you're assured that only your services module changed.
You also have a multitude of benefits to using multi-module projects that I'm not listing here but there is certainly a huge benefit to NOT keeping your version numbers of your modules in sync.
When you build your project, consider deploying it to a repository that will hold all compatible jars together for builds (each build creates a new folder with the parent-most pom version number). That way, you don't need to keep documentation about which jars are compatible...they're all just deployed together with a build number.
I was looking for a solution for this exact problem myself and versions-maven-plugin was exactly what I needed. I don't like the release plugin communicating with the SCM system. The versions plugin does just what we need: it sets a new version number in all poms of the project:
mvn versions:set -DnewVersion=2.0.0
Then I can proceed with commits, tags and an official build server build...
EDIT:
The versions plugin depends on how a maven multi-module project has been organised: as a result, it often does not update all POM files in a complex multi-module project.
I've found that sed and find do the job much more reliably:
sed -i 's/1.0.0-SNAPSHOT/1.0.0-RC1/g' `find . -name 'pom.xml'`
Typically you create a multi-module project because you have deemed that the various modules are parts of a single whole. Maybe the client-piece, the controller-piece and the services-piece. Or maybe the UI with services.
In any case, it makes sense to have the version numbers for the various modules to move in lock-step. However Maven does not enforce that as a rule.
As to your question
are the different modules released together (possibly sharing the same
version number)
I would think so. That is one of the reasons for having it a multi-module project. Otherwise you could have the modules as independent projects.
Of course this is the kind of stuff that is rife with edge cases and exceptions ;-)
I had the same problem with a project I`m working on. I also decided to use separate versions and even the dependency to the parent pom only has to be updated if some of the managed dependencies change. (so mostly as #vinnybad describes it)
Two additions
exists-maven-plugin
With the usage of the "org.honton.chas.exists-maven-plugin" only the modules will be deployed to the repository that have actually changed, which is really great, because also the according docker-images will only be published if something has changed on one of the service. This avoids "polluting" the image repository with different but unchanged versions.
versioning
One main downside of the "separated versions" approach are the questions regarding versioning:
What's the current version of my project?
Which module versions work with each other? (even thought they don't directly depend on each other, one does rely on what another does, e.g. they share the database schema)
To solve that I put all module versions into the dependency management part of the parent pom, even if no other module depends on them. A "integration-test" module could solve that by depending on all of the modules - and of course testing them together.
This way I would be "forced" to update the parent pom with every change, since it's referring the released module versions. This way the parent pom would have the "leading" version and at the dependency-management block state the versions of all modules that are compatible with each other (which will be ensured by the integration test).