is it possible in Maven to configure different local repositories for SNAPSHOT and RELEASE artifacts?
The reason I am asking, we are using Jenkins for continuous build for our project. To ensure the consistency (if same artifact is built from different Jenkins jobs because of race condition, we can experience chaotic behavior) before build start, we create a fresh local repository for Jenkins.
Now the problem is, our project is huge, so for every build we have to download lots of dependencies from our Nexus but when you think about it, there is no reason to download every time new the RELEASE artifacts. The RELEASE artifacts don't change from build to build, for ex, Spring 4.5, httpclient 4.0, aspectj 1.8.1 is same for one build to another.
So actually to ensure the consistency, we only should not have the SNAPSHOT dependencies in the repository. If we could have two local repositories one for RELEASE artifacts and the other for SNAPSHOT's, then before every build start, we could delete the SNAPSHOT repository but re-use the local RELEASE repository, which would save me gigabytes of download from Nexus.
I know we can do RELEASE, SNAPSHOT configurations for remote repositories, is it possible to do same sort of configuration for local repositories?
If this is not possible, how would you solve this problem.
There is currently no way to achieve this, and yes, I agree with the sentiment.
A reasonably recent versions of Jenkins' Maven plugin allow you to specify a custom local repository without having to edit a settings.xml file — the option is right there at the job definition screen (in the Advanced section, select Use private Maven repository).
So, what I would do is use this option, and precede the Maven build step with a script that deletes all directories, in the local private repository, which end in -SNAPSHOT.
It's repulsive, but I can't think of any other way.
Related
I have an .m2 repository on my Jenkins slave which is growing every day, currently it's nearly ~40 GB.
Since I have multiple jobs running and picking dependencies from .m2 I cannot remove everything, but I can see in each repo of .m2 there is an older and useless version of the artefact.
Are there any means of way available in maven so that when a job triggers $mvn install maven will keep the latest version only in the .m2 repo (example versioning x.y.z.w which is incremental) for every repo inside .m2?
If you don't care that external dependencies are pulled in every build, you could use a private Maven repository per job (Maven -> Advanced -> Check 'use private Maven repository') and clean the workspace at the start of your build. The private repository creates a .repository in your workspace, so cleaning your workspace will ensure you start with an empty repository.
Should you have many shared external dependencies, then you may be using even more diskspace, since they are present multiple times in the different repositories. In that case you could write a script that periodically (using a task scheduler like cron) removes unused files from the shared repository, see for example this Stack Overflow answer.
However be cautious with a shared Maven repository! Maven by default is not threadsafe, so concurrent jobs downloading the same artifact might use the incomplete downloads. Consider using the Takari extensions to make your Maven repository thread-safe.
Having been through a similar problem, I came up with a solution and made it open source as it might help others. The application is available on Github and it can clean up old dependencies and retain just the latest.
https://github.com/techpavan/mvn-repo-cleaner
Apart from cleaning old dependencies, it has other features like date based cleanup based on download date / last accessed date, removing snapshots, sources, javadocs, ignoring or enforcing deletion of specific groups or artifacts.
Additionally, this is cross platform and can run on both Windows and Unix / Linux environments.
We have a nexus repository environment that serves for multiple projects.
Each project has a repository group which includes
3rdParty
snapshot
test (release typed repo)
Projects prepare builds for testing which are deployed on test repository.
What we would like to do is to re-deploy a specific version, which is already deployed on test repository, to a Releases repository on another nexus instance. By this way only tested builds can be reached by customers.
Is there a way to do that without performing another build? Some builds may take up more than 3 hours and also it is risky to perform another build.
I've searched and come up with wagon plugin but I am not sure about it.
This operation should also perform skip/merge capability (may be fail never?). Because for a bug-fix release, most of the artifacts may not be changed. Thus; skipping those artifacts will save time in delivery.
Below scenario would be perfect:
Build the project and deploy to the test repository
Perform tests using test repository
Tests passed, re-deploy system version 1.0.1 (including child components) to releases repository.
Customer can only see tested versions from releases repository (1.0.1, 1.0.7 etc)
Notes:
using snapshot for test builds is not an option :(
Thanx in adv.
Sincerly;
Metan
I know how to do it for an external repository but not for my local repository, since I don't have a <repository> for my local repository in my settings.xml.
I use snapshot versions for my sub-projects, so when I re-build the parent project I want maven to get all the sub-projects snapshot versions from my local repository not only once a day (which seems to be what happens by default) but always.
If I'm understanding your comment, I think #FrVaBe may have the correct answer. When you change code for a child project on your development machine, it's up to you to rebuild the snapshot and get it into your local artifact repo (via mvn install) so it's available for the parent project to use.
If, however, you want your parent project build to pull in changes made by your teammates and published to the corporate remote repository more often than once per day, read on.
Here is a summary of how Maven central (and kin), remote repositories (e.g a company instance of Nexus or Artifactory) and your local repository work together. If you always want the latest version of snapshots to download on every build, go into your settings.xml file, find <snapshot> repository containing the snapshot you want, and change the <updatePolicy> value to "always". Personally I rarely do this, I simply add the '-U' option to my mvn command line when I want to ensure I have the latest version of a snapshot from my remote repo.
There is no update policy for the local repository!
The local repository is just a bunch of files. When you install to your local repository your local projects already reference the artifacts directly. There is no update that needs to be performed except that maybe your IDE needs to be refreshed to pickup the newer files.
In this manner you can build local snapshots all day long with no versioning headaches, no updates required and no old artifacts left hanging around afterwards. Nice and clean but not so obvious if you're new to Maven and still getting to grips with all these repositories and their fancy update mechanisms.
I think you missunderstood something. Maven will always take the latest/newest SNAPSHOT from your local respository. But in your project setup (Project Inheritance) you need to build the sub projects on their own if you changed something.
An automatical build of the sub project only happens on a Project Aggregation layout.
The difference is explained in the Project Inheritance vs Project Aggregation section of the documentation.
I'm building a project, which is made up from several (sometimes unrelated) modules and some more non standard java modules (built with ANT).
Each maven module is deployed to the releases repository on completion.
If the build fails in the middle, I might have some modules already deployed, so if I try to rebuild, the new attempt to deploy will fail since the artifacts are already deployed.
Is it possible to force a deploy or instead, remove the deployed artifact before I deploy again?
It sounds like the middleware admins have configured your remote repo instance (Nexus or Artifactory or whatever) to not allow artifact redeployment, and as #khmarbaise says there are good reasons for that. Nexus can be configured to allow artifact deletion by users in a particular role or with artifact deletion privileges. If your admins have it set up that way perhaps you can request the delete privilege and remove the offending artifacts. Or, perhaps the Nexus admin will agree to do it for you.
If neither of these is possible, here are some things to try which might keep this from happening in the future:
If you are using the release plugin, do a dry run (-DdryRun=true on the release:prepare command line) first. Maven should report any errors without committing to SCM.
Try running mvn install on your group of projects first. This will install the artifacts to your local repo, not the remote. If there's a problem you can whack the artifacts out of your local repo and start from scratch, repeating until you get a complete build.
If you are running a multi-module build, there are command line options that allow resuming a Maven build from a particular project forward.
Define -Dmaven.deploy.skip=true on the Maven command line. This is similar to suggestion #2, except Maven will actually load & configure the deploy plugin, it just won't do the actual deploy to the remote repo. Once everything works, remove the skip property.
I know it might be late, but in Nexus there's an option where allows the redeployment of artifacts.
Just select the repositories in the left, choose the repository you want to change the policy and then set it to Allow Redeploy.
The possible options have been increased ;)
Use the parameter deployAtEnd (More information: here). With this parameter the artifacts are deployed only, if all artifacts were built successfully.
I was wondering if there is a Hudson/Jenkins plugin that ties repository artifacts to the build that created them? I was looking at the question "remove artifacts from nexus repository" and thinking that deleting a build in Jenkins should also offer the option to remove the artifacts the build created.
We are currently running Jenkins 1.447 and the Nexus Open Source 1.9.2.3. Our Jenkins builds create artifacts in our Nexus repository using the maven deploy goal. I appears that once those artifacts have been deployed, there is no similarly automated mechanism to remove them. We would like to tie the Jenkins build to its Nexus artifacts. I figure if we have decided to remove the build from Jenkins, we have no use for the build and therefore, have no need to store the artifacts for that build either. We would like deleting the build to trigger deleting the Nexus artifacts.
If there's nothing available I figure I could start writing something, but I wanted to check and see how others handle this.
you can use the REST API from Nexus to build your own Jenkins Plugin that does that Job for you. You could store the Jenkins Build Job number using the Nexus custom-metadata plugin. Once the build is deleted you can have your custom Jenkins Plugin delete all artifacts in Nexus that have that build number in their metadate. I had a similar Problem and wrote a custom Jenkins Plugin. Have a look at the tutorial and the source code on github. It should be pretty straight forward.
Tutorial:
http://blog.codecentric.de/en/2012/08/tutorial-create-a-jenkins-plugin-to-integrate-jenkins-and-nexus-repository/
Sourcecode:
https://github.com/marcelbirkner/nexus-metadata-plugin/
You can purge artifacts from a local repository via the maven-dependency-plugin.
If you have a release it does not make sense to number it based on the numbers of the build server. The usual use case is to use SNAPSHOT's for exactly this purpose. Furthermore the usual use case is to delete SNAPSHOT's based on a scheduled task after a time from Nexus but releases will never be deleted from Nexus.
Since you know the name of the release, you could build a custom job or trigger to use a wget command to delete the artifact from the nexus repository.
As the proper user in Nexus you do have the ability to delete release artifacts, not just SNAPSHOTS.