How can I easily remove all stale snapshots from my local repo?
Many files in my repo results in poor performance and the time to fetch artifacts is long. I'd like to keep the repo trimmed down to latest snapshot.
I see that dependency:purge-local-repository can clear the local repository but I want to keep latest.
I can easily create a script to do this work (and wrap it in a plugin) but don't want to re-invent the wheel if there's already a tool to purge down to latest.
Is there a plugin that can purge my dependencies to latest-snapshot or last X snapshots?
This is similar to how-do-you-deal-with-maven-3-timestamped-snapshots-efficiently
I added the following and each build's package clears out the old snapshots keeping my local repository trim and ensuring faster builds on windows which has issues with a bulky repo
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>1.7</version>
<executions>
<execution>
<id>remove-old-artifacts</id>
<phase>package</phase>
<goals>
<goal>remove-project-artifact</goal>
</goals>
<configuration>
<removeAll>true</removeAll><!-- remove all versions of built artifacts including all versions. Install phase will regenerate -->
</configuration>
</execution>
</executions>
</plugin>
With Peter's solution, all old versions of current artifact (or group of artifacts, if applied in a parent pom) are deleted, including releases and all snapshots.
If any other projects needs a previous release version, will have to be downloaded again from your non-local repository (say Nexus), and once you create a 'package' of this snapshot, will be deleted again.
I've seen another option for this issue: http://maven.apache.org/plugins/maven-dependency-plugin/purge-local-repository-mojo.html
Has the option of defining 'snapshotsOnly', and will remove only snapshots, not previous stable releases.
Related
Question: How do you handle feature branches for maven multi project builds?
Jenkins builds and deploys these branches to keep build overhead on developers to a minimum but develop and feature branches cannot build the same maven version or we risk mismatch between artifacts and source.
We have a script to change parent version in child poms and version in root pom. While this segregates the branches in maven space, it results in extra work when merging.
I think nexus pro staging feature might help us avoid this requirement and make each feature branch use a specific repo which we easily drop after branch deletion/merge.
Again: how to handle the problem of multiple branches and maven?
How about the following approach:
Use the buildnumber-maven-plugin to fetch information from git and populate specific Maven properties (we are interested specifically in the scmBranch property (that is, the current git branch)
Use the build-helper-maven-plugin to check whether we are in a feature branch or not (via a regex, excluding well-known branches like master, develop, etc.) and populate (or not) a new Maven property, say branch.classifier
Use the maven-jar-plugin to set a classifier on the generated artifacts, based on what the previous step set, that is, using the new branch.classifier property: if empty, no classifier will be applied (default behavior, applied to the develop branch, for example); otherwise a classifier named after the current branch will be dynamically applied.
Here is a minimal example:
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>buildnumber-maven-plugin</artifactId>
<version>1.4</version>
<executions>
<execution>
<phase>validate</phase>
<goals>
<goal>create</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>1.10</version>
<executions>
<execution>
<id>regex-property</id>
<goals>
<goal>regex-property</goal>
</goals>
<configuration>
<name>branch.classifier</name>
<value>${scmBranch}</value>
<regex>(^develop)|(^master)|(^release.*)</regex>
<replacement></replacement>
<failIfNoMatch>false</failIfNoMatch>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-jar-plugin</artifactId>
<version>3.0.2</version>
<configuration>
<classifier>${branch.classifier}</classifier>
</configuration>
</plugin>
</plugins>
</build>
The snippet below is basically populating the scmBranch property dynamically, then setting the branch.classifier to its value only if different than develop, master or release*, then setting it as a classifier.
Main advantages of this approach:
No pom changes will be applied, hence no merging issues at all
No clashes on Nexus working on the same version of the project but on different branches: the classified artifact will have different Maven coordinates, that is, the GAV (groupId, artifactId, version) becomes unique GAVC (+classifier)
That's actually a meaningful usage of the classifier attribute of an artifact:
The classifier allows to distinguish artifacts that were built from the same POM but differ in their content.
The generated artifact will be dynamically different in Nexus, according to its source branch, hence having implicit traceability: no intervention from developers (no error prone, implicit convention), no intervention from CI job (easier maintenance), completely transparent
Using classifiers, will be easier to use the artifacts generated by a branch as a maven dependency (e.g. in case of library project): I want to use the dependency currently under development on branch xxx
Examples
Hence, you would have the following artifacts generated:
When working on develop: e.g. project-1.0.0-SNAPSHOT.jar (empty classifier, hence not applied, as handled by the regex)
When working on featureA: e.g. project-1.0.0-SNAPSHOT-featureA.jar
When working on hotfix-JIRA123: e.g. project-1.0.0-hotfix-JIRA123.jar
When working on release-sprint42: that's up to you, I added this case to not apply the branch name, simply because in these cases I prefer to esplicitely set a special classifier, RC<number>, for release candidates, but that's a matter of conventions/taste/habits, you can apply the same approach on this branch as well, as long as no clashes will be created on Nexus. Also note: when using JIRA/Stash/Git integration, the release branch name is normally something like release/v0.1.0, where the / character may cause issues in some OS (still something fixeable via further regex replacing though, if really required).
When working on master: hey, no one should work on master :) the case is there just as a double check, but that's actually not required
Warnings on this approach:
As explained in the discussion below via comments, if the concerned Maven project is already using classifiers and even more via inter-modules dependencies (e.g. dependendies on test scope classes from another module), then this approach should be carefully tested, since it might have some drawbacks
The publication of the <artifactId>.pom files containing branch classifier can get into conflicts with the mainline build (i.e. overriding it)
This does not work Results in warnings throughout the build and GC error when run from top parent.
Ideally, we want to use version as to differentiate feature branch from mainline because it is the normal maven way and classifier manipulation can result in all kinds of issues.
Since maven can use environment variables for properties and we already initialize build environment with a script (we also have git hook scripts that can set environment variables from branch names) we can use env to control the version.
<groupID>my.project</groupId>
<artifactID>database</artifactId>
<version>1.2.0${env.BRANCHMODIFIER}-SNAPSHOT</version>
If on develop our scripts set BRANCHMODIFIER to ""
If on feature/JIRA-30495 our scripts set BRANCHMODIFIER to ".30495"
How does this work in eclipse or Intellij? No clue as of yet.
We use a similar technique as Peter Kahn, modifying the version of the branch before building. We have three steps in our "Pre Steps":
Execute shell: echo VERSION=$(echo ${GIT_BRANCH} | sed 's_^.*\/__') > env.properties
Inject environment variables: env.properties
Invoke top level Maven targets: versions:set -DgenerateBackupPoms=false -DnewVersion=${VERSION}-SNAPSHOT
I am quite sure that this can be done with two or even one and only step as well, bit the principe behind it will be the same.
The reason why we don't change the version in the pom.xml files in the branch directly is indeed merging. With SVN this was possible (merging with --accept-mine-conflict. With GIT this does not exist anymore, so we stopped changing versions and created this pre-build steps.
For Maven ge 3.5.0 try this https://maven.apache.org/maven-ci-friendly.html
This is the recommended Maven solution.
The only problem (but unusual) can be the numeric versions resolution of the maven dependencies. But this only apears, if you use different SNAPSHOT dependencies of the the module, which is a bad idea anyway.
I want to rebuild my project structure from scratch from time to time and want to purge the built repository in order to do that. However, I don't want to remove downloaded files from maven central and other repositories. Is there a simple way to tell maven to install my built artifacts into a separate repository, ie. other then the one used to store downloaded, external files?
I am NOT talking about deploy, just mvn install.
UPDATE
I found an alternate solution using only one local repository for both downloaded and self-built artifacts: the self-built ones are accompanied by files called "maven-metadata-local.xml", so I select the repository directories to purge based on the existence of that file now...
You cannot do that with the install goal. maven-install-plugin will install the artifact to the same local repository that is used to fetch downloaded artifacts from. By default, this is ${user.home}/.m2/repository. You change that by setting the system variable maven.repo.local to another location (or by telling Maven to use a specific settins.xml). However, at the moment, Maven can't be configured to install specific artifacts to a different local repository than where it is fetching downloaded artifacts.
A possible work-around would be to declare an execution of the install-file goal, bound to the install phase and declare it to install all of the artifacts you want to to the specified local repository.
<plugin>
<artifactId>maven-install-plugin</artifactId>
<version>2.5.2</version>
<executions>
<execution>
<phase>install</phase>
<goals>
<goal>install-file</goal>
</goals>
<configuration>
<file><!-- path to artifact to install --></file>
<pomFile><!-- path to POM of artifact --></pomFile>
<localRepositoryPath><!-- path to repository you want to install to --></localRepositoryPath>
</configuration>
</execution>
</executions>
</plugin>
MINSTALL-126 enhancement about if this could be added to maven-install-plugin. In the mean time, see the following workaround, slightly extending what's proposed above, from a blog post I wrote on http://blog2.vorburger.ch/2016/06/maven-install-into-additional.html with some background about why this would be useful:
<profiles>
<profile>
<activation>
<property>
<name>addInstallRepositoryPath</name>
</property>
</activation>
<build>
<plugins>
<plugin>
<artifactId>maven-install-plugin</artifactId>
<executions>
<execution>
<id>additional-install</id>
<phase>install</phase>
<goals>
<goal>install-file</goal>
</goals>
<configuration>
<file>${project.build.directory}/${project.build.finalName}.jar</file>
<localRepositoryPath>${addInstallRepositoryPath}</localRepositoryPath>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
To formalize and expand on the “update” in the question (by the way you should not hesitate to answer your own question):
I came to a similar conclusion independently and include
find -L ~/.m2/repository \( -type d -name '*-SNAPSHOT' -prune -o -type f -name maven-metadata-local.xml \) -exec rm -rfv {} \;
in a general “cleanup” script that I run from time to time. Note that this differs from the ideal of install:install always going to a separate location in (at least) three ways:
You have to remember to run this script, so in the meantime you could have a local repository “polluted” with things you built. On occasion this will mean that a build will work locally for you but will not work for others. (Or even fail only or you, or succeed for everyone but behave subtly differently.) This defeats the goal of reproducible builds, unless you have plenty of Internet bandwidth and are willing to run docker run --rm -v "$PWD":/usr/src/mymaven -w /usr/src/mymaven maven mvn clean install!
If someone has intentionally deployed SNAPSHOTs to a shared repository, this script will delete them, so your next build will have to repeat the download.
Local installs of release versions are not deleted. Now if these came from release:perform because you were the one cutting the release, that is not so bad—presumably the remote artifact is identical to your local copy anyway. Where this gets really evil is if, in the course of trying to debug some problem in someone else’s released artifact by rebuilding from sources with some diagnostic patches (say), you forget to edit pom.xml to use a SNAPSHOT or other distinguishing version, and install the result. Maven will never notice that your local copy differs from the official version, and you can get into weird situations months later. Of course this has never happened to me.
The latter two problems could perhaps be addressed with a more complicated script that parsed maven-metadata-*.xml files rather than assuming that all, and only, SNAPSHOTs were local builds. Or as the submitter hints at, just delete the whole version directory if maven-metadata-local.xml is present (distinguishing this somehow from the parent artifact directory, which will also have such a file, and resolver-status.properties too).
While it is nice that Maven 3 records some information about where artifacts in the local repository came from, it is not good enough. What I for one would really appreciate is if install:install always saved to a distinct location, so that the main local repository could be trusted to be purely a cache of downloads. Local artifact resolution would then prefer one or the other repository in case of conflict based on a command-line switch (after issuing a warning).
I'm using Tycho extras to mirror repositories using a target file. The thing is in my .target file I have some installable units with version 0.0.0 meaning I want to get the latest version, but what if I want to run a script to freeze the version, I mean getting the latest one available now and explicitly put the number in the .target file so in the future I'll still mirror the same version I have now.
<plugin>
<groupId>org.jboss.tools.tycho-plugins</groupId>
<artifactId>target-platform-utils</artifactId>
<version>0.21.0</version>
<executions>
<execution>
<phase>compile</phase>
<goals>
<goal>mirror-target-to-repo</goal>
</goals>
</execution>
</executions>
<configuration>
<sourceTargetFile>${target-platform-file}</sourceTargetFile>
<includeSources>false</includeSources>
<outputRepository>${basedir}/target/repository</outputRepository>
</configuration>
</plugin>
There is a dedicated plug-in for this task, which you can call from the command line:
mvn org.eclipse.tycho.extras:tycho-version-bump-plugin:0.22.0:update-target -Dtarget=path/to/targetfile.target
This command needs a Maven/Tycho project, so you need to start it in the directory of your parent pom.xml (or specify the path to that file with -f). Also, the version number in the command needs to match the Tycho version configured in your project.
Also note that the command updates all artifact versions to the latest available version, not only the ones which had a 0.0.0 version literal. If you just want to update some versions, keep a backup of your target file and then pick the changes you want with a diff tool of your choice.
My SCM connection information:
<scm>
<connection>scm:svn:https://repo/project/trunk</connection>
<developerConnection>scm:svn:https://repo/project/trunk</developerConnection>
</scm>
My release plugin configuration:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-release-plugin</artifactId>
<version>2.3.2</version>
<configuration>
<tag>RC</tag>
<autoVersionSubmodules>true</autoVersionSubmodules>
</configuration>
</plugin>
Now when I run mvn release:prepare, instead of committing my tagged release under tags/RC, it does tags/RC/trunk. How do I stop it from adding trunk under RC?
Brian, you may be the victim of Maven's incomplete release:rollback feature. See my question on StackOverflow and the answer to it. If your tag already exists, Subversion (not Maven) will think you want to copy trunk inside the existing tag. Delete the tag and it will work - once. Try again and you'll get RC/trunk. Try yet again and you'll get an error from Subversion.
The solution is to svn delete the tag before you try to copy to it - we do this successfully from Maven during release:perform, by binding a couple of plugins to the deploy phase.
Basically:
Let release:prepare do its thing, create a tag with a unique name.
Bind org.codehaus.mojo:exec-maven-plugin:exec to the deploy phase to make it run during release:perform. Configure it to call svn delete <path to RC tag>.
Bind maven-scm-plugin:branch to deploy in order to create the tag fresh using Maven's SCM plugin.
This works for us and it has the added benefit that it gives us the unique tags too for reference. Worst case, you can ignore these tags.
I have a question that's probably pretty similar to this. I need to solve what I have to imagine to be a pretty common problem -- how to configure Maven to produce multiple variations on the same artifact -- but I have yet to find a good solution.
I have a multi-module project, that eventually results in the assembly plugin generating an artifact. However, part of the assembly includes libraries that have changed substantially in the recent past, with the result that some consumers of the project need library version N, while others need version N+1. Ideally, we'd just automatically generate multiple artifacts, e.g. theproject-1.2.3.thelib-1.0.tar.gz, theproject-1.2.3.thelib-1.1.tar.gz, etc. (where that's release 1.2.3 of our project, running against either library version 1.0 or 1.1).
Right now, I have a bunch of default properties, which build against the latest version of the library in question, plus a profile to build against the older version. I can deploy one or the other this way, but cannot deploy both in one build. Here's the key wrinkle that differs from the above question: I can't automate build-one-clean-build-the-other inside of the release plugin.
Normally, we'd mvn release:prepare release:perform from the root of the multi-module project to take care of deploying everything to our internal Nexus. However, in that case, we have to pick one -- either run the old-library profile, or run without and get the new one. I need the release plugin to deploy both. Is this just impossible? I have to imagine we're not the first people who want to have our automated builds generate support for different platforms....
You may install additional artifacts with differrent types/classifiers. Use attach-artifact goal of the build-helper-maven-plugin to achieve this. Here is a small example - we are deploying a Windows and a Unix installers of the product as windows/exe and unix/sh files. These files will be installed to the local repo and deploy to the distribution management.
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<executions>
<execution>
<id>install-installation</id>
<phase>install</phase>
<goals>
<goal>attach-artifact</goal>
</goals>
<configuration>
<artifacts>
<artifact>
<file>${basedir}/target/${project.artifactId}-${project.version}-windows.exe</file>
<classifier>windows</classifier>
<type>exe</type>
</artifact>
<artifact>
<file>${basedir}/target/${project.artifactId}-${project.version}-unix.sh</file>
<classifier>unix</classifier>
<type>sh</type>
</artifact>
</artifacts>
</configuration>
</execution>
</executions>
</plugin>
Hope this helps.