I can't seem to run mvn -o package because it complains with
The repository system is off line but the artifact
com.liferay.portal:util-bridges:jar:6.1.20 is not available in the
local repository.
But I checked my local repository and that artifact does exist there. I also tried the solution of setting updatePolicy to never in the settings.xml file but that failed to work.
Prior to Maven 3.0.x, Maven did not track the origin of files in the local repository.
This could result in build issues, especially if you were building something that listed the (now dead) very borked java.net2 repository... Not only did that repository change released artifacts (extremely bad and evil practice) but it also published artifacts at the same coordinates as artifacts on central but with different content (unbelievably evil)
So you could have the build work (because you had commons-io:commons-io:2.0 from central) wipe your local repo and the build fails (because you now get commons-io:commons-io:2.0 from java.net2 which was a completely different artifact with different dependencies in the pom) or vice versa.
The above situation is one of the drivers for using a maven repository manager, because that allows you to control the subset of a repository that you expose downstream and the order in which artifacts are resolved from multiple repositories (usually referred to as routing rules)
In any case, when maven switched to Aether as the repository access layer, the decision was made to start tracking where artifacts come from.
So with Maven 3.0.x, when an artifact is downloaded from a repository, maven leaves a _maven.repositories file to record where the file was resolved from. If you are building a project and the effective list of repositories does not include the location that the artifact was resolved from, then Maven decides that it is as if the artifact was not in the cache, and will seek to re-resolve the artifact...
There are a number of bugs in 3.0.x though... The most critical being how offline is handled... Namely: when offline, maven 3.0.x thinks there are no repositories, so will always find a mismatch against the _maven.repositories file!!!
The workaround for Maven 3.0.x is to delete these files from your local cache, eg
$ find ~/.m2/repository -name _maven.repositories -exec rm -v {} \;
The side effect is that you loose the protections that Maven 3.0.x is trying to provide.
The good news is that Maven 3.1 will have the required fix (if we can ever get our act together and get a release out the door)
With Maven 3.1 when in offline mode the _maven.repositories file is (semi-)ignored, and there is also an option to ignore that file for online builds (referred to as legacy mode)
At this point in time (June 1st 2013) the 4th attempt to cut a release that meets the legal and testing requirements is in progress... So, assuming that the 4th time is lucky, I would hope to see 3.1.0-alpha-1 released in 3-4 days time... But it could be longer given that we want to give the changes in 3.1 enough time to soak to ensure uses builds don't break (there was a change in an API exposed (by accident-ish - the API is needed by the site and dependency plugin) that plugin authors have depended on (even though they shouldn't have) so there is potential, though we think we have all the bases covered)
Hope that answers your question (and maybe a few more you didn't know you had ;-) )
I also had to remove _remote.repositories in the same way as the _maven.repositories described above. I'm using Maven 3.1.1
find ~/.m2/repository -name _remote.repositories -exec rm -v {} \;
I had this issue when i was using apache-maven-3.0.4, the issue is gone right after i move to apache-maven-3.3.1.
I had this issue in Ubuntu Linux when I've installed local artifacts via a shell script. The solution was to delete the local artifacts and install them again "manually" - calling mvn install:install-file via terminal.
Related
Is it possible to find all dependencies in a maven projects like mvn dependency:tree, save results to file and then use mvn command for download all dependencies from list file in command line?
thanks
In my experience the various plugins do not get a 100% coverage so the builds will fail. What worked for me, was to start with an empty .m2/repositories folder, and then do the maven commands you need to do, and then persist the now populated .m2/repositories folder.
I experimented with converting it to a local file repository in the project itself so we could do with offline builds, but this turned out to be too big for our purposes. It might not be that much of an obstacle today.
Unless you have very strict political requirements, the usual pragmatic solution is to have a local repository on your network like nexus or artifactory mirroring maven central which is then accessed as a mirror.
Scenario:
New enterprise policy forbids from using public repositories. For every dependency used (which are a lot) we have to provide to the devOps team the artifact group, id, version and from which repository it came.
We have used more than one external repository on our code base. Since all the artifacts are now on the local repository, we don't know where they came from.
What I was able to do:
I can use
mvn dependency:list -DoutputAbsoluteArtifactFilename=true -DprependGroupId
on each project to get artifact group, id, version.
And mvn dependency:list-repositories to get the repositories used in that project
What's missing:
Knowing which artifact is from which repository.
Thanks in advance
I am afraid this will not be possible in maven. Prior to maven 3.0.x, maven did not even kept track of remote repositories. After Maven 3.0.x, it moved to eclipse aether. Eclipse aether create _remote.repositories file for every repository which has repository information. It contains repository id which is defined in your settings.xml file. You could,
Delete your local repo contents.
Run following command,
find . -name _remote.repositories -exec grep "^[^#;]" {} \; > repo_info.txt
This will atleast give you the mapping.
Make a list of all external repositories that you use. Furthermore, take the list of all groupId, artifactId, version (Either by dependency:list each project or by scanning the local repository). Now for each dependency, ask the external repositories via REST if they contain this dependency.
If you start with MavenCentral, you have a high chance that you find your dependency in the first try.
In order to prevent that one build influences another it is possible to configure a Jenkins project to use a own private Maven repository. However, because we have actually a huge list of dependencies, this leads to a lot of wasted disk space and to slow builds. We use a Maven repository proxy, but still the time to download artifacts over the local network is significant.
I could set up another repository proxy directly on the Jenkins machine. Is there an easier solution?
I still want that any "maven install" goes to a project-specific repo, while reading of artifacts that have not been deployed to that project-specific repo should come from a central place on the local file-system. Those artifacts should not be copied for performance and disk space reasons.
To explain the background I append the help text of the "Use private Maven repository" option:
"Normally, Jenkins uses the local Maven repository as determined by
Maven — the exact process seems to be undocumented, but it's
~/.m2/repository and can be overridden by in
~/.m2/settings.xml (see the reference for more details.)
This normally means that all the jobs that are executed on the same
node shares a single Maven repository. The upside of this is that you
can save the disk space, but the downside of this is that sometimes
those builds could interfere with each other. For example, you might
end up having builds incorrectly succeed, just because your have all
the dependencies in your local repository, despite that fact that none
of the repositories in POM might have them.
There are also some reported problems regarding having concurrent
Maven processes trying to use the same local repository.
When this option is checked, Jenkins will tell Maven to use
$WORKSPACE/.repository as the local Maven repository. This means each
job will get its own isolated Maven repository just for itself. It
fixes the above problems, at the expense of additional disk space
consumption.
When using this option, consider setting up a Maven artifact manager
so that you don't have to hit remote Maven repositories too often.
If you'd prefer to activate this mode in all the Maven jobs executed
on Jenkins, refer to the technique described here."
You could have:
one settings.xml that points to a common local repository, used for every mvn clean package command
and one settings.xml per projet that uses a specific local repository for every mvn install command
In order you would:
mvn clean package -s settings-common.xml # using common-repo
mvn install -s settings-jobX.xml # using jobX-repo
The only issue is that the artifact installed by your job wouldn't be available to other jobs if they need it. You'd have to either deploy, or copy manually the artifact to the common-repo.
Please note that I do not understand fully what you mean by "one build influences another". You should clarify that in order to have a better answer (because what you want to do might not be what's best to do).
I don't understand why this is so difficult:
In a script, I need to copy an artifact from nexus to a certain directory. Using the Nexus REST API I would have to specify the repository which I don't want to have to know about. So I tried getting the artifact with maven-dependency-plugin's get goal instead, which works well. (In that case I get it from a group on our nexus which includes both, releases and snapshots.)
However, I now have the artifact in my local repo and the same plugin's "copy" goal does not seem to be able to get that artifact out of there. Is it really necessary to descend into the .m2 folder and grab that jar with the unix cp command? Anybody ever copied artifacts from their local repos to other dirs before?
Alternatively, if someone can tell me how to get an artifact via the Nexus API without specifying the repo, that would work, too.
Just tried this, and it worked for me:
mvn dependency:copy-dependencies -DincludeArtifactIds=jcharts -DincludeGroupIds=jcharts -DoutputDirectory=/tmp/
This copied the artifact jcharts:jcharts to /tmp/ It was in my local (and remote) repo when this was executed.
Seems like the problem was the _maven.repositories file in conjunction with the particular maven setup at my company.
We don't put the info about the local repo in our settings.xml. It's all in the parent pom that all our projects use. But if you want to do some pure mvn-CLI magic you don't have the parent-pom so you have to provide the URL to the local repo yourself. This is possible with the dependecy:get goal, which is why I was able to download my artifact from our Nexus into my local repo.
When using copy, however, you can't specify a URL. But why would I want to? I just downloaded that artifact into my local repo, right?
That's where the _maven.repositories file comes into play. Even with the -o switch, maven3 consults that file, which specifies the original repo that the artifact came from. (thanks to the guys in this thread for posting their findings!). If it can't reach the repo, it will claim that your file isn't there. (Btw., this is not helpful imho. It should say something about the original repo not being reachable and that the file therefore won't be copied.)
This was the reason why copy didn't work for me.
Simply renaming that file does the trick.
I will have to investigate a cleaner solution to this, though.
To make things even more complicated, I couldn't use dependency:copy or dependency:copy-dependecies. For some reasons they require a pom, which I don't have in my usecase. What does work is org.apache.maven.plugins:maven-dependency-plugin:2.8:copy which I believe is supposed to be the same thing, but that's another story.
Thanks for your answers!
If you are using Sonatype Nexus, you should consider disabling "Central" as outlined in their book.
Otherwise, per #Keith, dependency:copy-dependency will do what you are trying to do. dependency:get specially downloads from remote repositories, as stated in the documentation
You can also force Maven to not download from non-local repositories l by running it in offline mode: -o
Edit
You can also use Maven Wagon Plugin to copy the file from your local repository to an arbitrary directory. This shouldn't require a pom (but you may need to provide the full URL to the jar).
I needed to be able to specify the group ID, artifact ID and version (and packaging), this worked for me:
mvn dependency:copy \
-Dartifact=org.openmrs.web:openmrs-webapp:2.2.0-SNAPSHOT:war \
-DoutputDirectory=/tmp
In one of my Maven projects, dependency resolution will succeed once, then fail for later build attempts:
[WARNING] The POM for commons-logging:commons-logging:jar:1.1.1 is missing, no dependency information available
[WARNING] The POM for commons-httpclient:commons-httpclient:jar:3.1 is missing, no dependency information available
[WARNING] The POM for javax.mail:mail:jar:1.4.4 is missing, no dependency information available
…and so on, until I delete the maven-metadata-local.xml files corresponding to the failing artifacts (e.g. ~/.m2/repository/commons-logging/commons-logging/maven-metadata-local.xml). After those files are deleted, the next mvn invocation proceeds properly; the metadata files are restored by that invocation (presumably as part of the process of checking my upstream repositories/mirrors for updated artifacts), and I am again presented with the above errors until I again delete the metadata files.
This impacts multiple projects, though it appears to be limited to a particular set of dependencies. I suppose I could go nuclear and blow away my local repo, but I'd like to understand what the problem is.
Thoughts?
Update: It looks like it's the maven-invoker-plugin (which these builds are using for general-purpose integration testing) that is producing these maven-metadata-local.xml files. I'm not using an integration-testing-only local repo as described here, simply because doing so causes the re-downloading of all transitive dependencies (unless you want to maintain an integration-specific settings.xml file!!!). I've used the invoker plugin with a variety of other projects in this way with good results -- certainly never encountering a wedged local repository in the process like this.
Update 2 OK, this is repeatable, even after starting with a completely fresh local repository. This is on OS X, Java 1.6.0_24 with Maven 3.0.3; note that Maven 2.2.1 does NOT exhibit this problem.
Here's one of the projects in question: the 1.3.0-compat branch of rummage. To reproduce:
> mvn clean test
# no error -- can run this and other builds that don't involve maven-invoker-plugin all day w/o problems
> mvn clean integration-test
# FAIL: "Could not resolve dependencies", with warnings as noted above
> mvn clean test
# FAIL: "Could not resolve dependencies", with warnings as noted above
Once the local repository is borked (by the generation of the maven-metadata-local.xml files, AFAICT), no builds will get past the dependency resolution stage.
Running mvn -X reveals lines like this for each artifact that is later apparently not found:
[DEBUG] Verifying availability of /Users/chas/.m2/repository/javax/mail/mail/1.4.4/mail-1.4.4.jar from []
Of course, /Users/chas/.m2/repository/javax/mail/mail/1.4.4/mail-1.4.4.jar et al. does exist, as does /Users/chas/.m2/repository/javax/mail/mail/1.4.4/mail-1.4.4.pom. Totally puzzled. At this point, I'm assuming this is a bug in Maven 3 (or some underlying library), now that I see that 2.2.1 is clean.
Update 3 Bug report filed with Maven project.
This issue is resolved in aether 1.12, one rev above the aether 1.11 library that ships with Maven 3.0.3. Replacing aether 1.11 with 1.12 in one's Maven install results in expected behaviour (as noted in the bug I filed). Here's hoping Maven 3.0.4 is released with aether 1.12 ASAP. :-)
you do not mention what you may have tried, so maybe you didn't try this one: adding the -U option to force update ? (tho maybe this -U option is only relevant for SNAPSHOTs ...)
I've seen similar errors caused by corrupted files in my local repository. For example, if a download failed partway through, or a file in a remote repository changed after I downloaded it. Deleting the affected directories under ~/.m2 fixed it.