Transitively download a Maven artifact to the local repository - maven

I am trying to download a specific artifact (and all of its dependencies) to a machine's local repository.
It would seem that using the dependency:get goal would be the best option for this, but despite the documentation it does not seem to actually get the transitive dependencies.
Here is an example where I have tried to use dependency:get to download the spring-core jar and all of its many dependencies. You'll notice that the spring-core jar is the only thing downloaded despite the fact that this was done after cleaning the local repository.
$ mvn org.apache.maven.plugins:maven-dependency-plugin:2.2:get -DrepoUrl=http://repo1.maven.org/maven2/ -Dartifact=org.springframework:spring-core:3.0.5.RELEASE -Dtransitive=true
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:2.2:get (default-cli) # standalone-pom ---
Downloading: http://repo1.maven.org/maven2/org/springframework/spring-core/3.0.5.RELEASE/spring-core-3.0.5.RELEASE.jar
Downloaded: http://repo1.maven.org/maven2/org/springframework/spring-core/3.0.5.RELEASE/spring-core-3.0.5.RELEASE.jar (374 KB at 548.4 KB/sec)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4.401s
[INFO] Finished at: Wed May 25 00:29:47 CDT 2011
[INFO] Final Memory: 7M/107M
[INFO] ------------------------------------------------------------------------
My questions are:
Is this a bug with the dependency:get goal?
If not, what am I doing wrong?
Are there any alternatives methods I could use to accomplish my initially stated goal?

If this is a one time or irregular occurrence for you, The simplest thing to do would be to define the dependency in a POM and run mvn package or similar to retrieve the dependency artifacts. You could also try mvn dependency:sources if you'd like to have the source jars too.
If this is something you want to do more regularly or as part of a process, you could look at using Aether directly to retrieve the dependencies for you.
Another approach if this is something you need to do regularly to manage groups of artifacts into your internal development ecosystem is to use Nexus' procurement suite to retrieve the dependencies and manage them into your repository.

You might can go with this solution
1) Download the artifact as you described (I tested with version 2.5.2)
c:\test>mvn -DrepoUrl=http://repo1.maven.org/maven2/ -Dartifact=org.springframework:spring-core:2.5.2 -Dtransitive=true
2) Download the pom (-Dpackaging=pom) of this artifact
c:\test>mvn -DrepoUrl=http://repo1.maven.org/maven2/ -Dartifact=org.springframework:spring-core:2.5.2 -Dtransitive=true -Dpackaging=pom org.apache.maven.plugins:maven-dependency-plugin:2.2:get
3) Use the downloaded pom to copy all dependencies via the dependency:copy-dependency gaol
c:\test>mvn -DoutputDirectory=C:/test/dependency -f C:/<path-to-repository>/org/springframework/spring-core/2.5.2/spring-core-2.5.2.pom dependency:copy-dependencies
You will find the dependencies (including test and optional scope!) in the created c:\test\dependency folder. To exclude test and optional scope use -DincludeScope=runtime.
You need to dynamically build some path information (e.g. path to the pom in your repository) to set up this solution and also need to bring the artifact itself together with its dependencies but it should work in a script without generating a special pom (which might be easier).

It would appear the answer to question #1 (Is this a bug with the dependency:get goal?) is yes. As of 5/25/2011 issue MDEP-308 is still unresolved.

Related

Jenkins configuration to download jar file from Maven central and put it in Nexus

I have an environment and we skipped to download all the dependencies from Maven central. So we need to create a Jenkins job and it has a parameter with detail for which component jar it should download from central, and if we provide that details it should download and it needs to be put in a particular repository in Nexus.
I'm totally new to this environment, can someone provide inputs of any plugins to use or some scripts to be used ? Thanks in advance !
Unfortunately that is not so trivial as it might seem. At first glance there are two plugins which might help:
maven-dependency-plugin - may download artifact from repository
maven-deploy-plugin - may upload artifact to repository
however that won't work due to a "couple" of reasons:
artifacts have dependencies - that is not enough to transfer single artifact and tell developers something like "well, now you may use it" - you need to traverse all transitive dependencies and transfer them as well
artifacts have classifiers - most useful of them are javadoc and sources, and if artifact has those useful classifiers it is a good idea to transfer them as well
artifact descriptor (pom) may refer to another (parent) pom - there is a dedicated cauldron in the hell for developers who do not flatten poms when publishing artifacts, however we can do nothing with that - we need to transfer parent poms as well
dependencies may have snapshot or range versions - there is another dedicated cauldron in the hell for developers who use such dependencies
and the reasonings mentioned above lead us to the idea that we need a specialised software (read: basic shell scripting won't work) which is capable to transfer consistent copy of artifact from one repository to another. And all the time I did believe the guys, who are responsible for such decisions (create own trusted repository and manage it), do have idea how to do that properly, unfortunately it seems that they don't.
I have tried to implement some PoC using maven API which is capable to transfer consistent copies (transitive dependencies, pom and parent poms) of artifacts across repositories, hope that will help.
% mvn tel.panfilov.maven:reposync-maven-plugin:0.1.0:single \
-Dartifact=org.springframework:spring-tx:5.3.22 \
-DsourceRepositories=central::default::https://repo.maven.apache.org/maven2 \
-DtargetRepository=local::::http://localhost:8081/repository/maven-releases \
-Dtransitive=true \
-DdryRun=true \
-DsyncSources=true \
-DsyncJavadoc=true
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------< org.apache.maven:standalone-pom >-------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] --- reposync-maven-plugin:0.1.0:single (default-cli) # standalone-pom ---
[INFO] Processing Dependency {groupId=org.springframework, artifactId=spring-tx, version=5.3.22, type=jar}
[INFO] Source repositories: [central (https://repo.maven.apache.org/maven2, default, releases+snapshots)]
[INFO] Target repository: local (http://localhost:8081/repository/maven-releases/, default, releases+snapshots)
[INFO] Discovered 16 artifacts
[INFO] org.springframework:spring-beans:jar:5.3.22
...
[INFO] org.springframework:spring-tx:pom:5.3.22
[INFO] Found 16 missing artifacts
[INFO] org.springframework:spring-beans:jar:5.3.22
...
[INFO] org.springframework:spring-tx:pom:5.3.22
[INFO] Dry run, exiting
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

How can I get a list of GAVs a maven package command will produce?

I'm looking for a (supported) mvn based command, which will give me a list of all the GroupID:ArtifactID:Version (GAV) for all artifacts that running a mvn package command would produce.
For a single module Maven project, with no parent pom, this is trivial: you can look inside the pom.
For a single module Maven project, with a parent pom, you could use help:effective-pom and it will present a pom file with the <version> element present.
For a multi module Maven project (reactor), you could actually do the same (didn't think so, learned so just now by trying it out). This will allow parsing the file for (multiple) <project> elements.
Anything else to consider?
The overall goal of this, is to be able to feed a downstream Continuous Delivery (http://go.cd/) stage/step/job with information on what version of it's upstream dependencies should be used.
In general you can't produce a list before the build has run...The problem is that based on the pom model not all artifacts are described, cause some plugins can produce supplemental artifacts (maven-assembly-plugin, maven-shade-plugin, maven-jar-plugin via test-jar etc.)
What you can make is to get a list of produced artifacts after a build has run..(installed). The question of yours inspired me to implement an EventSpy which produces such list at the end of the build...which looks like this:
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4.423 s
[INFO] Finished at: 2016-05-08T13:22:10+02:00
[INFO] Final Memory: 24M/310M
[INFO] ------------------------------------------------------------------------
[INFO] -- Maven Artifact Collector Summary --
[INFO] ------------------------------------------------------------------------
[INFO] test.maven.plugin.profiler:parse-pom:0.1.0-SNAPSHOT:jar
[INFO] test.maven.plugin.profiler:parse-pom:0.1.0-SNAPSHOT:pom
[INFO] test.maven.plugin.profiler:parse-pom:0.1.0-SNAPSHOT:jar:jar-with-dependencies
What i can do is to enhance that and write a file which contains the information (in more or less any format)...At the moment it's just a PoC...May be you can give some more information or create issues or PR's and request what might be needed...may be this is also interesting for others...
Furthermore your downstream part must have those artifacts within a repository cache available (either on a file system or via a repository manager or docker data container)...

maven deploy-file failed with 503: cannot find maven-metadata.xml

I am using Maven to integrate our project with others, maybe in a unpopular way and encountered issue.
We have a project that used to compile with Ant. Ant script is big and awesome, so when we are using Maven for integration, it is decided to keep compiling with Ant.
Now let's say Ant compile output is res-1.0-SNAPSHOT.tar.gz. (any filename can be possible but it IS tar.gz) And I am deploying the file to a nexus-hosted snapshot repository called "snapshots".
I tried to deploy with this command:
mvn deploy:deploy-file \
-DgroupId="com.my-company" \
-DartifactId="res" \
-Dversion="1.0-SNAPSHOT" \
-Dpackaging="tar.gz" \
-Dfile="res-1.0-SNAPSHOT.tar.gz" \
-Durl="http://our-nexus-ip/nexus/content/repositories/snapshots" \
-DrepositoryId="snapshots"
I have a simple settings.xml in ~/.m2 with proxy and server settings. However server settings is not being used in current progress yet, wrong passwords don't get errors.
The output is like this:
[[root#cnbi maven]# ./run.sh
+ mvn deploy:deploy-file -DgroupId=com.my-company -DartifactId=res -Dversion=1.0-SNAPSHOT -Dpackaging=tar.gz -Dfile=res-1.0-SNAPSHOT.tar.gz -Durl=http://135.252.234.142:8081/nexus/content/repositories/snapshots -DrepositoryId=snapshots
Warning: JAVA_HOME environment variable is not set.
[INFO] Scanning for projects...
[INFO] Searching repository for plugin with prefix: 'deploy'.
[INFO] ------------------------------------------------------------------------
[INFO] Building Maven Default Project
[INFO] task-segment: [deploy:deploy-file] (aggregator-style)
[INFO] ------------------------------------------------------------------------
[INFO] [deploy:deploy-file]
[INFO] Retrieving previous build number from snapshots
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Error retrieving previous build number for artifact 'com.my-company:res:tar.gz': repository metadata for: 'snapshot com.my-company:res:1.0-SNAPSHOT' could not be retrieved from repository: snapshots due to an error: Error transferring file
Server returned HTTP response code: 503 for URL: http://135.252.234.142:8081/nexus/content/repositories/snapshots/com/my-company/res/1.0-SNAPSHOT/maven-metadata.xml
[INFO] ------------------------------------------------------------------------
[INFO] For more information, run Maven with the -e switch
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 51 seconds
[INFO] Finished at: Tue Jun 12 08:44:13 CST 2012
[INFO] Final Memory: 7M/209M
[INFO] ------------------------------------------------------------------------
You see, it fails to find maven-metadata.xml. It is for certain, the file and its folder "com/my-company" do not exist at all in the repository.
Besides any misuse of Maven you may find, my questions are:
1) am I using Maven in the right way? (deploying tar.gz, using deploy:deploy-file...)
2) are there incorrect parameters?
3) what is maven-metadata.xml for? It is not there but Maven insists to find it -- I guess it is generated, am I missing some steps?
The solution might be stupid, I am really not familiar with Maven. Unfortunately it has to be done... Please, help me out of this.
Maven version is 2.0.11. Let me know if you want to know more.
I had the same problem and there was a bug in our nexus.
Using maven3 (with same settings.xml, pom.xml and .m2 repo) solved our problem and its easier than upgrading nexus.
You can check this bug also.
If you're still stuck with using Ant, I would recommend you at least consider adding Ivy to the picture, as your dependency manager. If I recall correctly, it was able to update maven-metadata.xml files in the repository.
Have a look at this example.

What is Maven dependency:purge-local-repository supposed to do?

I'm trying to purge the local repository of a project dependency before launching releasing it in order to make sure every dependency required is on the central repository and is downloaded from it.
In the project folder (containing the pom.xml), I launch the following command:
mvn clean dependency:purge-local-repository -DreResolve=false -Dverbose=true
The project's POM is very simple and just have a declared dependency to junit:junit:3.8.1
The command's output give me:
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building corelib-api 0.1.2-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) # corelib-api ---
[INFO] Deleting d:\Users\fpaillard\git-repositories\TEST_CORELIB\corelib-api\target
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building corelib-api 0.1.2-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:2.1:purge-local-repository (default-cli) # corelib-api ---
[WARNING] Missing POM for junit:junit:jar:3.8.1
[INFO] Skipping: corelib-api. It cannot be resolved.
[INFO] Nothing to do for project: test:corelib-api:jar:0.1.2-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.953s
[INFO] Finished at: Mon May 14 11:34:40 CEST 2012
[INFO] Final Memory: 6M/15M
[INFO] ------------------------------------------------------------------------
When I look in the local repository (path checked with mvn help:effetive-settings), junit JARs nor POMs are still in .m2/repository/junit/junit/3.8.1 folder.
Isn't dependency:purge-local-repository supposed to delete it?
I don't understand the WARNING of the output above. Why is junit:junit:jar:3.8.1 POM missing? It is still present at .m2/repository/junit/junit/3.8.1/junit-3.8.1.pom
Is the problem related to the INFO line Skipping: corelib-api. It cannot be resolved.? corelib-api is the artifact name of the project I ran mvn dependency:purge-local-repository against.
I know this is old, but I had the same issue, and adding -DactTransitively=false to the command line fixed this issue. I'm unable to tell why it helped, but it did...
I hope this helps.
Looking at the documentation, disabling the actTransitively option causes the purge goal to only purge the dependencies that are named directly by your pom.xml. When it is time for the build, Maven automatically pulls not only your direct dependencies, but all of the TRANSITIVE dependencies down into your local repo as well.
When the purge goal is looking for what to delete, if it finds other dependencies in the dependencies' poms, it transverses those dependencies to figure out the entire tree in your local repository that can be purged. To do this, it at least needs the transitive project's pom.xml. If it cannot find it in the local repo, or if it thinks there might be a more recent version to analyze, it will go to the external repositories to find it.
I don't think it actually tries to download full project content before it starts purging. But since it at least pulls down the projects' pom.xml files, it will complain if it can't find one just like it would if it were resolving dependencies for an actual build.
Besides just preventing Maven from accessing external repositories while purging, another practical reason would be if you have two projects that have the same transitive dependency, and you don't want the purge from one to affect the performance of the other (since the latter will have to download any missing dependencies again).
On the other hand, something to carefully consider is that if you do NOT allow the purge to consider all of the transitive dependencies possible, you stand to leave a set of downstream dependencies sitting in your local repository that you would otherwise have wanted to remove.
I could make a case for saying that the output you are getting is either unnecessary or preventable with another flag like "reportInaccessibleDependencies=false". But unless it is killing your build, I wouldn't say it is anything to worry about.

Get the dependency tree of a maven project with a missing dependency

First of all, let me start by saying mvn dependency:tree does not work in my scenario.
I'm working on an already existing codebase which is giving a build failure. The issue was that the maven repos (including nexus) does not have a POM for a transitive dependency (org.apache.ws.security:wss4j:pom:1.5.2) in this project.
I want to find where this dependency came from. It's probably a transitive dependency, because it isn't listed in the project's pom nor in parent poms. Invoking mvn dependency:tree does not work because it also fails with the same error I get when I use mvn install (Connection timeout). The error is given below.
So, how can I identify which dependency tries to download this pom? I'd like a general answer to find the dependency tree rather than focusing on wss4j pom stated above.
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Sample/XKMS 4.5.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
Downloading: http://ws.zones.apache.org/repository2/org/apache/ws/security/wss4j/1.5.2/wss4j-1.5.2.pom
[WARNING] The POM for bouncycastle:bcprov-jdk13:jar:132 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3:10.155s
[INFO] Finished at: Wed Mar 14 10:35:20 IST 2012
[INFO] Final Memory: 7M/490M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project wso2appserver-samples-xkms: Could not resolve dependencies
for project org.wso2.appserver:wso2appserver-samples-xkms:jar:4.5.0-SNAPSHOT: Failed to collect dependencies for [org.wso2.xkms:xkms:jar:2.2 (compile)]: Failed to read artifact descriptor for org.apache.ws.security:wss4j:jar:1.5.2:
Could not transfer artifact org.apache.ws.security:wss4j:pom:1.5.2 from/to ws-zones-repository (http://ws.zones.apache.org/repository2): Error transferring file: Connection timed out -> [Help 1]
mvn dependency:tree --debug
outputs the tree before failing in my case.
One way to find this out, is to install the m2eclipse Maven plugin for Eclipse. (If you have and are using Eclipse, that is.) The name is unrelated to the version; it works happily with at least Maven 2.2.1 and 3.
http://www.eclipse.org/m2e/
Then, open your root pom in Eclipse, and click on the Dependency Hierarchy tab. On the right-hand side will be all the dependencies the project will download, and on the left is what dependencies ask for what. If a dependency says (managed from X), it means another dependency wants a different version of this dependency This isn't flawless, however. I still had too look at some of the other poms to find the dependency I was looking for. But it should make life easier, and at least point you in the right direction.

Resources