We use the maven dependency plugin (with maven3) to unzip dependencies at runtime for running a set of tests. However, because the number of dependencies and their size is large it can take upto 30 minutes for the download to finish (not all teams are on LAN). I checked with the maven docs and maven only downloads dependencies that belong to seperate groups in parallel. Is there any way that I can force maven or maybe the maven dependency plugin to download these dependencies in parallel?
Have you already tried to use
mvn -T3.0C phase
Related
Trying to download all maven dependencies for a complex project to speed up subsequent builds. There are some dependencies that are not outlined on pom.xml, such as com.apollographql.apollo:apollo-compiler which are triggered as part of the generate-sources phase (from a dependency I do include apollo-client-maven-plugin)
my original plan of using maven dependency:go-offline did not work because these jars are not included.
i'm currently using the work around maven dependency:go-offline generate-sources which ensure that those extra jars are downloaded, but now i have to wait for the actual generation of the sources...
Is there a way to just force maven to detect dependencies fully?
Alternatively, any way to have incremental mvn --offline, i.e. uses the jar locally without updating repository info, but do download missing ones?
I cloned the git repository of Apache ActiveMQ Artemis project (https://github.com/apache/activemq-artemis) and then typed
mvn -Ptests test -pl :integration-tests
I was surprised to see log messages like the following
...
Downloading: http://repository.apache.org/snapshots/org/apache/activemq/artemis-selector/1.4.0-SNAPSHOT/artemis-selector-1.4.0-20160625.030221-11.jar
Downloading: http://repository.apache.org/snapshots/org/apache/activemq/artemis-core-client/1.4.0-SNAPSHOT/artemis-core-client-1.4.0-20160625.030211-11.jar
...
Since e.g. artemis-core-client is contained in the git repository I cloned in the beginning, I'd have expected maven just builds it from there.
That way, when I make changes in the core client source, they get picked up by the integration tests.
Instead, maven is downloading the jar from the repository.
Question: How do I configure maven to always build all modules that are in the git repository and download only "true" dependencies, which I mean things not in the git repository?
You are not executing the Maven build on the main project, on the main pom.xml which indeed defines the artemis-selector and artemis-core-client modules, among others.
You are executing the Maven build on the tests and its pom.xml, where only tests modules are defined. This is a side/test project, which has as parent the previous pom file, but it doesn't play any role in its parent modules definition. Hence, dependencies are not resolved as modules but as Maven dependencies.
You should firstly install (via mvn clean install) the former project, so that libraries will be available in your local Maven cache (hence no downloading would be triggered), then execute the tests project.
Check the Maven docs for a inheritance vs aggregation difference to further clarify it.
From the Stack Overflow, the follow threads could also be interesting:
What is the difference between using maven -pl option and running maven from module level?
Maven multi module project cannot find sibling module
Is there a way to force artifactoryPublish task to publish the jar file first and then the pom file ? Or make pom file visible to others only when jar file is available.
We have multiple Git repositories where the jars are shared across the repositories. The Hudson build for them is triggered almost at the same time. So we have hit this case of gradle finding the pom but not the jar; But when we try to look into artifactory we see both. Just that we see jar gets uploaded couple of seconds later than the pom.
What is the best way to solve this (other than changing the time when Hudson build triggers).
Are there any plugins or ways to download the dependencies for a maven project from Jenkins? I am using Jenkins for a multi-module desktop application. Although I know I could just archive all dependencies, I don't see why there isn't the ability to download dependencies using maven which installed on the same machine as Jenkins. Preferably one would specify the location of a pom and then have the ability with one click to download all the dependencies for that pom. Can you do this? I do not need or want an entire binary repository for this feature.
Edit: I will try and rephrase this as I don't think people are understanding.
In Jenkins one has the ability to archive artifacts at the end of a build. Also in jenkins you have integration with maven. When building a jar in maven you have arguablly 2 options:
You can either use the assembly plugin which zips all .class files
together with those produced from your source code resulting in 1 jar
You can create a jar just source code which references all
dependency jars which are located in a separate folder.
In Jenkins one also has the ability to download the latest artifact. Now if I am using Option 2, I can either archieve just the jar which my sources produced, which I would say is more desirable for space and is the whole purpose of the archive functionality, or you can also archive the libraries too.
Here is the PROBLEM!! If I don't archive the libraries then I cannot easily run this jar, as it is a desktop application and its dependencies cannot be obtained in the same mannor as clicking on a link from jenkins. So lets say my question is what is the easiest way to obtain them? Extra info: assume jenkins is running as a server and you can't use artifactory or another server application, that seems to me to be massive over kill.
Use the maven plugin and create a maven job for your project. Jenkins will then use the maven command you provide in the job configuration to build the project. This means maven will download the projects dependencies and store them on the machine jenkins is running. Normally this would be <JENKINS_HOME>/.m2/repository. This way you get a local repository that only contains the dependencies of the projects you created maven jobs for.
How can I warm up the dependency cache of my maven tests? E.g. mvn test -DskipTests downloads some of the dependencies, but not all, e.g. some of the maven surefire plugin dependencies are only downloaded by mvn test.
I want to create a snapshot of my filesystem to boost up the execution of my tests. Therefore I'd like to have all the dependencies downloaded, but I want to achieve this without executing the tests itself.
Some of the dependencies that are only downloaded during mvn test and not by mvn test -DskipTests:
Downloading: http://repo1.maven.org/maven2/org/apache/maven/surefire/surefire-junit47/2.14/surefire-junit47-2.14.pom
Downloaded: http://repo1.maven.org/maven2/org/apache/maven/surefire/surefire- junit47/2.14/surefire-junit47-2.14.pom (4 KB at 13.9 KB/sec)
Downloading: http://repo1.maven.org/maven2/org/apache/maven/surefire/surefire-providers/2.14/surefire-providers-2.14.pom
Downloaded: http://repo1.maven.org/maven2/org/apache/maven/surefire/surefire-
The dependencies are resolved on a need-by-need basis:
The compile time ones for the application's code are downloaded as a very first step when you start the build.
Any plugins required are downloaded afterwards during the respective phases. This also includes their transitive dependencies.
The dependencies for the tests are downloaded when the tests are to be compiled and executed. These are the dependencies with <scope>test</scope>.
Therefore, at the point where you're at the test phase, you already must have the latest dependencies, unless you have them cached locally and installed and you're in offline mode.
To resolve all your dependencies, you can do:
mvn dependency:go-offline
To resolve all the plugins, you can do:
mvn dependency:resolve-plugins
maven surefire plugin gets downloaded only for mvn test because default Goal execution phase for this plugin is test.
I guess if you want to execute surefire plugin you can specify goal execution phase.
We have the following approach for filling the Maven cache on our Jenkins slaves:
Do a full build
Zip the ~/.m2 folder (which contains the cached instances)
Place it and extract it on the machine where you want to run your "warmed-up" tests
Automate steps above steps to keep it recent and up to date.