I am setting up a CI/CD with the following components:
-Bitbucket
-Jenkins
-Docker
-Maven
The desired flow:
Committing code to Bitbucket
Webhook Jenkins on changes
Jenkins pipeline does the following:
mvn install 2 projects, SDK+API
takes the API jar and build image
pushes image to repository
deploys service or container to docker
Where I am currently stuck is the following:
I have a PROJECT-A, that has multiple dependency projects that has to be maven installed, before maven installing PROJECT-A and creating an image of it.
Could anybody advise on what is the best practice here?
I have googled an it's say that my only feasible and maintainable option should be using Parent POM. However I failed to understand how do I do that.
<modules>
<module>project1</module>
<module>project2</module>
<module>project3</module>
</modules>
Even if your project would be structured differently with parent pom, where would those additional projects come from? In general mvn install does something different then you mean in this question -
install: install the package into the local repository, for use as a dependency in other projects locally
The best practice here (and missing element) is a package repository. For example your private artifactory or nexus. You would mvn install all the packages to it and maven would automatically resolve the dependencies from it basing on its POM and appropriate configuration.
Related
I've discovered the wonderful test-jar facility in Maven: https://maven.apache.org/plugins/maven-jar-plugin/examples/create-test-jar.html
But it may so happen that one project needs to use the test-jar of another project. From https://stackoverflow.com/a/6469256/421049 and experimentation, it would seem that using mvn install does not install the test-jar to the local ~/.m2/repository. So how does one project on my machine use the test jars of another project not in the same aggregate POM?
Yet it would seem from Maven deploy not to upload test jar that deployment of a project to Maven Central does in fact deploy the test-jar? So I can deploy it to Nexus but not install it locally? And if I deploy it to Nexus, will my local project using a dependency of <type>test-jar</type> go find it on Maven Central?
It turns out that maven-jar-plugin does in fact install the test-jar (e.g. foo-1.2.3-tests.jar) in the local Maven repository ~/.m2/repository/.... Wonderful!
My problem is that I had inadvertently configured the maven-jar-plugin to be in a separate profile named release. (I had copied and pasted to the wrong location in my POM.) That's why the test-jar didn't show up in my local repository after a mvn install, and that's why it suddenly showed up later (after I used -P release once in testing), and I thought I had just missed it when I looked the first time.
I move the maven-jar-plugin to the <build> section and everything is working fine: the test-jar gets put into the local maven repository using mvn install.
In my case, I was setting maven.test.skip for a particular build profile. That property actually causes tests to not be compiled/installed, thus also preventing their deploy. The solution was to set the skipTests property instead.
I cloned the git repository of Apache ActiveMQ Artemis project (https://github.com/apache/activemq-artemis) and then typed
mvn -Ptests test -pl :integration-tests
I was surprised to see log messages like the following
...
Downloading: http://repository.apache.org/snapshots/org/apache/activemq/artemis-selector/1.4.0-SNAPSHOT/artemis-selector-1.4.0-20160625.030221-11.jar
Downloading: http://repository.apache.org/snapshots/org/apache/activemq/artemis-core-client/1.4.0-SNAPSHOT/artemis-core-client-1.4.0-20160625.030211-11.jar
...
Since e.g. artemis-core-client is contained in the git repository I cloned in the beginning, I'd have expected maven just builds it from there.
That way, when I make changes in the core client source, they get picked up by the integration tests.
Instead, maven is downloading the jar from the repository.
Question: How do I configure maven to always build all modules that are in the git repository and download only "true" dependencies, which I mean things not in the git repository?
You are not executing the Maven build on the main project, on the main pom.xml which indeed defines the artemis-selector and artemis-core-client modules, among others.
You are executing the Maven build on the tests and its pom.xml, where only tests modules are defined. This is a side/test project, which has as parent the previous pom file, but it doesn't play any role in its parent modules definition. Hence, dependencies are not resolved as modules but as Maven dependencies.
You should firstly install (via mvn clean install) the former project, so that libraries will be available in your local Maven cache (hence no downloading would be triggered), then execute the tests project.
Check the Maven docs for a inheritance vs aggregation difference to further clarify it.
From the Stack Overflow, the follow threads could also be interesting:
What is the difference between using maven -pl option and running maven from module level?
Maven multi module project cannot find sibling module
I'm trying to build a p2 repository from Tycho feature artifacts which are deployed in a remote Maven repository, without having to install the artifacts into the local Maven repository first (as in Tycho fails to resolve reference from product to eclipse-feature from a different reactor build), and without having to build all features and the repository together in a single reactor build.
Background
I have a multi-module Tycho project that builds several Eclipse plugins and features.
So that I can build each module separately - and so that I can reference OSGI artifacts in our Nexus Maven repository - I have enabled <pomDependencies>consider</pomDependencies> in my target platform, and added Maven dependencies between the modules or to the repository artifacts as usual with <dependency/> elements.
This works well - I can build the features or run the plugin tests without their dependant plugins being either in my local Maven repository or in the same reactor build. For example, when I run mvn test on a plugin test project, the relevant dependencies will be downloaded from Nexus and Tycho will happily resolve the Import-Packages in my manifest against these, build everything and run the tests. So far so good.
I would like to generate a p2 repository from these features so that I can install them in Eclipse from an update site, and the advertised way to do this is with the eclipse-repository packaging type. But here the plan falls down - Tycho doesn't seem to be able to resolve feature dependencies when building repositories in the same way as it can resolve plugin dependencies when building features. All attempts yield:
[ERROR] Cannot resolve project dependencies:
[ERROR] Software being installed: my.eclipse.repository raw:0.0.1.'SNAPSHOT'/format(n[.n=0;[.n=0;[-S]]]):0.0.1-SNAPSHOT
[ERROR] Missing requirement: my.eclipse.repository raw:0.0.1.'SNAPSHOT'/format(n[.n=0;[.n=0;[-S]]]):0.0.1-SNAPSHOT requires 'my.prj.eclipse.project.feature.feature.group 0.0.0' but it could not be found
There are two ways I have successfully built the p2 repository:
As part of the same reactor build. If I make the eclipse-repository a module within the Tycho multi-module project, and build the whole project at once with e.g. mvn verify, the features are resolved fine. But I don't want to do this. I would prefer to build modules individually. This means our CI can have an indicator for each module, and we can immediately see what module tests have failed in; it gives us opportunities for parallelising builds; and we avoid having to be constantly running builds on modules that haven't changed. It would be a shame to have to use a monolithic Maven build.
If I install the Tycho project into my local Maven repository, by running mvn install on the dependency. But I don't want to do this either, because this would mean the build is inherently irreproducable, as it would be sensitive to the state of the local repository. Our CI is currently set up to maintain a Maven repository per job and to completely wipe it at the start of execution, to shield us from this potential messiness.
So my question is: is there a third way? Is there any way I can get the Tycho plugin responsible for building eclipse-repository packaging types to download features from a remote Maven repository? Or any other way I can build the p2 repository from plugins that have been individually built and deployed to the Maven repository?
Things I've tried include:
specifiying the Maven feature depedencies as both jar and eclipse-feature
explicitly adding the features to the target platform, like
...
<artifactId>target-platform-configuration</artifactId>
<version>${tycho.version}</version>
<configuration>
<dependency-resolution>
<extraRequirements>
<requirement>
<type>eclipse-feature</type>
<id>my.prj.eclipse.project.feature</id>
<versionRange>0.0.0</versionRange>
</requirement>
...
The closest thing I've found to a decent solution is have a multi-module Tycho project that just contains the repository and features.
feature-project
|- feature1 (eclipse-feature)
|- feature2 (eclipse-feature)
|- repository (eclipse-repository)
Building this works - all plugins added to the top-level POM are downloaded from Nexus, available for inclusion in each feature and included in the generated repository.
However this is far from ideal because I can no longer store my features logically alongside my plugins; they need to be in separate project hierarchies. Attempting to build the features and repository separately, like with mvn clean verify -pl :feature1,feature2,repository, fails presumably due to Bug 380152.
Is there a better way? Any help would be gratefully received.
Many thanks
(As an aside: building the repository with mvn clean verify -Dtycho.localArtifacts=ignore will succeed if the features are present in the local Maven repository, and won't show you the warning that artifacts are being resolved from the local repo... is this a bug?)
I am pretty impressed by your thorough analysis. You've almost got everything covered which is possible with the current Tycho version (0.22.0) - except for the solution which is so unintuitive that I wouldn't have expected anyone to be able to guess it (see below). Note however that there is a small fix required to also make the solution work for SNAPSHOT artifacts.
But first, I'd like to provide some technical (and historical) background for what you have observed:
pomDependencies=consider only works for plug-ins: The use case for this functionality was to allow referencing plug-ins (or more precisely OSGi bundles) from Maven repositories. So when the flag is set and the project has dependencies to JARs, Tycho will check if they are OSGi bundles, generate the p2 metadata for them on-the-fly, and add them to the target platform. There is no similar support for feature JARs because these usually don't exist in Maven repositories.
But what about Tycho-built projects? These may deploy into Maven repositories! Yes, this is true, and this is why I tried to extend the pomDependencies concept to allow for what you are trying to do. The idea was that every time Tycho considers a POM dependency for the target platform, it also checks if the p2 index files ...-p2metadata.xml and ...-p2artifacts.xml exist. However this turned out to infer a massive performance penalty because it generally takes very long for a Maven repository server to figure out that an artifact does not exist. So the remote download was disabled, and replaced with a look-up in the local Maven repository. In this way, two Tycho builds could set -Dtycho.localArtifacts=ignore and would still be able to exchange the artifacts specified in the POM via the local Maven repository.
Knowing these implementation details, we get to the following solution: Instead of only adding a POM dependency from the repository to the feature artifact, you also need to add dependencies to the p2metadata and p2artifacts files. Example:
<dependencies>
<dependency>
<groupId>myproject</groupId>
<artifactId>myproject.feature</artifactId>
<version>0.1.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>myproject</groupId>
<artifactId>myproject.feature</artifactId>
<version>0.1.0-SNAPSHOT</version>
<classifier>p2metadata</classifier>
<type>xml</type>
</dependency>
<dependency>
<groupId>myproject</groupId>
<artifactId>myproject.feature</artifactId>
<version>0.1.0-SNAPSHOT</version>
<classifier>p2artifacts</classifier>
<type>xml</type>
</dependency>
</dependencies>
This makes Maven also download these p2 index files, so Tycho recognizes the main artifact as Tycho artifact. In this way, you can also get an eclipse-feature into the target platform via POM dependencies - at least almost: With 0.22.0, the repository build passes, but the feature.jar artifact is missing. I already debugged this issue, and it is easy to fix.
Obviously the syntax with three <dependency> elements for every actual dependency is not nice. It should be possible to boil this down to a single p2artifacts element - but this is more work. In case you are interested in this feature, you could open an enhancement request in Tycho's issue tracker.
I have a master POM which has many modules defined in it.
I want to make some minor changes to the master POM and deploy the new Master POM to our nexus instance.
Everything is confgured ok to allow this, but when I do a mvn clean install deploy all the modules are built and deployed too. I dont want this.
How do I tell maven to just deploy the master POM and not build/deploy the modules? Can this be done?
Use this option:
-N,--non-recursive Do not recurse into sub-projects
Reference: http://maven.apache.org/guides/mini/guide-multiple-modules.html
I want to share all the External Jars currently being managed by MAVEN with my other team members. I am using Mercurial as my SCM and i am trying to figure what is the easiest way to commit my entire project (include libs) to a repository so that my team members can clone and get running without severe eclipse configuration?
Maven is there in order to help you retrieving libraries. So that, all you have to do is to commit all your files including the pom.xml (.hg should contains everything in target, and other unrelevant files)
Then your project members can pull the sources and run mvn eclipse:eclipse (see eclipse & maven.
And finally import the project in Eclipse.
That was is they need sources...
If they only need the jars, you must put in your infrastructure a company repo that will handle your deployment using mvn deploy. Some information there maven repo::Intro, take special care at the wagon you could use (ftp, ...)
This way, when you're done with your devel and you have pushed your code, you just have to deploy the jar on your repo.
Doing so, your project member'll have to run mvn -U eclipse:eclipse or any goal to update their local repository with your lastest deployed version.