Building "out of source" with Maven - maven

Question

Summary:
I would like to know how to build Maven applications consisting of several Maven projects with all inputs from read-only source and all outputs to a given (temporary) folder, without breaking IDE (Netbeans) support. None of the proposed solutions work for me, so I ask in detail to explain my problem as good as possible.
In detail:
Since quite a while I try to get my Maven builds integrated in our productive building environment which has several requirements:
no internet access (all inputs must be under version control)
source tree is read-only
all build artifacts must be below a build directory
well defined toolchain (under version version control or in read-only Virtual Machine)
libraries shall not know which application use them (i.e. "no parent POM")
to support development cycle, preferably it should work with IDEs (Netbeans), optional
to support development cycle, preferably it should work incrementally,
optional
This is similar to Out-of-tree build with maven. Is it possible? and other questions related to maven build directories, but unfortunately none of the proposed solutions works here.
I have the "central repository" accessible via "file:" URL and as well as the "local repository" below the (temporary) build directory, but I did not find any way to have mavens "target" directories below the build directory without breaking anything else.
I have several applications that share some libraries. For each library I have an own "parent POM" (violating the DRY principle, because I found no better way). Each contain application and environment specific settings such as the distributionManagement repository path, which is defined using ${env.variable} to allow the build tool to inject the system and application specific values. Default values are provided to make developers using Netbeans happy.
Unfortunately this does not work for build directory.
I could set a build directory as in Maven: How to change path to target directory from command line?. Then all class files (of all libraries and applications) build by one parent POM will be put into one and the same directory. Maven can be ran once and works - but in any later run it will fail! At least unit tests fail, because Maven (surefire) will find all tests for all projects, not only the one Maven is currently processing. So it will try to run unit tests from the application while building a library A (which in my case fails because it also needs library B which is not provided as dependency in library A).
I also tried constructions like "-DbuildDirectory=$BUILDDIR/maven-target/\${project.name}". This works for compilation of sources, but again not for tests. Although Maven (3.1.1) evaluates ${project.name} correctly when storing the files, it passes it literally (!) to the classpath, so when compiling unit test java doesn't find the test objects ("error: cannot find symbol"). Having a symlink ${project.name} to the correct name of course works only for one project (library A for example), because I found no way how to change it from within a pom.xml.
Currently my closest approach is using a global target directory and clean it before each build, but obviously this is bad when developing using this build system (working incrementally without IDE).
Now I'm even considering generating the pom.xml files from the build system with filled values and copy all sources to the build tree. This should work for our defined release builds, but be uncomfortable during normal development (with IDE).
Is there any trick or is this really impossible with Maven?
EDITED to answer the questions
Thanks so much for spending time on my issue / help request! I answer as clearly as possible.
A team mate suggested to add the motivation for all this:
It should be possible to reproduce any release in ten or 20 years, even if internet changed.
Why do you need to change the value of "target"?
Because the source tree is read-only, Maven cannot create the directory at its default position. A writable build directory is available, but outside the source code tree.
Maven will not write to /src, so no problem there
"source tree" does not reference to the "src" folder within the sources / inputs for the build process, but to the whole structure - including POM files, resources and even binary libs if needed. So the directory that contains src also contains pom.xml and belongs to read-only VCS controlled "source tree".
Could you give an example of something that does not work with the standard project layout?
One different example of such a read-only source tree could be: burn a whole project structure (application with libaries) on a CD-R and ensure "mvn compile test package deploy" works.
(One motivation for the read-only source tree requirement is to ensure that the build process does not accidentally (or intentionally) manipulate source code, which could lead to automatically changing build artifacts by the number of repeats, breaking reproducibility.)
using a repository manager
As I understood, a repository manager is a complex service typically running on a host. I think using file URLs instead of such a complex service keeps the dependencies smaller. Please note that the whole repository has to be version controlled, so automatically updating must not work anyway, but as I understand is the advantage of a repository manager.
Do you use dependency jar's
Yes, sure. The libraries and applications depend on several JARs. Even Maven depends on several JARs, such as the compiler plugin.
If your request states that you need to build all dependencies yourself ?
Actually I'm afraid that this could be the case and yes, I'm aware this would be huge effort, unfortunately. Even with Debian packages this was quite difficult to realize and they put a lot of effort to make it possible. At the moment I don't see a realistic chance to build these libraries, unfortunately.
"...all build artifacts must be below a build directory"- Can you please explain that in detail?
The build process is supposed to create all files below a given build directory, for example /tmp/$USER/build/$PROJECTNAME/$PROJECTVERSION/$APPLICATION/$VARIANT/. The build system can create arbitrary structures below that, but is not supposed to change anything outside this. A list of files within this tree is defined as output (build result) and taken from there. Typically some binaries or ZIP files are copied to version control system.
Some call this "out of tree build", "build out of source" or similar.
well defined toolchain
The idea is to have a Virtual Machine image (under version control), for each build create a dynamic "clone" of it, install needed toolchains (from version control), for example the build tools, the Maven installation and its repository, give access to the specific version of the sources to be built (for example, in form of a read-only ClearCase view), run a build script entry point script, collect created outputs (for example some "release.zip" and "buildlogs.zip") and finally discard the whole virtual machine including all its contents. Of course the toolchains could be fully preinstalled in the image, it is just not done for practical reasons (and yes we also use Docker).
For impact analysis it's very important thing to know who is using which library ?
Yes, this surely is very true. However, the sources should abstract from it, and usually do. For example, the authors of JUnit dont' know everybody who is using it. We like to use a library in several projects at the same time, so we cannot mention a parent POM, because there are multiple parent POMs.
I hope I was able to explain better this time.
EDIT #2 to answer the new questions
Thanks again for your time getting through this long post! I hope this time I was able to explain the last missing bits and make it clear. I think some of the questions seem to be comments about the requirements. I'm afraid we could slip into a discussion.
In maven you have a directory structure like:
root
+- pom.xml
+- src
The root directory is under version control. So I don't understand the following:
"source tree" does not reference to the "src" folder within the
sources / inputs for the build process, but to the whole structure -
root
+- pom.xml
+- src
+-- ...
+- target
+-- ...
I added target" to the structre to better illustrate the problem. As you say, the root directory is under version control and thus read-only. Thus, root/pom.xml, root/src and root itself are read-only.
When Maven would try to create root/target, it will get an error (Read-only file system), because root and all its sub-folders are read-only.
Here even the parent folder is under version control and in case of ClearCase, there are even more additional read-only parent directories, but I think this does not matter here.
We are talking about terra bytes of artifacts....They have to be backed up but not checked into version control.
This is out of scope, but I try to answer nevertheless.
The one ClearCase instance I'm working on right now for example has 2.2 TB, so tera bytes, exactly. Backup may not be sufficient if standards require traceability of all changes between releases, for example. However I think this is out of scope of my question.
a file base repository does not work well and needed to be copied to each machine where you use it which is a hassle having a kind of network file system which is mounted to many machines. A repository manager works http(s) based which much more simpler to handle..
I think this is out of scope and not related to my question, but I try to answer anyway.
The problem is that for reproducibility, you would need to archive everything of this http(s) for the next ten or 20 years and keep it running. With history, because it might be required to reproduce a five year old state. Of course the needed packages might not be available in the internet anymore (remember Codehaus.org?). Challenging tasks.
a repository manager is much more simpler and keeps the conventions
Unfortunately it does not fulfill the requirements, or require a high price (managing additional virtual machines plus version control of the stored packages).
building all dependencies yourself I think that is simply not worth the effort. I don't a real advantage of that
I think such requirements are not uncommon (https://en.wikipedia.org/wiki/Source_code_escrow), but I'm afraid we are getting off topic and start discussing requirements.
If the JUnit team can access the request from maven central (which can be done) they can see who is using JUnit
I think its off-topic, but how should JUnit developers be able to know all "my" projects that use JUnit?
Where is the relationship to the parent pom?
I'm sorry if I caused confusion, this was a side note only. I think it is out of scope of my question. I'll try to answer nevertheless.
To avoid redundancy, one approach in Maven is using a parent POM and inherit from it. For example, you can define distributionManagement in a parent POM and make the library POMs inherit from it. When you have project specific distributionManagement requirements, you need project specific parent POM files. If two projects share one and the same library, which of the two project POMs should the library inherit (having as parent)? Having the setting distributionManagement in an aggregator POM does not work as it is not propagated.
EDIT #3 to explain read-only
Hi #JF Meier
thank you for your comment.
Read-only means it cannot be written to, so Maven cannot create the target directory. Because of this, javac cannot store the class files and compilation aborts.
Maybe it is too abstract, so let me give a full real example:
cat pom.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>test</groupId>
<artifactId>mavenproject1</artifactId>
<version>1.0-SNAPSHOT</version>
</project>
cat src/main/java/Hello.java:
public class HelloWorld {
public static void main(String[] args) { System.out.println("Hello, World!"); }
}
Now copy & paste from the shell (I cut a few boring lines):
steffen#node1:/view/dev_test_project_v1/vobs/playground/steffen/minimal_pom $ mvn compile
[...]
[ERROR] Failure executing javac, but could not parse the error:
javac: directory not found: /view/dev_test_project_v1/vobs/playground/steffen/minimal_pom/target/classes
This is one problem. A small detail I have to correct, it is not wshowing the Read-only file sytem error, I think a bug, but in strace we can see it:
21192 mkdir("/view/dev_test_project_v1/vobs/playground/steffen/minimal_pom/target/classes", 0777) = -1 ENOENT (No such file or directory)
21192 mkdir("/view/dev_test_project_v1/vobs/playground/steffen/minimal_pom/target", 0777) = -1 EROFS (Read-only file system)
21192 mkdir("/view/dev_test_project_v1/vobs/playground/steffen/minimal_pom/target", 0777) = -1 EROFS (Read-only file system)
This leads to the javac: directory not found error.
It is read-only:
steffen#node1:/view/dev_test_project_v1/vobs/playground/steffen/minimal_pom $ mkdir -p target/classes
mkdir: cannot create directory ‘target’: Read-only file system
The mkdir tool correctly shows the error message.

Answer 1

Out-of-tree builds with Maven and no Internet
In case someone else faces similar requirements, I describe my solution, in form of an example. I understand and happily accept that most people surely do not want this or at all need "out-of-tree" builds with Maven. This is for the very few others, like me, who have no choice.
So this is not suited for "normal Maven builds", but only for the specific requirements described in the question (no internet access, everything read-only except a /tmp folder). Such a configuration could be a build system in a virtual machine which is ran from a VW template, dynamically instantiated by Jenkins jobs.
A word about ClearCase
This approach is not ClearCase specific, but the example uses ClearCase paths.
The following example uses a read-only ClearCase view toolchain_view mounted to /view/toolchain_view/ (containing Maven, all Java packages and other build tooling) and a read-only source code view (containing an application) available below "/view/steffen_hellojava_source_view/". Instead of "/view/toolchain_view/", someone could use "/opt/vendor/name/toolchain/version/" or such and for the sources use any other version control system, so this approach is not ClearCase-specific.
For those who don't know ClearCase a word about it: The view behaves similar like a NFS file system, where the server selects the file content based on a version description. The file contents itself is in a database called Versioned Object Base (VOB). Using a so called Configuration Specification (CS) someone can define which content version to show for which file (element) in form of selection rules (such as selecting by a LABEL). For example:
---[ toolchain-1_4_7_0.cs ]---------------------------------------->8=======
element /vobs/playvob/toolchain/... TOOLCHAIN_VERSION_1_4_7_0
# TOOLCHAIN_VERSION_1_4_7_0 automatically also selects:
# element /vobs/playvob/toolchain/3p/maven/... TOOLCHAIN_MAVEN_3_1_1_0
# element /vobs/playvob/toolchain/3p/maven-repo/... TOOLCHAIN_MAVENREPO_1_0_1_0
# element /vobs/playvob/toolchain/3p/java/... TOOLCHAIN_JAVA_1_7_0_U60
=======8<-------------------------------------------------------------------
Now a ClearCase view such as "toolchain_view" can be created and configured to use these selection rules cleartool -tag toolchain_view -setcs toolchain-1_4_7_0.cs. In the example it is mounted as /view/toolchain_view/, which prefixes the element (file) paths.
Configuring Maven
Now we need to configure Maven to
use our file structure /view/toolchain_view/vobs/playvob/toolchain/3p/maven-repo/ as only Central Repository
store the local repository below /tmp and
have the target directories also below /tmp
The first two can be configured in Maven settings file, but apparently the last one unfortunately seems to require specific changes in the POM file.
mvn_settings.xml
Here an excerpt from a --global-settings file for Maven. I compactified it a bit. Essential is the usage of file:/// URLs for both Central Repositories and also enforcing this as one and only mirror:
<!-- Automatically generated by toolchain creator scripts.
This sets the maven-repo version to the correct value, for example
toolchain version 1.4.7.0 defines Maven repo version 1.0.1. -->
<settings ...>
<localRepository>${env.MAVEN_LOCAL_REPO}</localRepository>
<profiles>
<profile>
<id>1</id>
<activation><activeByDefault>true</activeByDefault></activation>
<repositories><repository>
<id>3rdparty</id><name>third party repo</name>
<url>file:///view/toolchain_view/vobs/playvob/toolchain/3p/maven-repo/</url>
<snapshots><enabled>true</enabled><updatePolicy>never</updatePolicy></snapshots>
<releases><enabled>true</enabled><updatePolicy>never</updatePolicy></releases>
</repository></repositories>
<pluginRepositories><pluginRepository>
<id>3rdparty</id><name>third party repo</name>
<url>file:///view/toolchain_view/vobs/playvob/toolchain/3p/maven-repo/</url>
<snapshots><enabled>true</enabled><updatePolicy>never</updatePolicy></snapshots>
<releases><enabled>true</enabled><updatePolicy>never</updatePolicy></releases>
</pluginRepository></pluginRepositories>
</profile>
</profiles>
<!-- Unfortunately **required** for file-based "central repository": -->
<mirrors><mirror>
<id>dehe_repo_1.0.1</id><name>Vendor Name Central 1.0.1</name>
<url>file:///view/toolchain_view/vobs/playvob/toolchain/3p/maven-repo/</url>
<mirrorOf>*,!3rdparty</mirrorOf>
</mirror></mirrors>
</settings>
To ease deployment we could use paths that include version numbers, such as /opt/vendor/name/toolchain/1.4.7.0/mvn_settings.xml, which could be provided by own Debian packages.
Pointing localRepository to a build-specific (temporary) directory by setting an environment variable allows building when everything is read-only except the temporary directory, which is needed for "out-of-tree" builds.
When building with IDE (Netbeans) we can use this settings file as well, but usually it is more comfortable not to do so. In this case, however, someone has to pay attention not to accidentally add dependencies. If these are not included in the pinned Maven Repo, compilation on the build system will break.
hellojava/pom.xml
To support out-of-tree builds, we also need to move the target folders out of the read-only source tree (normally they are created beside pom.xml and src in the Java package directory, which itself is under version control and thus read-only here). This is implemented by using two properties: buildDirectory and deployDirectory. The default buildDirectory is the normal target folder, so when not setting buildDirectory, Maven builds as normal. This is nice, because we don't need a specific POM file for the IDE (Netbeans).
Surefire generates unit test reports, which of course also need to go to our build directory.
<project>
...
<properties>
<project.skipTests>false</project.skipTests>
<project.testFailureIgnore>false</project.testFailureIgnore>
<buildDirectory>${project.basedir}/target</buildDirectory>
<deployDirectory>defaultDeploy</deployDirectory>
</properties>
...
<build>
<!-- https://stackoverflow.com/a/3908061 -->
<directory>${buildDirectory}/hellojava</directory>
<!-- https://stackoverflow.com/a/6733858/9095109 -->
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.12.4</version>
<configuration>
<skipTests>${project.skipTests}</skipTests>
<testFailureIgnore>${project.testFailureIgnore}</testFailureIgnore>
<workingDirectory>${project.build.directory}/test-run</workingDirectory>
</configuration>
</plugin>
</plugins>
</build>
...
<distributionManagement>
<repository>
<id>build-integration</id>
<name>Deployment Repository</name>
<url>file:///${deployDirectory}</url>
</repository>
</distributionManagement>
...
</project>
Putting it all together
The values for the properties in this POM file now have to be set by command line parameters, and MAVEN_LOCAL_REPO has to be configured, for example:
#!/bin/bash
# Normally in a wrapper which is automatically generated by toolchain creator.
# The path to the pom.xml and build directories then of course are parameters.
export JAVA_HOME="/view/toolchain_view/vobs/playvob/toolchain/3p/java/"
export PATH="/view/toolchain_view/vobs/playvob/toolchain/bin:$PATH"
# or: export PATH="/opt/vendor/name/toolchain/1.4.7.0/bin:$PATH"
: ${MAVEN_LOCAL_REPO:="$HOME/.m2/repository"}
: ${MAVEN_OPTS:="-XX:PermSize=64m -XX:MaxPermSize=192m"}
export MAVEN_LOCAL_REPO="/tmp/steffen-build/hellojava/mavenbuild/repo"
/view/toolchain_view/vobs/playvob/toolchain/3p/maven/bin/mvn -e \
--global-settings /view/toolchain_view/vobs/playvob/toolchain/mvn_settings.xml \
-DbuildDirectory=/tmp/steffen-build/hellojava/mavenbuild/target/ \
-DdeployDirectory=/tmp/steffen-build/hellojava/mavenbuild/output/ \
-Dproject.skipTests \
-f /view/steffen_hellojava_source_view/hellojava/pom.xml \
compile package deploy
Now /tmp can be a RAM disk on a "read-only system". All artifacts are written below the dedicated build directory. We could have toolchain and complete source tree on a DVD or read-only NFS archive server, and still could compile it, without Internet access. This should still work in 20 years, even if Maven Central has been renamed or whatever.
Of course wrapper scripts can hide all the details. In my case, they are integrated in a cmake-based build system and the top level build directory is configured by a Jenkins job.
Checking Requirements
For each requirement from the original question we can check whether this approach meets it:
no internet access (all inputs must be under version control)
OK, "downlading" from file:///
source tree is read-only
OK, works with read-only /view tree
all build artifacts must be below a build directory
OK, Maven Local Repository is configured by setting MAVEN_LOCAL_REPO
well defined toolchain (under version version control or in read-only Virtual Machine)
OK, is in /view/ or /opt and all versions are "pinned" (fixed)
libraries shall not know which application use them (i.e. "no parent POM")
OK, but not nice, since it is needed to adjust all POM files
to support development cycle, preferably it should work with IDEs (Netbeans), optional
OK, same POM files work for IDEs
to support development cycle, preferably it should work incrementally, optional
OK, as long as the build tree and the Local Repository are kept, Maven works incrementally
So all the (very specific) input requirements are fulfilled. Good. So this implements out-of-tree builds without Internet.

Answer 2

So based on your update I add another update here:
Why do you need to change the value of "target"?
Because the source tree is read-only, Maven cannot create the
directory at its default position. A writable build directory is
available, but outside the source code tree.
You seemed to misunderstand the idea of target directory or you have a misunderstanding how Maven works.
The target directory is intended to contain all generated/compiled things which is NOT checked into source control. Or in other words target directory is by default ignored during checkin and will never and should never being checked into version control.
Maven will not write to /src, so no problem there
In maven you have a directory structure like:
root
+- pom.xml
+- src
+-- ...
The root directory is under version control. So I don't understand the following:
"source tree" does not reference to the "src" folder within the
sources / inputs for the build process, but to the whole structure -
including POM files, resources and even binary libs if needed. So the
directory that contains src also contains pom.xml and belongs to
read-only VCS controlled "source tree".
Coming to the next point:
As I understood, a repository manager is a complex service typically
running on a host. I think using file URLs instead of such a complex
service keeps the dependencies smaller. Please note that the whole
repository has to be version controlled, so automatically updating
must not work anyway, but as I understand is the advantage of a
repository manager.
The whole repository should be version controlled. That sounds like you don't have experience with real builds in corporate environments. We are talking about terra bytes of artifacts....They have to be backed up but not checked into version control.
The reason is simply cause each artifact is uniquely defined by it's coordinates which groupId, artifactId, version.
Apart from the argument you gave the complexity of setting up is simpler than you think. Apart from that a file base repository does not work well and needed to be copied to each machine where you use it which is a hassle having a kind of network file system which is mounted to many machines. A repository manager works http(s) based which much more simpler to handle..
Coming to next point:
The build process is supposed to create all files below a given build
directory, for example
/tmp/$USER/build/$PROJECTNAME/$PROJECTVERSION/$APPLICATION/$VARIANT/.
Maven does this in target directory..
The build system can create arbitrary structures below that, but is
not supposed to change anything outside this. A list of files within
this tree is defined as output (build result) and taken from there.
Typically some binaries or ZIP files are copied to version control
system. Some call this "out of tree build", "build out of source" or
similar.
As I mentioned before the resulting artifact I would not recommend to put them under version control cause a repository manager is much more simpler and keeps the conventions...
Coming back to your point of building all dependencies yourself I think that is simply not worth the effort. I don't a real advantage of that...In particular cause you can consume all artifacts based correct coordinates from Maven repositories (or from your own repository manager inside your organisation)...
Next point:
Yes, this surely is very true. However, the sources should abstract
from it, and usually do. For example, the authors of JUnit dont' know
everybody who is using it.
If the JUnit team can access the request from maven central (which can be done) they can see who is using JUnit..This is much more simpler inside an organisation by using the access log of the repository manager and extract the usage information...
What I don't understand is:
We like to use a library in several
projects at the same time, so we cannot mention a parent POM, because
there are multiple parent POMs.
This is called a simple dependency in Maven. Where is the relationship to the parent pom ? You seemed to misunderstand / or have no understand how Maven works.. ?

Answer 3

Let me add the following to khmarbaise's answer.
I don't see your "in ten years reproducible" issue with the standard Maven structure. You just:
Check out the source from the version control system to an empty directory.
Build your project with Maven in this directory.
Deploy the results to wherever you want.
The source in the version control system is not changed, everything will work exactly the same every time. You only have to make sure that you only use release versions in your dependencies (not SNAPSHOT versions), so that Nexus/Artifactory give you the same artifact every time.