I am trying to use caching in gitlab runner, which builds a Maven Java project. Currently Gitlab runner only allow caching specific paths defined in gitlab yaml file in the cache: clause. When maven builds projects, it generate everything inside target/ folder, which are untracked files in git. So I can simply use untracked: true option to cache everything under target/ folder. The purpose of caching is to skip compiling the files, which have already been compiled by maven under the target/ folder.
However this cache amounts to about 6GB, which is completely unreasonable for its size and time required to create and restore such a giant cache. It caches all jar and war artifacts built during compiling multi-module maven project. However, maven only needs .class files to check changes for re-compilation
So if their was some way using which I can cache only *.class files, and make them available in subsequent builds, then maven could check the .class files and skip re-compiling unchanged files and cache size would also be pretty small. Currently gitlab-runner only allow specifying absolute paths for caching. It does not support regex patterns for paths such as \.class$ (which would have been very useful).
Is there any way I could cache only specific file types using gitlab runner yaml settings?
So based on cascaval's comment, I was able to figure out a solution.
At the end of maven build I ran a command to clean all build artifacts created by maven, which are not used for checking stale status of .java resources. Here is what I wrote -
cd ./projects/directory
find . | grep --perl-regexp --regexp='\/target\/(?!classes|maven)' | xargs rm --recursive --force
This saves all .class files in target/classes folder, including folder structure and also files in maven-status folder, which are probably used by maven to check file status for recompilation.
Related
I'm using gitignore, and I'm good at push and pull for personal management.
The focus is on my own creation, so I have all the configuration files, so it runs well locally.
However, if I "Clone" this project and download it to an empty folder, it cannot be run anywhere.
It seems as if there is no configuration file necessary for project configuration or server execution.
I am attaching my gitignore.
Is this what gitignore originally intended?
I want to clone a project so that it can run.
Do I have to delete gitignore to do that?
p.s If I delete gitignore and clone all projects (including build artifacts) it works fine.
use tool & etc : Intellij, Spring, Tomcat
### Java template
*.class
# Package Files #
*.jar
*.war
*.ear
### macOS template
*.DS_Store
.AppleDouble
.LSOverride
# IntelliJ project files
.idea
.idea/*.xml
*.iml
out
gen
build
rebel.xml
# Compliled files
/target/
**/target
/example/
# Gradle
.gradle
/build/
.gradletasknamecache
Normally, you don't distribute compiled binaries with your git repo. The convention is to either have users compile the code locally, or distribute that code using a package management system like Maven, Ivy, npm, Nuget, pip or gem. This isn't a hard and fast rule however.
I want to store some additional files in the JAR that gets created. Those files are in a directory that is a subdirectory of a repository which is pulled in via a git submodule.
I want to copy that submodule to my src resources directory before compiling, but I also want to make sure that any old files at that location are removed first.
How can that be achieved best with Maven plugins? I did not find any option to remove any destination files with the copy-resources goal of the maven-resources-plugin and I could not get the maven-clean-plugin to run right before the copy-resources either. So how does one accomplish such a trivial task with Maven?
UPDATE: as mentioned above, the reason why I want to do this is because what is copied should become part of what gets added to the resulting jar (and could potentially be part of what gets compiled). So I need to copy these files into the src directory and NOT the target directory. What should get copied before each build is the input to the build, not an additional output.
There is one flaw in your approach, and it probably explains most of the obstructions you encountered.
During a build, the only directory in which you may write is target. Copying files to src or changing them is strictly discouraged.
The target folder is erased by clean, so no need to tidy up yourself or to manage old files.
I hope someone can help me with a simple setup of maven CI scripts for GitLab.
I tried to search stackoverflow and google, which results in several questions and answers, but either they seem to be completely different or not that I understand them.
I have a simple setup of two projects. project B depends on project A (= pom packaging).
I have in the runner configuration /etc/gitlab-runner/config.toml the line with the volumes added
[[runners]]
...
[runners.docker]
...
volumes = ["/cache", "/.m2"]
...
my .gitlab-ci.yml for both projects look like this
image: maven:3.6.1-jdk-12
cache:
paths:
- /.m2/repository
- target/
variables:
MAVEN_OPTS: "-Dmaven.repo.local=/.m2/repository"
maven_job:
script:
- mvn clean install
with this - the first project builds correctly and I can see that the caching is working, as it does not download all maven related plugins for building the project, when executed again and again.
It also states
[INFO] Installing /builds/end2end/projectA/pom.xml to /.m2/repository/de/end2end/projectA/0.4.4-SNAPSHOT/projectA-0.4.4-SNAPSHOT.pom
It reports though at the end
WARNING: /.m2/repository: not supported: outside build directory
WARNING: /.m2/repository/classworlds: not supported: outside build directory
WARNING: /.m2/repository/classworlds/classworlds: not supported: outside build directory
WARNING: /.m2/repository/classworlds/classworlds/1.1-alpha-2: not supported: outside build directory
WARNING: /.m2/repository/classworlds/classworlds/1.1-alpha-2/_remote.repositories: not supported: outside build directory
[...]
When executing projectB, the job fails with the info, that it cannot find projectA.
So - what is wrong with the configuration of the runner / .gitlab-ci.yml files ?
I tried
cache:
paths:
- .m2/repository
which removes the warnings, but then the projectA gets in its local .m2 installed
[INFO] Installing /builds/end2end/projectA/pom.xml to /builds/end2end/projectAt/.m2/repository/de/end2end/projectA/0.4.4-SNAPSHOT/projectA-0.4.4-SNAPSHOT.pom
and projectB fails with the same error as above.
In fact, as described in gitlab doc, you use the dynamic storage so the volume is shared between subsequent runs of the same concurrent job for one project. I you want to share data between projects you must use the host-bound storage.
For the warning, the cache is only for working directory, so absolute path like /.m2/repository is not supported. In your case, you don't have to use cache for maven repository because you use a volume.
I downloaded maven gz file, unzipped same, but i dont know where the m2 is stored. I imagine im missing a step but i cant see what one?
Is there aninstall script etc?
[root#atddpvm5 apache-maven-3.5.4]# cd /var/tmp/apache-maven-3.5.4/
[root#atddpvm5 apache-maven-3.5.4]# ls
apache-maven DEPENDENCIES doap_Maven.rdf LICENSE maven-builder-
support maven-core maven-model maven-plugin-api
maven-resolver-provider maven-settings-builder NOTICE README.md
CONTRIBUTING.md deploySite.sh Jenkinsfile maven-artifact maven-compat
maven-embedder maven-model-builder maven-repository-metadata maven-
settings maven-slf4j-provider pom.xml src
By default the .m2 folder is stored in the home folder of the user. In this case since you are using root, the path is most likely /root/.m2. You also have to use the -a switch with ls to see that folder, since it's a hidden folder (it starts with a .). Note that the folder will only be created on the first usage of Maven, i.e. when you call a maven command on a maven project, like mvn clean install.
Additionally it looks like you have downloaded the source distribution of Maven, which only makes sense if you want to work on Maven itself. You might want to download the binary distribution, if you just want to use it.
I'm using the maven-compiler plugin to generate my .jar
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.2</version>
When I do a clean install it will generate folders such as "generated-sources", "maven-archiver", "maven-status" amd "classes" in addition to my .jar
How could I automatically deleted those folders after the install or prevent them from being generated?
You cannot prevent those folders from being generated as they are essential to making the build work. generated-sources most likely contains Java source code that was generated during the build and is needed to make the rest of the code compile; classes contains the compiled Java source code that was under src/main/java and is needed to make a subsequent JAR or WAR, etc. So, without those folders, the build cannot properly work.
However, they are inherently temporary. In fact, the whole target folder is temporary. It contains data that is generated / copied at build-time and is needed to make the final artifacts. This is why it is generally a good idea to always clean before building a Maven project: it makes sure that this build folder is cleaned so that new fresh data is created (otherwise, it might rely on old build data, potentially making hard to track down bugs).
Once the final artifacts are created, they will be the only one to be considered when installing or deploying the project. If you really want to get rid of those files after the build (but I don't see why), you could always run mvn clean install clean. This will delete the target folder once the project's artifacts are installed.