I'm using GitLab CI to build a C# solution and try to pass some build artifacts from one build stage to another.
The problem is, that the artifacts are not located in a single directory but in different subdirectories, which however all have the same names bin/ or obj/.
My .gitlab-ci.yml looks like the following:
...
stages:
- build
- test
build:
stage: build
script:
CALL %MSBuild% ...
artifacts:
paths:
- /**/bin/
- /**/obj/
expire_in: 6 hrs
test:
stage: test
dependencies:
- build
...
I tried to capture the artifacts using different ways, e.g.
**/bin/
**/obj/
(invalid syntax), or
.*/bin/
.*/obj/
but that one did not find any artifacts, just as /**/bin/ and /**/obj/, giving me following errors:
Uploading artifacts...
WARNING: /**/bin/: no matching files
WARNING: /**/obj/: no matching files
How can I specify a subdirectory pattern to be scanned for artifacts? Or is this even possible at all?
Simply using
artifacts:
untracked: true
is not an option, because of a huge untracked packages/ subdirectory, which causes artifacts upload to fail because of a too large archive:
Uploading artifacts...
untracked: found 4513 files
ERROR: Uploading artifacts to coordinator... too large archive id=36 responseStatus=413 Request Entity Too Large token=...
FATAL: Too large
The gitlab-ci-multi-runner build runner is built using Go and currently uses filepath.Glob() to scan for any specified artifacts in file_archiver.go.
Go doesn't seem to support the double star glob expression as discussed in another question here at SO. So there seem to be no way to use a full-featured **/bin expression at the moment.
Because however all my projects are located at the same level below the solution root, it is still possible to use something like
artifacts:
paths:
- "*/bin"
- "*/obj"
Note that the quotes (") seem to be required, as well as no trailing path separator at the end.
It should also be possible to explicitly add more levels by adding more globbing expressions (as described here):
paths:
...
- "*/obj"
- "*/*/bin"
- "*/*/obj"
...
GitLab is tracking this issue here, and will possibly be fixed in a future version.
UPD: due to mentioned issue is closed and due to documentation, since GitLab runner 13.0 it supports doublestar.Glob.
So it is now possible to write something like this:
paths:
- "**/bin"
- "**/obj"
And it will include all nested directories with names bin and obj in artifacts
Related
I have a GitLab pipeline setup that has a package step to do a maven build during the tag event and a release to upload the jar to the GitLab generic package registry using curl and GitLab-release cli.
What I'm expecting to happen is a cache of the .m2 to be loaded into the package step to allow the mvn clean package to do its thing. Then archive the created jar and test results only.
The release step should begin clean with no git clone, no cache and only the jar and test results.
Instead the 'find .' shows the release step contains everything including
Git directory (.git)
Full checked out repository
.m2 cache
target (fully built as the Package step produced)
From the cache documentation (https://docs.gitlab.com/ee/ci/caching/) on GitLab it states
Archive: 'dependencies' keyword to control which job fetches the artifacts
Disable Cache uses the 'cache: []'
Why is GitLab putting so much content into the release job? The release job fails at times because its finding multiple Jar files from previous tags (IE the clean and the archiving are holding past version).
gitlab-ci.yml
variables:
MAVEN_CLI_OPTS: "-s $CI_PROJECT_DIR/.m2/settings.xml"
MAVEN_VERSION_PLUGIN_VERSION: 2.11.0
MAVEN_ARTIFACT_NAME: test-component
GIT_CLEAN_FLAGS: -ffd
PACKAGE_REGISTRY_URL: "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/${MAVEN_ARTIFACT_NAME}"
cache:
key: primary
paths:
- .m2/repository
stages:
- package
- release
package:
stage: package
image: maven:latest
script:
- mvn ${MAVEN_CLI_OPTS} clean package
artifacts:
paths:
- target/*.jar
- target/surefire-reports
only:
- tags
- merge_requests
- branches
except:
- main
release:
stage: release
image: alpine:latest
cache: []
variables:
GIT_STRATEGY: none
dependencies:
- package
script:
- |
apk add curl gitlab-release-cli
find .
JAR_NAME=`basename target/${MAVEN_ARTIFACT_NAME}-${CI_COMMIT_TAG}.jar`
'curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" --upload-file target/${JAR_NAME} ${PACKAGE_REGISTRY_URL}/${CI_COMMIT_TAG}/${JAR_NAME}'
release-cli create --name "Release $CI_COMMIT_TAG" --description "$TAG_MESSAGE" --tag-name ${CI_COMMIT_TAG} --assets-link "{\"name\":\"jar\",\"url\":\"${PACKAGE_REGISTRY_URL}/${CI_COMMIT_TAG}/${JAR_NAME}\"}"
only:
- tags
See the GitLab docs on GIT_STRATEGY:
A Git strategy of none also re-uses the local working copy, but skips all Git operations normally done by GitLab. GitLab Runner pre-clone scripts are also skipped, if present. This strategy could mean you need to add fetch and checkout commands to your .gitlab-ci.yml script.
It can be used for jobs that operate exclusively on artifacts, like a deployment job. Git repository data may be present, but it’s likely out of date. You should only rely on files brought into the local working copy from cache or artifacts.
So GitLab documentation is pretty clear that you should always expect the git repository to be present. When you want to work exclusively with artifacts, I you can create a new temporary directory and reference the path to the artifacts explicitly rather than relying on a totally clean working directory.
I hope someone can help me with a simple setup of maven CI scripts for GitLab.
I tried to search stackoverflow and google, which results in several questions and answers, but either they seem to be completely different or not that I understand them.
I have a simple setup of two projects. project B depends on project A (= pom packaging).
I have in the runner configuration /etc/gitlab-runner/config.toml the line with the volumes added
[[runners]]
...
[runners.docker]
...
volumes = ["/cache", "/.m2"]
...
my .gitlab-ci.yml for both projects look like this
image: maven:3.6.1-jdk-12
cache:
paths:
- /.m2/repository
- target/
variables:
MAVEN_OPTS: "-Dmaven.repo.local=/.m2/repository"
maven_job:
script:
- mvn clean install
with this - the first project builds correctly and I can see that the caching is working, as it does not download all maven related plugins for building the project, when executed again and again.
It also states
[INFO] Installing /builds/end2end/projectA/pom.xml to /.m2/repository/de/end2end/projectA/0.4.4-SNAPSHOT/projectA-0.4.4-SNAPSHOT.pom
It reports though at the end
WARNING: /.m2/repository: not supported: outside build directory
WARNING: /.m2/repository/classworlds: not supported: outside build directory
WARNING: /.m2/repository/classworlds/classworlds: not supported: outside build directory
WARNING: /.m2/repository/classworlds/classworlds/1.1-alpha-2: not supported: outside build directory
WARNING: /.m2/repository/classworlds/classworlds/1.1-alpha-2/_remote.repositories: not supported: outside build directory
[...]
When executing projectB, the job fails with the info, that it cannot find projectA.
So - what is wrong with the configuration of the runner / .gitlab-ci.yml files ?
I tried
cache:
paths:
- .m2/repository
which removes the warnings, but then the projectA gets in its local .m2 installed
[INFO] Installing /builds/end2end/projectA/pom.xml to /builds/end2end/projectAt/.m2/repository/de/end2end/projectA/0.4.4-SNAPSHOT/projectA-0.4.4-SNAPSHOT.pom
and projectB fails with the same error as above.
In fact, as described in gitlab doc, you use the dynamic storage so the volume is shared between subsequent runs of the same concurrent job for one project. I you want to share data between projects you must use the host-bound storage.
For the warning, the cache is only for working directory, so absolute path like /.m2/repository is not supported. In your case, you don't have to use cache for maven repository because you use a volume.
Background
Related question: Google Container Builder: How to install govendor dependencies during build step?
I am trying to use Google Cloud Container Builder to automate the building of my Docker containers using Build Triggers.
My code is in Go, and I have a vendor folder (checked in to Git) in my project root which contains all of my Go dependencies.
My project has four binaries that need to be Dockerized, structured as follows:
vendor/
...
program1/
program1.go
main/
main.go
Dockerfile
program2/
program2.go
main/
main.go
Dockerfile
...
Each program's Dockerfile is simple:
FROM alpine
ADD main main
ENTRYPOINT ["/main"]
I have a Build Trigger set up to track my master branch. The trigger runs the following build request (cloudbuild.yaml), which makes use of the open sourced Docker build step:
steps:
- name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'gcr.io/$PROJECT_ID/program1:0.1.15-$SHORT_SHA', '.']
dir: 'program1/main'
... (repeated for each program)
(images, tags omitted)
To summarize, my current build process is as follows:
Edit code.
Build each Go executable using go build. The executable is called main, and is saved in the programX/main/ directory, alongside main.go.
Commit and push code (since the main executables are tracked by Git) to my master branch.
Build Trigger makes four Docker images using the main file built in step 1.
Goal
I would like to eliminate Step 1 from my build process, so that I no longer need to compile my executables locally, and do not need to track my main executables in Git.
In sum, here is my ideal process:
Edit code, commit, push to remote.
Build Trigger compiles all four programs, builds all four images.
Relax :)
Attempted solution
I used the open source Go build step, as follows:
cloudbuild.yaml: (updated)
steps:
- id: 'build-program1'
name: 'gcr.io/cloud-builders/go'
args: ['build', '-a', '-installsuffix', 'cgo', '-ldflags', '''-w''', '-o', 'main', './main.go']
env: ['PROJECT_ROOT=/workspace', 'CGO_ENABLED=0', 'GOOS=linux']
dir: 'program1/main'
- name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'gcr.io/$PROJECT_ID/program1:0.1.15-$SHORT_SHA', '.']
dir: 'program1/main'
waitFor: ['build-program1']
... (repeated for each program)
(images, tags omitted)
After trying various combinations of setting PROJECT_ROOT and GOPATH in the env field of the build-programX, I kept getting the same error for every single package used in my project (file path varies):
cannot find package "github.com/acoshift/go-firebase-admin" in any of
Step #0 - "build-program1": /usr/local/go/src/github.com/acoshift/go-firebase-admin (from $GOROOT)
Step #0 - "build-program1": /workspace/auth/main/gopath/src/github.com/acoshift/go-firebase-admin (from $GOPATH)
It's not even looking for a vendor directory?
What next?
My guess is that one of the following is true:
I am not specifying the GOPATH/PROJECT_ROOT correctly in the build request file. But if so, what is the correct setting?
My project is not structured correctly.
What I am trying to do is impossible :(*
I need to make a custom build step, somehow.
The version of Go used is old - but how can I check this?
I can find no examples online of what I am trying to achieve, and I find GCP's documentation on this subject quite lacking.
Any help would be greatly appreciated!
The issue is with #1: PROJECT_ROOT refers to the desired import path of your binaries. For example, if in program1/main/main.go you import "github.com/foo/bar/program1" to get the package defined in program1/program1.go, you'd set PROJECT_ROOT=github.com/foo/bar.
Fixed the problem (but not exactly sure how...), thanks to these changes:
Set PROJECT_ROOT to my_root, such that the code I want to compile sits at my_root/program1/main/main.go (thanks to John Asmuth for his answer: https://stackoverflow.com/a/46526875/6905609)
Remove the dirs field for the Go build step
Set the -o flag to ./program1/main/main, and the final build arg to ./program1/main/main.go
Previously I was cding into the program1/main directory during the build step, and for some reason go build was looking for packages within my_root/program1/main instead of my_root. Weird!
We have currently three separated appveyor projects, one for each branch in our repository.
Our problem is follwing:
Appveyor ignores my filter on github branches. Everytime we make a commit to master, stage or dev it builds on all three projects instead of the single one we did make a commit to.
Each branch has a unique appveyor.yml file looking like this:
This is the appveyor.yml for dev
version: 0.0.{build}
branches:
only:
- dev
image: Visual Studio 2017
configuration: dev
before_build:
- nuget restore
build:
project: Core.Api.sln
publish_wap: true
verbosity: minimal
build_script:
- ps: .\build.ps1
after_build:
- cmd: dotnet publish src\Core.Api --output %appveyor_build_folder%\dist
test: off
artifacts:
- path: dist
name: dist.web
deploy:
...
When we make a commit, it builds on all projects. Any idea??
This happens because each project has Webhook configured on GitHub and each time someone makes a commit, each project build is triggered by webhook. Then, regardless of what branch is configured for project (that is only default branch for manual/API builds), AppVeyor reads appveyor.yml from the branch where commit was done.
Solution is to use either alternative YAML file names or alternative YAML file location.
With alternative YAML file names you can have something like appveyor-dev.yml, appveyor-stage.yml files and set specific AppVeyor project to use specific file. With alternative YAML file location is it basically the same, but in other location than repo. I personally like alternative YAML file location more because of less duplication and potential merging issues.
In both cases when webhook in say branch dev come to stage project, it still will read appveyor-dev.yml and do the right filtering.
I am trying to use caching in gitlab runner, which builds a Maven Java project. Currently Gitlab runner only allow caching specific paths defined in gitlab yaml file in the cache: clause. When maven builds projects, it generate everything inside target/ folder, which are untracked files in git. So I can simply use untracked: true option to cache everything under target/ folder. The purpose of caching is to skip compiling the files, which have already been compiled by maven under the target/ folder.
However this cache amounts to about 6GB, which is completely unreasonable for its size and time required to create and restore such a giant cache. It caches all jar and war artifacts built during compiling multi-module maven project. However, maven only needs .class files to check changes for re-compilation
So if their was some way using which I can cache only *.class files, and make them available in subsequent builds, then maven could check the .class files and skip re-compiling unchanged files and cache size would also be pretty small. Currently gitlab-runner only allow specifying absolute paths for caching. It does not support regex patterns for paths such as \.class$ (which would have been very useful).
Is there any way I could cache only specific file types using gitlab runner yaml settings?
So based on cascaval's comment, I was able to figure out a solution.
At the end of maven build I ran a command to clean all build artifacts created by maven, which are not used for checking stale status of .java resources. Here is what I wrote -
cd ./projects/directory
find . | grep --perl-regexp --regexp='\/target\/(?!classes|maven)' | xargs rm --recursive --force
This saves all .class files in target/classes folder, including folder structure and also files in maven-status folder, which are probably used by maven to check file status for recompilation.