We have a complex build system, with a many to many relationship between our libraries and our applications. We put each library and application in it's own repository, and use the output of the library builds in our application builds.
On our old Jenkins server, we simply set up a custom workspace and checked out the projects into standardized relative paths so they could find each other. Post build steps assured that only successful builds copied to the central bin folder at the expected relative path.
On our Bamboo server, our repository was fetched to a Checkout Directory at the expected relative path, and we could fetch artifacts from other builds and put them in the central bin folder at the expected relative path.
Now I'm trying to set up some Bitbucket Pipelines builds, and I can't see an obvious way to do a similar thing. The working folder is set automatically by pipelines, I can't push that repository into a subfolder that is relative to other build outputs. I can create artifacts, but I can't seem to import them into other pipelines. I can create caches, but again I can't seem to import them into other pipelines.
Library bitbucket-pipelines.yml
image: mcr.microsoft.com/dotnet/sdk:5.0
pipelines:
branches:
master:
eCRF2:
- step:
name: Build and Test
caches:
- dotnetcore
- platform2
script:
- dotnet restore ./NET5/Platform2.sln
- dotnet build ./NET5/Platform2.sln --no-restore --configuration Release
artifacts:
- NET5/Platform2/bin/**
definitions:
caches:
platform2: NET5/Platform2/bin
App bitbucket-pipelines.yml
image: mcr.microsoft.com/dotnet/sdk:5.0
pipelines:
default:
- step:
name: Build and Test
caches:
- dotnetcore
- platform2
script:
- export PROJECT_NAME=./PlatformDataService.sln
- dotnet restore ${PROJECT_NAME}
- dotnet build ${PROJECT_NAME} --no-restore --configuration Release
artifacts:
- PlatformDataService/bin/**
https://support.atlassian.com/bitbucket-cloud/docs/deploy-build-artifacts-to-bitbucket-downloads/ did get me to upload a file to the Downloads section of the repository, but how do I pull it into the other pipeline?
Is there a way to solve this within bitbucket pipelines itself or do I have to get a nuget server that's available outside my VPN?
Related
I have a GitLab pipeline setup that has a package step to do a maven build during the tag event and a release to upload the jar to the GitLab generic package registry using curl and GitLab-release cli.
What I'm expecting to happen is a cache of the .m2 to be loaded into the package step to allow the mvn clean package to do its thing. Then archive the created jar and test results only.
The release step should begin clean with no git clone, no cache and only the jar and test results.
Instead the 'find .' shows the release step contains everything including
Git directory (.git)
Full checked out repository
.m2 cache
target (fully built as the Package step produced)
From the cache documentation (https://docs.gitlab.com/ee/ci/caching/) on GitLab it states
Archive: 'dependencies' keyword to control which job fetches the artifacts
Disable Cache uses the 'cache: []'
Why is GitLab putting so much content into the release job? The release job fails at times because its finding multiple Jar files from previous tags (IE the clean and the archiving are holding past version).
gitlab-ci.yml
variables:
MAVEN_CLI_OPTS: "-s $CI_PROJECT_DIR/.m2/settings.xml"
MAVEN_VERSION_PLUGIN_VERSION: 2.11.0
MAVEN_ARTIFACT_NAME: test-component
GIT_CLEAN_FLAGS: -ffd
PACKAGE_REGISTRY_URL: "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/${MAVEN_ARTIFACT_NAME}"
cache:
key: primary
paths:
- .m2/repository
stages:
- package
- release
package:
stage: package
image: maven:latest
script:
- mvn ${MAVEN_CLI_OPTS} clean package
artifacts:
paths:
- target/*.jar
- target/surefire-reports
only:
- tags
- merge_requests
- branches
except:
- main
release:
stage: release
image: alpine:latest
cache: []
variables:
GIT_STRATEGY: none
dependencies:
- package
script:
- |
apk add curl gitlab-release-cli
find .
JAR_NAME=`basename target/${MAVEN_ARTIFACT_NAME}-${CI_COMMIT_TAG}.jar`
'curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" --upload-file target/${JAR_NAME} ${PACKAGE_REGISTRY_URL}/${CI_COMMIT_TAG}/${JAR_NAME}'
release-cli create --name "Release $CI_COMMIT_TAG" --description "$TAG_MESSAGE" --tag-name ${CI_COMMIT_TAG} --assets-link "{\"name\":\"jar\",\"url\":\"${PACKAGE_REGISTRY_URL}/${CI_COMMIT_TAG}/${JAR_NAME}\"}"
only:
- tags
See the GitLab docs on GIT_STRATEGY:
A Git strategy of none also re-uses the local working copy, but skips all Git operations normally done by GitLab. GitLab Runner pre-clone scripts are also skipped, if present. This strategy could mean you need to add fetch and checkout commands to your .gitlab-ci.yml script.
It can be used for jobs that operate exclusively on artifacts, like a deployment job. Git repository data may be present, but it’s likely out of date. You should only rely on files brought into the local working copy from cache or artifacts.
So GitLab documentation is pretty clear that you should always expect the git repository to be present. When you want to work exclusively with artifacts, I you can create a new temporary directory and reference the path to the artifacts explicitly rather than relying on a totally clean working directory.
My appveyor.yml looks like below:
environment:
matrix:
- solution_name: Src/App1.sln
- solution_name: Src/App2.sln
before_build:
- cmd: dotnet restore %solution_name%
build:
project: '%solution_name%'
publish_nuget: true
after_build:
- dotnet pack Src\App1.csproj
deploy:
- provider: NuGet
api_key:
secure: [my api key]
on:
branch: master
artifact: /.*(\.|\.s)nupkg/
artifacts:
- path: Out\
name: App1
- path: '**\*.nupkg'
It works well with one catch: it seems deploy twice during the build process. Below is part of the AppVeyor log:
Collecting artifacts...
Found artifact 'Out' matching 'Out' path
Found artifact 'Out\App1\Release\App1.6.0.3.nupkg' matching '**\*.nupkg' path
Uploading artifacts...
[1/2] App1.6.0.3.nupkg (164,127 bytes)...100%
[2/2] Out\App1\Release\App1.6.0.3.nupkg (164,127 bytes)...100%
Deploying using NuGet provider
Publishing App1.6.0.3.nupkg to https://www.nuget.org/api/v2/package...OK
Publishing App1.6.0.3.nupkg to https://www.nuget.org/api/v2/package...Skipped (A package with ID 'App1' and version '6.0.3' already exists and cannot be modified.)
No packages were pushed.
So I login NuGet.org and I can see App1.6.0.3 is published successfull. However, in the above log, the same package is published twice, hence the second publish is skipped, and end up with message "No packages we pushed." But actually there was one package pushed.
Maybe it is because I have a matrix configuration under environment section, so AppVeyor runs deploy twice? If so, how can tell AppVeyor to deploy only once?
After some experiments, I found a fix, that is removing publish_nuget: true under build section. This line will cause AppVeyor to create a nuget package, and dotnet pack Src\App1.csproj command under after_build section will create a nuget package, too. So there will be two identical .nupkg files generated in different location, and AppVeyor will find both files and push them to nuget server.
Result:
build:
project: '%solution_name%'
after_build:
- dotnet pack Src\App1.csproj
I am new to gitlab CI and trying to build my maven project using gitlab.
I created my own docker image for builds and I am using my local runner.
My .gitlab-ci.yml looks as below: Edited:
image: registry.gitlab.com/girishenoy2003/docker-java-8-mvn-3.6.3:latest
services:
- docker:dind
stages:
- compile
- test
before_script:
- export MVN_USER_HOME=`pwd`/.m2
cache:
paths:
- $MVN_USER_HOME/.m2/repository/
- target/
maven-compile:
stage: compile
script:
- mvn compile
tags:
- my-local-runner
only:
- master
maven-test:
stage: test
script:
- mvn test
tags:
- my-local-runner
only:
- master
During the compile stage I expect all the dependencies to download and cache it and while running test I expect it to get it from cache as I am using my local runner for both jobs.
How can we avoid download of dependent jars from downloading in different jobs?
P.S.:
I already looked at this question - but didn't help
I know that if I use mvn test-compile it will do the trick but I wanted to make it in different jobs to segregate the stages
Edit:
Some runner logs:
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/aws-learning-path/spring-boot-rest/.git/
Checking out f86f9c63 as master...
Removing "..\\..\\..\\cache\\aws-learning-path\\spring-boot-rest\\default\\cache.zip"
Removing target/
Skipping Git submodules setup
Restoring cache
00:00
Checking cache for default...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache
Executing "step_script" stage of the job script
$ export MVN_USER_HOME=/root
$ mvn test
[INFO] Scanning for projects...
Downloading from central: https://repo.maven.apache.org/maven2/org/springframework/boot/spring-boot-starter-parent/2.2.2.RELEASE/spring-boot-starter-parent-2.2.2.RELEASE.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/springframework/boot/spring-boot-starter-parent/2.2.2.RELEASE/spring-boot-starter-parent-2.2.2.RELEASE.pom (8.1 kB at 3.3 kB/s)
Your issue is related to your /root folder
The gitlab-runner do not cache any paths outside of project directory, you need to configure your CI to download and store the dependencies inside your project path
Currently, I have two main project.
1-) Vue Project which contains (webviews for IOS and Android, websites, and renderer for our Electron ) they are sharing components & API's.
2-) Electron Project which builds desktop app for (windows, darwin, linux)
i would like to automate our building, releasing process. my current setup..
before_script:
- apt-get update
- apt-get install zip unzip
- rm -rf vue-project
- git clone vue-project
- cd vue-project
- git checkout dev
- git pull
- sed -i "/\b\(areaCode\|inline-svg-loader\)\b/d" ./packages/devtool/package.json
- yarn install
- ln -s vue-project/packages/desktop/ web
- npm install
build_darwin:
stage: build
script:
- npm run package -- darwin --deploy
cache:
paths:
- vue-project/node_modules
- node_modules
which basically before bundling electron project it's cloning vue-project install dependencies and bundling electron-renderer then when it's finish. i'm running package.
I would like to separate this two different job from each other. is there anyway i could use artifacts from different project gitlab-CI pipelines ?
any help would be an appreciated.
Gitlab has a API for do a lot of tricks.
curl --header "PRIVATE-TOKEN:YOURPRIVATETOKEN" "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/master/download?job=test"
for download it as a file.
curl --header "PRIVATE-TOKEN:YOURPRIVATETOKEN" -o artifacts.zip "http://gitlab.example.net/api/v4/projects/<projectnumber>/jobs/artifacts/master/download?job=build_desktop
Gitlab can certainly support this. To accomplish this, follow these steps:
ARTIFACT GENERATION
In your Vue Project, modify your job(s) of interest to store artifacts relevant to the Electron project. Each job's artifacts are defined using Gitlab Job Artifacts notation, and are uploaded at job completion to Gitlab, and stored associated to your Project, Branch, and Job.
Note: Branch is often overlooked, and matters when you want to retrieve your artifacts, more on this later.
Illustrating:
Vue Project .gitlab_ci.yml
stages:
- stage1
- ...
vue-job1:
stage: stage1
script:
- echo "vue-job1 artifact1" > ./artifact1
- echo "vue-job1 artifact2" > ./artifact2
artifacts:
when: always
paths:
- ./artifact1
- ./artifact2
expire_in: 90 days
vue-job2:
stage: stage1
script:
# error, would overwrite job1's artifacts since working
# directory is a global space shared by all pipeline jobs
# - echo "vue-job2 artifact1" > ./artifact1
- echo "vue-job2 artifact1" > ./artifact3
artifacts:
when: always
paths:
- ./artifact3
expire_in: 90 days
The artifacts generated above are written to the working directory, which is a clone of your project's repo. So be careful with filename conflicts. To be safe, put your artifacts in a subdirectory (eg: cat "foo" > ./subdir/artifact) and reference them in paths the same way (paths: - ./subdir/artifact). You can use 'ls' in your script to view the working directory.
When your job completes, you can confirm the artifacts stored in Gitlb by using the Gitlab UI. View the job output, and use the Browse button under Job Artifacts on the right panel.
ARTIFACT RETRIEVAL
In your Electron Project, modify your job(s) of interest to retrieve artifacts stored in the Vue Project using the Gitlab Job Artifacts API and curl. In order to access the Vue artifacts, you will need the Vue Project, Branch, and Job that the artifacts were created under.
Project: For Project, use the Project ID displayed in the Gitlab UI Project Details screen.
Branch: Usually master, but depends on the branch your pipeline executes against. Although this is not relevant for your problem, if you are generating and consuming artifacts across executions of the same pipeline, use the Gitlab variable $CI_COMMIT_BRANCH for the branch.
Job: Generally the Job Name that generated the artifacts for your Project. But if you need artifacts produced by a specific Job, then use the Job Number and the corresponding retrieval API.
Illustrating:
Electron Project .gitlab_ci.yml
stages:
- stage1
- ...
electron-job1:
stage: stage1
script:
- curl -o ./artifact1 -H "PRIVATE-TOKEN:$TOKEN" https://gitlab.example.com/api/v4/projects/$VUE_PROJECT_ID/jobs/artifacts/$BRANCH/raw/artifact1?job=vue-job1
- curl -o ./artifact2 -H "PRIVATE-TOKEN:$TOKEN" https://gitlab.example.com/api/v4/projects/$VUE_PROJECT_ID/jobs/artifacts/$BRANCH/raw/artifact2?job=vue-job1
- curl -o ./artifact3 -H "PRIVATE-TOKEN:$TOKEN" https://gitlab.example.com/api/v4/projects/$VUE_PROJECT_ID/jobs/artifacts/$BRANCH/raw/artifact3?job=vue-job2
This script retrieves artifacts individually to the working directory of your Electron Project. There are also options for retrieving all artifacts for your job at once as a zip archive.
MISCELLANEOUS
Although this is not in the problem posed, it is worth noting that you can use artifacts both within the lifespan of a single pipeline execution to pass information between jobs. You can also use this to pass information across pipeline executions within the same project.
With recent versions of gitlab, this can be achieved simply by using either the multi-project pipeline feature (then starting a pipeline in one project triggers a build from the other project): see the documentation
Or you can also use the "needs:project" mechanism which allows one job to download artifacts from other pipelines (see the documentation Use needs:project to download artifacts from up to five jobs in other pipelines.)
We have currently three separated appveyor projects, one for each branch in our repository.
Our problem is follwing:
Appveyor ignores my filter on github branches. Everytime we make a commit to master, stage or dev it builds on all three projects instead of the single one we did make a commit to.
Each branch has a unique appveyor.yml file looking like this:
This is the appveyor.yml for dev
version: 0.0.{build}
branches:
only:
- dev
image: Visual Studio 2017
configuration: dev
before_build:
- nuget restore
build:
project: Core.Api.sln
publish_wap: true
verbosity: minimal
build_script:
- ps: .\build.ps1
after_build:
- cmd: dotnet publish src\Core.Api --output %appveyor_build_folder%\dist
test: off
artifacts:
- path: dist
name: dist.web
deploy:
...
When we make a commit, it builds on all projects. Any idea??
This happens because each project has Webhook configured on GitHub and each time someone makes a commit, each project build is triggered by webhook. Then, regardless of what branch is configured for project (that is only default branch for manual/API builds), AppVeyor reads appveyor.yml from the branch where commit was done.
Solution is to use either alternative YAML file names or alternative YAML file location.
With alternative YAML file names you can have something like appveyor-dev.yml, appveyor-stage.yml files and set specific AppVeyor project to use specific file. With alternative YAML file location is it basically the same, but in other location than repo. I personally like alternative YAML file location more because of less duplication and potential merging issues.
In both cases when webhook in say branch dev come to stage project, it still will read appveyor-dev.yml and do the right filtering.