Artifact dependencies (destination) using Bamboo YAML specs - yaml

I'm trying to set up the Bamboo build plan configuration using Bamboo YAML specs (.yml file below). In the last stage (Create deployment artifacts) I want to use the shared artifacts from the previous stage. By specifying the artifacts of the jobs as "shared: true" I can use them in the second stage. However, they are in the same destination folder. Using the UI this can be easily edited.
Artifact dependencies
But how can I specify the destination folder of the two artifacts in the Bamboo YAML specs, e.g. from "Root of working directory" to "./app" and "./wwwroot", respectively?
---
version: 2
plan:
project-key: COCKPIT
key: BE
name: Cockpit - Continuous Build - Windows
stages:
- Build Stage:
- Build Backend
- Build Frontend
- Build Artifact:
- Create Deployment Artifact
Build Backend:
requirements:
- Visual Studio Build Tools (32-bit)
tasks:
- checkout:
repository: cockpit_backend
path: 'cockpit_backend'
force-clean-build: false
- script:
- dotnet publish .\cockpit_backend\src\Cockpit.WebApi\ --configuration Release
artifacts:
-
name: BackendBuild
location: cockpit_backend/src/Cockpit.WebApi/bin/Release/netcoreapp3.1/publish
pattern: '**/*.*'
required: true
shared: true
Build Frontend:
requirements:
- os_linux
tasks:
- checkout:
repository: 'Cockpit / cockpit_frontend'
path: 'cockpit_frontend'
force-clean-build: false
- script:
- cd cockpit_frontend
- npm install
- script:
- cd cockpit_frontend
- npm run build-prod
docker:
image: node:alpine
artifacts:
-
name: FrontendBuild
location: cockpit_frontend/dist
pattern: '**/*.*'
required: true
shared: true
Create Deployment Artifact:
requirements:
- os_windows
tasks:
- script:
interpreter: powershell
scripts:
- $buildDir = "Cockpit"
- $dest = "Cockpit_${bamboo.buildNumber}.zip"
- Add-Type -assembly "system.io.compression.filesystem"
- '[io.compression.zipfile]::CreateFromDirectory($buildDir, $dest)'
artifacts:
-
name: Completebuild
pattern: 'Cockpit_${bamboo.buildNumber}.zip'
required: true

YAML specs doesn't support artifact dependency management and you need to have script task at "Create Deployment Artifact" job to put them into separate folders from root before compressing

Related

Azure Spring Gradle: files in src/test/resouces not accessible

Im building Spring boot app using gradle. Integration tests (Spock) need to access code/src/resouces/docker-compose.yml file to prepare TestContainers container:
static DockerComposeContainer postgresContainer = new DockerComposeContainer(
ResourceUtils.getFile("classpath:docker-compose.yml"))
The git file structure is:
- code
- src
- main
- test
- resources
- docker-compose.yml
This is working fine on my local machine, but once I run it in Azure pipeline, it gives
No such file or directory: '/__w/1/s/code/build/resources/test/docker-compose.yml'
My pipeline yaml is like bellow. I use Ubuntu container with Java17 as I need to build with 17 but Azure's latest is 11 (maybe this plays any role in the error I get?)
trigger: none
stages:
- stage: Test
displayName: Test Stage
jobs:
- job: Test
pool:
vmImage: 'ubuntu-22.04'
container: gradle:7.6.0-jdk17
variables:
- name: JAVA_HOME_11_X64
value: /opt/java/openjdk
displayName: Test
steps:
- script: java --version
displayName: Java version
- script: |
echo "Build reason is: $(Build.Reason)"
displayName: Build reason
- checkout: self
clean: true
- task: Gradle#2
displayName: 'Gradle Build'
enabled: True
inputs:
javaHomeSelection: 'path'
jdkDirectory: '/opt/java/openjdk'
wrapperScript: code/gradlew
cwd: code
tasks: clean build
publishJUnitResults: true
jdkVersionOption: 1.17
Thanks for help!
I've solved it by "workarround" - I realied that I dont need to use the container with jdk17 that was causing the problem (could not access files on host machine of course)
The true is the Azure silently supports Jkd17 by directive: jdkVersionOption: 1.17
But once someone will need to use the container to build the code and access the repository files that are not on classpath, the problem will raise again
trigger: none
stages:
- stage: Test
displayName: Test Stage
jobs:
- job: Test
pool:
vmImage: 'ubuntu-22.04'
displayName: Test
steps:
- script: java --version
displayName: Java version
- script: |
echo "Build reason is: $(Build.Reason)"
displayName: Build reason
- checkout: self
clean: true
- task: Gradle#2
displayName: 'Gradle Build'
enabled: True
inputs:
wrapperScript: server/code/gradlew
cwd: server/code
tasks: test
publishJUnitResults: true
jdkVersionOption: 1.17
Please follow Azure pipeline issue for more details

Gradle build failed in Tekton CI/CD

I have below pipeline task for Gradle build, which clones the project from bitbucket repo and try to build the application.
tasks:
- name: clone-repository
taskRef:
name: git-clone
workspaces:
- name: output
workspace: shared-workspace
params:
- name: url
value: "$(params.repo-url)"
- name: deleteExisting
value: "true"
- name: build
taskRef:
name: gradle
runAfter:
- "clone-repository"
params:
- name: TASKS
value: build -x test
- name: GRADLE_IMAGE
value: docker.io/library/gradle:jdk17-alpine#sha256:dd16ae381eed88d2b33f977b504fb37456e553a1b9c62100b8811e4d8dec99ff
workspaces:
- name: source
workspace: shared-workspace
I have the below project structure
The settings.gradle contain the below projects
rootProject.name = 'discount'
include 'core'
include 'infrastructure'
include 'shared'
include 'discount-api'
when running the pipeline with below code
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
name: run-pipeline
namespace: tekton-pipelines
spec:
serviceAccountName: git-service-account
pipelineRef:
name: git-clone-pipeline
workspaces:
- name: shared-workspace
persistentVolumeClaim:
claimName: fetebird-discount-pvc
params:
- name: repo-url
value: git#bitbucket.org:anandjaisy/discount.git
Getting an exception as
FAILURE: Build failed with an exception.
* What went wrong:
Task 'build -x test' not found in root project 'discount'.
I have used the task from the Tekton catalog https://github.com/tektoncd/catalog/blob/main/task/gradle/0.1/gradle.yaml
If I pass the PROJECT_DIR value as ./discount-api to the Gradle task. I get an exception as settings.gradle not found. which is correct because that project has no setting.gradle file.
The main project is discount-api. I need to build the application. Quite not sure what is wrong. On the local env if I do ./gradlew build in the root directory the application successfully builds.
The error message tells about Task 'build -x test' not found in root project 'discount'
Checking that Task, in tekton catalog, we can read:
....
- name: TASKS
description: 'The gradle tasks to run (default: build)'
type: string
default: build
steps:
- name: gradle-tasks
image: $(params.GRADLE_IMAGE)
workingDir: $(workspaces.source.path)/$(params.PROJECT_DIR)
command:
- gradle
args:
- $(params.TASKS)
Now, in your Pipeline, you set that TASKS param to build -x test. This is your issue.
As you can read above, that TASKS param is a string. While you want to use an array.
You should be able to change the param definition, such as:
....
- name: TASKS
description: 'The gradle tasks to run (default: build)'
type: array
default:
- build
steps:
- name: gradle-tasks
image: $(params.GRADLE_IMAGE)
workingDir: $(workspaces.source.path)/$(params.PROJECT_DIR)
command:
- gradle
args: [ "$(params.TASKS)" ]
This would ensure "build", "-x" and "test" are sent to gradle as separate strings. While your current attempt would be equivalent to running gradle "build -x test", resulting in your error.

Can't checkout the same repo multiple times in a pipeline

I have self-hosted agents on multiple environments that I am trying to run the same build/deploy processes on. I would like to be able to deploy the same code from a single repo to multiple systems concurrently. Thus, I have created an "overhead" pipeline, and several "processes" pipeline templates. Everything seems to be going very well, except for when I try to perform checkouts of the same repo twice in the same pipeline execution. I get the following error:
An error occurred while loading the YAML build pipeline. An item with the same key has already been added.
I would really like to be able to just click ONE button to trigger a main pipeline that calls all the templates requires and gives the parameters needed to get all my jobs done at once. I could of course define this "overhead" pipeline and then queue up as many instances as I need of it per systems that I need to deploy to, but I'm lazy, hence why I'm using pipelines!
As soon as I remove the checkout from Common.yml, the validation succeeds without any issues. If I keep the checkout in there but only call the Common.yml once for the entire Overhead pipeline, then it succeeds without any issues as well. But the problem is: I need to pull the contents of the repo to EACH of my agents that are running on completely separate environments that are in no way ever able to talk to each other (can't pull the information to one agent and have it do some sort of a "copy" to all the other agent locations.....).
Any assistance is very much welcomed, thank you!
The following is my "overhead" pipeline:
# azure-pipelines.yml
trigger:
none
parameters:
- name: vLAN
type: string
default: 851
values:
- 851
- 1105
stages:
- stage: vLAN851
condition: eq('${{ parameters.vLAN }}', '851')
pool:
name: xxxxx
demands:
- vLAN -equals 851
jobs:
- job: Common_851
steps:
- template: Procedures/Common.yml
- job: Export_851
dependsOn: Common_851
steps:
- template: Procedures/Export.yml
parameters:
Server: ABTS-01
- stage: vLAN1105
condition: eq('${{ parameters.vLAN }}', '1105')
pool:
name: xxxxx
demands:
- vLAN -equals 1105
jobs:
- job: Common_1105
steps:
- template: Procedures/Common.yml
- job: Export_1105
dependsOn: Common_1105
steps:
- template: Procedures/Export.yml
parameters:
Server: OTS-01
And here is the "Procedures/Common.yml":
steps:
- checkout: git://xxxxx/yyyyy#$(Build.SourceBranchName)
clean: true
enabled: true
timeoutInMinutes: 1
- task: UsePythonVersion#0
enabled: true
timeoutInMinutes: 1
displayName: Select correct version of Python
inputs:
versionSpec: '3.8'
addToPath: true
architecture: 'x64'
- task: CmdLine#2
enabled: true
timeoutInMinutes: 5
displayName: Ensure Python Requirements Installed
inputs:
script: |
python -m pip install GitPython
And here is the "Procedures/Export.yml":
parameters:
- name: Server
type: string
steps:
- task: PythonScript#0
enabled: true
timeoutInMinutes: 3
displayName: xxxxx
inputs:
arguments: --name "xxxxx" --mode True --Server ${{ parameters.Server }}
scriptSource: 'filePath'
scriptPath: 'xxxxx/main.py'
I managed to make checkout work with variable branch names by using template expression variables ${{ ... }} instead of macro syntax $(...) variables.
The difference is that, template expressions are processed at compile time while macros are processed at runtime.
So in my case I have something like:
- checkout: git://xxx/yyy#${{ variables.BRANCH_NAME }}
For more information about variables syntax :
Understand variable syntax
I couldn't get it to work with expressions but I was able to get it to work using repository resources following the documentation at: https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/multi-repo-checkout?view=azure-devops
resources:
repositories:
- repository: MyGitHubRepo # The name used to reference this repository in the checkout step
type: git
name: MyAzureProjectName/MyGitRepo
ref: $(Build.SourceBranch)
trigger:
- main
pool:
vmImage: 'ubuntu-latest'
#some job
steps:
- checkout: MyGitHubRepo
#some other job
steps:
- checkout: MyGitHubRepo
- script: dir $(Build.SourcesDirectory)

GITLAB CI/CD Improving Pipeline Speed with cache or Artifacts?

I really don't understand how cache works and how i can really speed up my CI/CD. I would like to just make it run in less than 2 minutes but all i got is a 30m/40m/1h pipeline.
I have already tried a lot of caching techniques (Global cache, per job cache)
I am not going to write all of them here (i am working on this 3 days by now so i tried a lot of stuff and read a lot of stuff too..).
The project is a .NET MVC Project with some static bundling through NPM.
The CI/CD is running on a Windows Server 2008 R2.
No docker images. Just gitlab-runner inside the C:\gitlab_runner folder and all the tools are natively installed on the machine tagged as deploy-vmdev
(The project is called Iom.Site and it is saved inside the Iom.Site folder)
The best i was able to do was this (but it runs on average of 30m and sometimes it breaks up because depedencies caches, node_modules and packages, aren't present.):
stages:
- install_dependencies
- build_app
- deploy
#Execute dependencies only on package-lock.json e packages.config changes
install_dependencies:
stage: install_dependencies
only:
changes:
- Iom.Site/package-lock.json
- Iom.Site/packages.config
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- Iom.Site/node_modules
- packages/
tags:
- deploy-vmdev
script:
- if(!((nuget sources) -like "*Promoscience*").Length -gt 0) { nuget sources add -Name Promoscience -Source http://nupromo.promoscience.com/nuget };
- nuget restore
- cd Iom.Site
- npm install
#Bundling through gulp npm packages static files (SCSS, JS,..)
#Copying .\site\content inside .\dist folder at root for packaging
build_static_files:
stage: build_app
only:
changes:
- Iom.Site\scripts\bundle\**\*
- Iom.Site\scripts\react\**\*
- Iom.Site\scss\**\*
- ".gitlab-ci.yml"
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- Iom.Site/node_modules
policy: pull
tags:
- deploy-vmdev
script:
- gulp build
- cd ..
- Copy-Item Iom.Site\content .\dist\
artifacts:
paths:
- .\dist
when: on_success
expire_in: 30 days
#Building .NET App through MSBUILD. Msbuild profile saves the output in .\dist folder
#for packaging the whole app
build_app:
stage: build_app
only:
changes:
- "**/*.cs"
- "**/*.cshtml"
- "**/*.config"
- ".gitlab-ci.yml"
tags:
- deploy-vmdev
script:
- msbuild Iom.Site\Iom.Site.csproj /p:Configuration=test /p:PublishProfile=FolderProfile /p:transformConfigFiles=true /p:deployOnBuild=true
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- packages/
policy: pull
artifacts:
paths:
- .\dist
when: on_success
expire_in: 30 days
#Deploying: Downloading artifacts and merging them.
#Copying the content of dist folder inside the IIS Site
deploy_test:
stage: deploy
dependencies:
- build_static_files
- build_app
tags:
- deploy-vmdev
only:
- master
- dev
cache: {}
variables:
SiteAppPath: "E:\\websites\\CnrIom2020"
script:
- echo $env:SiteAppPath
- dir .\dist
- Copy-Item .\dist\* $env:SiteAppPath -Recurse -Force
UPDATE 1:
In the end i found this on gitlab ci wiki:
Caching is an optimization, but it isn’t guaranteed to always work. You need to be prepared to regenerate any cached files in each job that needs them.
And changed my pipeline in this way:
stages:
- build_app
- package
- deploy
build_static_files:
stage: build_app
before_script:
- cd Iom.Site
- if(-Not(Test-Path node_modules)){npm install} #Installiamo le dipendenze di npm solo se non presenti
only:
changes:
- Iom.Site\scripts\bundle\**\*
- Iom.Site\scripts\react\**\*
- Iom.Site\scss\**\*
- ".gitlab-ci.yml"
tags:
- deploy-vmdev
cache:
key: static_${CI_COMMIT_REF_SLUG}
paths:
- Iom.Site/node_modules
policy: pull-push
script:
- gulp build
- cd ..
- Copy-Item Iom.Site\content .\dist\content -Recurse -Force #Copiamo i file statici in dist
artifacts:
paths:
- .\dist
when: on_success
expire_in: 2 hours
build_app:
stage: build_app
before_script: #Installiamo le dipendenze di Nuget solo se già non presenti
- if(-Not(Test-Path packages)){ nuget restore }
only:
changes:
- "**/*.cs"
- "**/*.cshtml"
- "**/*.config"
- ".gitlab-ci.yml"
tags:
- deploy-vmdev
cache:
key: app_${CI_COMMIT_REF_SLUG}
paths:
- packages
policy: pull-push
script:
- msbuild Iom.Site\Iom.Site.csproj /p:Configuration=test /p:PublishProfile=FolderProfile /p:transformConfigFiles=true /p:deployOnBuild=true
artifacts:
paths:
- .\dist
when: on_success
expire_in: 2 hours
package_app:
stage: package
dependencies:
- build_static_files
- build_app
tags:
- deploy-vmdev
only:
- master
- dev
- release
cache: { }
script:
- dir .\dist
artifacts:
paths:
- .\dist
expire_in: never
deploy_test:
stage: deploy
dependencies:
- package_app
tags:
- deploy-vmdev
only: #TODO: When: manual
- master
- dev
- release
cache: {}
variables:
SiteAppPath: "E:\\websites\\CnrIom2020"
script:
- echo $env:SiteAppPath
- dir .\dist
- Copy-Item .\dist\* $env:SiteAppPath -Recurse -Force
# deploy_release: To define

CircleCI: Create Workflow with separate jobs. One build and different Deploys per environment

everyone. I need some help with some issues that I am facing configuring circleCi for my Angular project.
The config.yml that I am using for a build and deploy process is detailed below. Currently I have decided to do separate jobs for each environment and each one includes the building and deploy. The problem with this approach is that I am repeating myself and I can't find the correct way to deploy an artifact builded in the previous job for the same workflow.
version: 2
jobs:
build:
docker:
- image: circleci/node:8-browsers
steps:
- checkout
- restore_cache:
key: dependency-cache-{{ checksum "package.json" }}
- run:
name: Install dependencies
command: npm install
- save_cache:
key: dependency-cache-{{ checksum "package.json" }}
paths:
- .node_modules
- run:
name: Build Application (Production mode - aot enabled)
command: npm run build:prod
- store_artifacts:
path: dist
destination: dist
deploy_prod:
docker:
- image: circleci/node:8-browsers
environment:
- FIREBASE_TOKEN: "1/AFF2414141ASdASDAKDA4141421sxscq"
steps:
- checkout
- restore_cache:
key: dependency-cache-{{ checksum "package.json" }}
- run:
name: Install dependencies
command: npm install
- save_cache:
key: dependency-cache-{{ checksum "package.json" }}
paths:
- .node_modules
- run:
name: Build Application (Production mode - aot enabled)
command: npm run build:prod
- store_artifacts:
path: dist
destination: dist
- run:
command: ./node_modules/.bin/firebase use default
- deploy:
command: ./node_modules/.bin/firebase deploy --token=$FIREBASE_TOKEN
deploy_qa:
docker:
- image: circleci/node:8-browsers
environment:
- FIREBASE_TOKEN: "1/AFF2414141ASdASDAKDA4141421sxscq"
steps:
- checkout
- restore_cache:
key: dependency-cache-{{ checksum "package.json" }}
- run:
name: Install dependencies
command: npm install
- save_cache:
key: dependency-cache-{{ checksum "package.json" }}
paths:
- .node_modules
- run:
name: Build Application (Production mode - aot enabled)
command: npm run build:prod
- store_artifacts:
path: dist
destination: dist
- run:
command: ./node_modules/.bin/firebase use qa
- deploy:
command: ./node_modules/.bin/firebase deploy --token=$FIREBASE_TOKEN
workflows:
version: 2
build-and-deploy:
jobs:
- build:
filters:
branches:
only:
- master
ignore:
- /feat-.*/
- deploy_prod:
filters:
branches:
ignore:
- /.*/
tags:
only:
- /v[0-9]+(\.[0-9]+){2}/
- deploy_qa:
filters:
branches:
ignore:
- /.*/
tags:
only:
- /v[0-9]+(\.[0-9]+){2}-BETA-([0-9]*)/
I understand that each job is using a different docker image, so this prevents me work on the same workspace.
Q: How can I use the same docker image for different jobs in the same workflow?
I included the store_artifacts thinking it could help me, but what I read about it is that it only works for using through the dashboard or the API.
Q: Am I able to recover an artifact on a job that requires a different job that stored the artifact?
I know that I am repeating myself, my goal is to have a build job required for a deploy job per environment depending on the tags' name. So my deploy_{env} jobs are mainly the firebase commands.
workflows:
version: 2
build-and-deploy:
jobs:
- build:
filters:
branches:
only:
- master
ignore:
- /feat-.*/
tags:
only:
- /v[0-9]+(\.[0-9]+){2}/
- /v[0-9]+(\.[0-9]+){2}-BETA-([0-9]*)/
- deploy_prod:
requires:
- build
filters:
branches:
ignore:
- /.*/
tags:
only:
- /v[0-9]+(\.[0-9]+){2}/
- deploy_qa:
requires:
- build
filters:
branches:
ignore:
- /.*/
tags:
only:
- /v[0-9]+(\.[0-9]+){2}-BETA-([0-9]*)/
Q: What are the recommended steps (best practices) for this solution?
Q: How can I use the same docker image for different jobs in the same workflow?
There might be two methods of going about this:
1.) Docker Layer Caching: https://circleci.com/docs/2.0/docker-layer-caching/
2.) Caching the .tar file: https://circleci.com/blog/how-to-build-a-docker-image-on-circleci-2-0/
Q: Am I able to recover an artifact on a job that requires a different
job that stored the artifact?
The persist_to_workspace and attach_workspace keys should be helpful here: https://circleci.com/docs/2.0/workflows/#using-workspaces-to-share-data-among-jobs
Q: What are the recommended steps (best practices) for this solution?
Not sure here! Whatever works the fastest and cleanest for you. :)

Resources