Build docker image including version with bitbucket pipelines - continuous-integration

I'm pretty new to Bitbucket Pipelines and I encountered a problem. I'm creating a pipeline to deploy a new version of our Spring Boot application (which runs in a Kubernetes cluster) to our test environment. The problem I encountered is the versioning of our docker build. Our versioning is set up as the following:
alpha_0.1
alpha_0.2
beta_1.0
gamma_1.0
gamma_1.1
So every minor update/bugfix increases the build number by 0.1, and a major update increases the version by 1.0 + every major update gets a new version name.
Currently I have the next setup:
image: java:8
options:
docker: true
branches:
master:
- step:
caches:
- gradle
script:
- ./gradlew test
- ./gradlew build
- docker build -t <application_name>/<version_name>_<version_number>
What is the best way to include the version_name and the version_number in the bitbucket pipeline? Until now we runned ruby script which allowed user input for version numbering, but bitbucket pipelines are not interactive.

Assuming that alpha_0.1 etc. are tags and that the pipeline runs if a commit is tagged, you can get the tag for the current commit like this:
TAG=$(git tag --contains $BITBUCKET_COMMIT)
You can then use your favorite language or command-line tool to create the <version_name> and <version_number> from the tag you got. It may make sense to export the tag as a shell variable to be able to use it in a script.

This is one of the shippable.yml files I have, feel free to adapt it to Atlassian's pipelines.yml and Gradle:
language: java
jdk:
- oraclejdk8
branches:
only:
- master
...
build:
ci:
# Generates build number
- BUILD_NUMBER=`git log --oneline | wc -l`
- echo "Build number':' ${BUILD_NUMBER}"
# Sets version
- mvn versions:set -DnewVersion=1.0.${BUILD_NUMBER}
# Builds and pushes to Docker Hub
- mvn package
- docker login -u ${DOCKERHUB_USERNAME} -p ${DOCKERHUB_PASSWD} --email ${DOCKERHUB_EMAIL} https://index.docker.io/v1/
- mvn -X docker:build -Dpush.image=true
My projects version (in pom.xml) are set to 0-SNAPSHOPT
This also uses Spotify's Maven plugin to build the Docker image instead of docker build -t ...

Related

Using different Docker images during a single Gitlab-CI job

Hello there i have a particular question regarding using different images under a single gitlab-ci pipeline job. Initially i am using a maven 3.6.1-jdk-8 because the project runs per those requirements. What i want to do is to use another image maven 3.6.3-jdk-11 before running the script so the sonarqube scanner be able to run properly and does not fails. Both images are required for the job.
sonarqube:
image: maven:3.6.1-jdk-8
variables:
SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar" # Defines the location of the analysis task cache
GIT_DEPTH: "0" # Tells git to fetch all the branches of the project, required by the analysis task
cache:
key: "${CI_JOB_NAME}"
paths:
- .sonar/cache
script:
- mvn verify sonar:sonar -Dsonar.qualitygate.wait=true
allow_failure: true
only:
- master

Gitlab working directory not clean when using cache with CLONE_STRATEGY: none

I have a GitLab pipeline setup that has a package step to do a maven build during the tag event and a release to upload the jar to the GitLab generic package registry using curl and GitLab-release cli.
What I'm expecting to happen is a cache of the .m2 to be loaded into the package step to allow the mvn clean package to do its thing. Then archive the created jar and test results only.
The release step should begin clean with no git clone, no cache and only the jar and test results.
Instead the 'find .' shows the release step contains everything including
Git directory (.git)
Full checked out repository
.m2 cache
target (fully built as the Package step produced)
From the cache documentation (https://docs.gitlab.com/ee/ci/caching/) on GitLab it states
Archive: 'dependencies' keyword to control which job fetches the artifacts
Disable Cache uses the 'cache: []'
Why is GitLab putting so much content into the release job? The release job fails at times because its finding multiple Jar files from previous tags (IE the clean and the archiving are holding past version).
gitlab-ci.yml
variables:
MAVEN_CLI_OPTS: "-s $CI_PROJECT_DIR/.m2/settings.xml"
MAVEN_VERSION_PLUGIN_VERSION: 2.11.0
MAVEN_ARTIFACT_NAME: test-component
GIT_CLEAN_FLAGS: -ffd
PACKAGE_REGISTRY_URL: "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/${MAVEN_ARTIFACT_NAME}"
cache:
key: primary
paths:
- .m2/repository
stages:
- package
- release
package:
stage: package
image: maven:latest
script:
- mvn ${MAVEN_CLI_OPTS} clean package
artifacts:
paths:
- target/*.jar
- target/surefire-reports
only:
- tags
- merge_requests
- branches
except:
- main
release:
stage: release
image: alpine:latest
cache: []
variables:
GIT_STRATEGY: none
dependencies:
- package
script:
- |
apk add curl gitlab-release-cli
find .
JAR_NAME=`basename target/${MAVEN_ARTIFACT_NAME}-${CI_COMMIT_TAG}.jar`
'curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" --upload-file target/${JAR_NAME} ${PACKAGE_REGISTRY_URL}/${CI_COMMIT_TAG}/${JAR_NAME}'
release-cli create --name "Release $CI_COMMIT_TAG" --description "$TAG_MESSAGE" --tag-name ${CI_COMMIT_TAG} --assets-link "{\"name\":\"jar\",\"url\":\"${PACKAGE_REGISTRY_URL}/${CI_COMMIT_TAG}/${JAR_NAME}\"}"
only:
- tags
See the GitLab docs on GIT_STRATEGY:
A Git strategy of none also re-uses the local working copy, but skips all Git operations normally done by GitLab. GitLab Runner pre-clone scripts are also skipped, if present. This strategy could mean you need to add fetch and checkout commands to your .gitlab-ci.yml script.
It can be used for jobs that operate exclusively on artifacts, like a deployment job. Git repository data may be present, but it’s likely out of date. You should only rely on files brought into the local working copy from cache or artifacts.
So GitLab documentation is pretty clear that you should always expect the git repository to be present. When you want to work exclusively with artifacts, I you can create a new temporary directory and reference the path to the artifacts explicitly rather than relying on a totally clean working directory.

How to install maven for a docker shell gitlab runner

(Please note i'm a total beginner in all Docker and CI in general)
I am trying to set up a simple CI environment with GitLab. I'm using a local GitLab runner which is configured to build and test in Docker. In registering the runner I chose the 'shell' option. This installation succeeded.
After pushing my code I got an email saying 'build failed'. In the build log I found the following:
on desktop-docker-runner cDD_yf4V
Using Shell executor...
Running on b567d1ba4654...
DEPRECATION: this GitLab server doesn't support refspecs, gitlab-runner 12.0 will no longer work with this version of GitLab
Fetching changes...
Checking out fd20ca86 as dev...
Skipping object checkout, Git LFS is not installed.
Skipping Git submodules setup
Checking cache for default...
Runtime platform arch=amd64 os=linux pid=722 revision=1f513601 version=11.10.1
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache
$ mvn $MAVEN_CLI_OPTS test-compile
bash: line 74: mvn: command not found
ERROR: Job failed: exit status 1
How can I add Maven to resolve this?
(EDIT: added gitlab-ci.yml file)
# Build JAVA applications using Apache Maven (http://maven.apache.org)
# For docker image tags see https://hub.docker.com/_/maven/
#
# For general lifecycle information see https://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html
#
# This template will build and test your projects as well as create the documentation.
#
# * Caches downloaded dependencies and plugins between invocation.
# * Verify but don't deploy merge requests.
# * Deploy built artifacts from master branch only.
# * Shows how to use multiple jobs in test stage for verifying functionality
# with multiple JDKs.
# * Uses site:stage to collect the documentation for multi-module projects.
# * Publishes the documentation for `master` branch.
variables:
# This will suppress any download for dependencies and plugins or upload messages which would clutter the console log.
# `showDateTime` will show the passed time in milliseconds. You need to specify `--batch-mode` to make this work.
MAVEN_OPTS: "-Dhttps.protocols=TLSv1.2 -Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=WARN -Dorg.slf4j.simpleLogger.showDateTime=true -Djava.awt.headless=true"
# As of Maven 3.3.0 instead of this you may define these options in `.mvn/maven.config` so the same config is used
# when running from the command line.
# `installAtEnd` and `deployAtEnd` are only effective with recent version of the corresponding plugins.
MAVEN_CLI_OPTS: "--batch-mode --errors --fail-at-end --show-version -DinstallAtEnd=true -DdeployAtEnd=true"
# Cache downloaded dependencies and plugins between builds.
# To keep cache across branches add 'key: "$CI_JOB_NAME"'
cache:
paths:
- .m2/repository
# This will only validate and compile stuff and run e.g. maven-enforcer-plugin.
# Because some enforcer rules might check dependency convergence and class duplications
# we use `test-compile` here instead of `validate`, so the correct classpath is picked up.
.validate: &validate
stage: build
script:
- 'mvn $MAVEN_CLI_OPTS test-compile'
# For merge requests do not `deploy` but only run `verify`.
# See https://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html
.verify: &verify
stage: test
script:
- 'mvn $MAVEN_CLI_OPTS verify site site:stage'
except:
- master
# Validate merge requests using JDK7
validate:jdk7:
<<: *validate
image: maven:3.3.9-jdk-7
# Validate merge requests using JDK8
validate:jdk8:
<<: *validate
image: maven:3.3.9-jdk-8
# Verify merge requests using JDK7
verify:jdk7:
<<: *verify
image: maven:3.3.9-jdk-7
# Verify merge requests using JDK8
verify:jdk8:
<<: *verify
image: maven:3.3.9-jdk-8
# For `master` branch run `mvn deploy` automatically.
# Here you need to decide whether you want to use JDK7 or 8.
# To get this working you need to define a volume while configuring your gitlab-ci-multi-runner.
# Mount your `settings.xml` as `/root/.m2/settings.xml` which holds your secrets.
# See https://maven.apache.org/settings.html
deploy:jdk8:
# Use stage test here, so the pages job may later pickup the created site.
stage: test
script:
- 'mvn $MAVEN_CLI_OPTS deploy site site:stage'
only:
- master
# Archive up the built documentation site.
artifacts:
paths:
- target/staging
image: maven:3.3.9-jdk-8
pages:
image: busybox:latest
stage: deploy
script:
# Because Maven appends the artifactId automatically to the staging path if you did define a parent pom,
# you might need to use `mv target/staging/YOUR_ARTIFACT_ID public` instead.
- mv target/staging public
dependencies:
- deploy:jdk8
artifacts:
paths:
- public
only:
- master
I'm using a local GitLab runner which is configured to build and test in Docker. In registering the runner I chose the 'shell' option
From the sounds of it, you have registered the gitlab-runner incorrectly for the mode you are after. You said you want to use a runner to build and test in Docker, however you have registered the runner in shell mode.
To use the runner in Docker, follow the instructions set in the documentation. Make sure to set the runner executor to Docker when registering.
You'd also ideally remove the shell runner you have created.
If you have issues registering the runner, see this answer, which may also help with setting up your environment.

Break Push in GitLab based on SonarQube Analysis Result

I have an application in springboot which uses gradle to build the code.
I have setup https://github.com/gabrie-allaigre/sonar-gitlab-plugin on SonarQube and have integrated gitlab CI
to analyse code on every push/commit. What I want to achieve is to break the push/commit if the analysis fails.
Below is my .gitlab-ci.yml
image: XXXXXX:oraclejdk:1.8.0_121
before_script:
- export GRADLE_USER_HOME=`pwd`/.gradle
sonarqube_master_job:
stage: test
only:
- master
- release2.0
script:
- ./gradlew assemble
- ./gradlew -x test sonarqube -Dsonar.host.url=http://sonarqube.XXX.XXX.XXX:9000/sonarqube -Dsonar.login=xxxxxxxxxxxxxxxxxxxx
sonarqube_preview_feature_job:
stage: test
only:
- /^feature\/*/
- development
script:
- git checkout $CI_COMMIT_REF_NAME
- git merge --no-commit --no-ff
- ./gradlew assemble
- ./gradlew -x test sonarqube -Dsonar.host.url=http://XXXX.XXXXX.com:9000/sonarqube -Dsonar.login=xxxxxxxxxxxxxxxxxxxxx -Dsonar.analysis.mode=preview -Dsonar.gitlab.commit_sha=$CI_COMMIT_REF -Dsonar.gitlab.ref_name=$CI_COMMIT_REF_NAME -Dsonar.gitlab.project_id=$CI_PROJECT_ID --stacktrace
How do I make sure the push fails if the analysis fails? Do I need to use webhooks. Is there a sample CI file?
#jibsonline, You can refer to my answer provided in the below link.
However the script answers only how to break the build on sonar analysis and display the results.
How to integrate Sonar Quality Gates with Gitlab-CI
Since gitlab triggers the build, once the changes were pushed, it is not advisable to set up an automated tool to revert the code changes on your behalf. Whenever a build fails, write script (dependencies) such that the code will not be deployed. Since the code is not deployed, your environment will not be effected. Also,set up an email configuration whenever build fails.

Gitlab-CI how to use artifacts in different pipeline

Currently, I have two main project.
1-) Vue Project which contains (webviews for IOS and Android, websites, and renderer for our Electron ) they are sharing components & API's.
2-) Electron Project which builds desktop app for (windows, darwin, linux)
i would like to automate our building, releasing process. my current setup..
before_script:
- apt-get update
- apt-get install zip unzip
- rm -rf vue-project
- git clone vue-project
- cd vue-project
- git checkout dev
- git pull
- sed -i "/\b\(areaCode\|inline-svg-loader\)\b/d" ./packages/devtool/package.json
- yarn install
- ln -s vue-project/packages/desktop/ web
- npm install
build_darwin:
stage: build
script:
- npm run package -- darwin --deploy
cache:
paths:
- vue-project/node_modules
- node_modules
which basically before bundling electron project it's cloning vue-project install dependencies and bundling electron-renderer then when it's finish. i'm running package.
I would like to separate this two different job from each other. is there anyway i could use artifacts from different project gitlab-CI pipelines ?
any help would be an appreciated.
Gitlab has a API for do a lot of tricks.
curl --header "PRIVATE-TOKEN:YOURPRIVATETOKEN" "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/master/download?job=test"
for download it as a file.
curl --header "PRIVATE-TOKEN:YOURPRIVATETOKEN" -o artifacts.zip "http://gitlab.example.net/api/v4/projects/<projectnumber>/jobs/artifacts/master/download?job=build_desktop
Gitlab can certainly support this. To accomplish this, follow these steps:
ARTIFACT GENERATION
In your Vue Project, modify your job(s) of interest to store artifacts relevant to the Electron project. Each job's artifacts are defined using Gitlab Job Artifacts notation, and are uploaded at job completion to Gitlab, and stored associated to your Project, Branch, and Job.
Note: Branch is often overlooked, and matters when you want to retrieve your artifacts, more on this later.
Illustrating:
Vue Project .gitlab_ci.yml
stages:
- stage1
- ...
vue-job1:
stage: stage1
script:
- echo "vue-job1 artifact1" > ./artifact1
- echo "vue-job1 artifact2" > ./artifact2
artifacts:
when: always
paths:
- ./artifact1
- ./artifact2
expire_in: 90 days
vue-job2:
stage: stage1
script:
# error, would overwrite job1's artifacts since working
# directory is a global space shared by all pipeline jobs
# - echo "vue-job2 artifact1" > ./artifact1
- echo "vue-job2 artifact1" > ./artifact3
artifacts:
when: always
paths:
- ./artifact3
expire_in: 90 days
The artifacts generated above are written to the working directory, which is a clone of your project's repo. So be careful with filename conflicts. To be safe, put your artifacts in a subdirectory (eg: cat "foo" > ./subdir/artifact) and reference them in paths the same way (paths: - ./subdir/artifact). You can use 'ls' in your script to view the working directory.
When your job completes, you can confirm the artifacts stored in Gitlb by using the Gitlab UI. View the job output, and use the Browse button under Job Artifacts on the right panel.
ARTIFACT RETRIEVAL
In your Electron Project, modify your job(s) of interest to retrieve artifacts stored in the Vue Project using the Gitlab Job Artifacts API and curl. In order to access the Vue artifacts, you will need the Vue Project, Branch, and Job that the artifacts were created under.
Project: For Project, use the Project ID displayed in the Gitlab UI Project Details screen.
Branch: Usually master, but depends on the branch your pipeline executes against. Although this is not relevant for your problem, if you are generating and consuming artifacts across executions of the same pipeline, use the Gitlab variable $CI_COMMIT_BRANCH for the branch.
Job: Generally the Job Name that generated the artifacts for your Project. But if you need artifacts produced by a specific Job, then use the Job Number and the corresponding retrieval API.
Illustrating:
Electron Project .gitlab_ci.yml
stages:
- stage1
- ...
electron-job1:
stage: stage1
script:
- curl -o ./artifact1 -H "PRIVATE-TOKEN:$TOKEN" https://gitlab.example.com/api/v4/projects/$VUE_PROJECT_ID/jobs/artifacts/$BRANCH/raw/artifact1?job=vue-job1
- curl -o ./artifact2 -H "PRIVATE-TOKEN:$TOKEN" https://gitlab.example.com/api/v4/projects/$VUE_PROJECT_ID/jobs/artifacts/$BRANCH/raw/artifact2?job=vue-job1
- curl -o ./artifact3 -H "PRIVATE-TOKEN:$TOKEN" https://gitlab.example.com/api/v4/projects/$VUE_PROJECT_ID/jobs/artifacts/$BRANCH/raw/artifact3?job=vue-job2
This script retrieves artifacts individually to the working directory of your Electron Project. There are also options for retrieving all artifacts for your job at once as a zip archive.
MISCELLANEOUS
Although this is not in the problem posed, it is worth noting that you can use artifacts both within the lifespan of a single pipeline execution to pass information between jobs. You can also use this to pass information across pipeline executions within the same project.
With recent versions of gitlab, this can be achieved simply by using either the multi-project pipeline feature (then starting a pipeline in one project triggers a build from the other project): see the documentation
Or you can also use the "needs:project" mechanism which allows one job to download artifacts from other pipelines (see the documentation Use needs:project to download artifacts from up to five jobs in other pipelines.)

Resources