Firstly I set up 1 worker for 1 job. Deploying my backend for the API.
I'm using "shell" as the executer. The .toml file is this structure:
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "Gitlab Runner Josere Backend"
url = "https://gitlab.com/"
token = "sOmEtOkeN1G0Tfr0mGitlab"
executor = "shell"
[runners.custom_build_dir]
[runners.cache]
[some mumbo jumbo about caching.. does it matter?]
With some struggle I got that to work fine with this .gitlab-ci.yml:
deploy-production:
stage: deploy
variables:
GIT_STRATEGY: clone
script:
- cd ./lumen/
- composer install
- sudo cp -r $CI_PROJECT_DIR/lumen/. /home/josere/public_html/api/
- sudo cp /home/josere/env/.env /home/josere/public_html/api
This is the execution output of the runner:
Running with gitlab-runner 15.2.1 (32fc1585)
on Gitlab Runner Josere backend 9JxGrMLz
Preparing the "shell" executor
00:00
Using Shell executor...
Preparing environment
00:00
Running on ####[my server]#####...
Getting source from Git repository
00:03
Fetching changes with git depth set to 50...
Initialized empty Git repository in /home/gitlab-runner/builds/9JxGrMLz/0/paspalas/josere/.git/
Created fresh repository.
... etc ...
In my frontend repo in Gitlab I went to the same runners settings. I can't really install a runner (its allready running I guess) but I can copy the token that is shown there.
Then I changed my .toml file according to this doc from gitlab (https://docs.gitlab.com/runner/fleet_scaling/):
concurrent = 2
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "Gitlab Runner Josere Backend"
url = "https://gitlab.com/"
token = "sOmEtOkeN1G0Tfr0mGitlab"
executor = "shell"
[runners.custom_build_dir]
[runners.cache]
[some mumbo jumbo about caching.. does it matter?]
[[runners]]
name = "Gitlab Runner Josere Frontend"
url = "https://gitlab.com/"
token = "TheOtherTokenThatIgotFromFrontendRepo!"
executor = "shell"
[runners.custom_build_dir]
[runners.cache]
[some mumbo jumbo about caching.. does it matter?]
notice I do keep the executor on "shell".
this is the script for .gitlab-ci.yml that goes in the root of the frontend repo:
deploy-production:
stage: deploy
variables:
GIT_STRATEGY: clone
script:
- npm install
- npm run build
- sudo cp -r $CI_PROJECT_DIR/public/. /home/josere/public_html/
But when I commit my frontend and check the (failing) log for the worker it writes this:
Running with gitlab-runner 15.4.0~beta.5.gdefc7017 (defc7017)
on green-1.shared.runners-manager.gitlab.com/default JLgUopmM
Preparing the "docker+machine" executor
00:06
Using Docker executor with image ruby:2.5 ...
Pulling docker image ruby:2.5 ...
Using docker image sha256:27d###mumbojumbo###2383b for ruby:2.5 with digest ruby#sha256:ecc3###mumbojumbo###444b ...
Preparing environment
00:00
Running on runner-jlguopmm-project-39467125-concurrent-0 via runner-jlguopmm-shared-1665674167-6adf45bf...
Getting source from Git repository
00:02
$ eval "$CI_PRE_CLONE_SCRIPT"
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/paspalas/josere-frontend/.git/
Created fresh repository.
Checking out c39e641c as materialui...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:01
Using docker image sha256:27d###mumbojumbo###3b for ruby:2.5 with digest ruby#sha256:ecc3e###mumbojumbo####44b ...
$ sudo npm install
/bin/bash: line 126: sudo: command not found
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1
clearly it seems multiple things go wrong, to start with: why is it using docker while I explicitly tell it to be "shell"?
I fixed the issue. Even though the docs of GitLab differentiates between "runner" and "job", the gitlab-runner calls these "registrations" of a "runner". I did the (extra) registeration like so:
- gitlab-runner register
[filling in info]
- nano /etc/gitlab-runner/config.toml
[check if you have the additional runner]
- gitlab-runner run
[according to gitlab-runner help this is to fire up multiple runners]
- gitlab-runner list
[ now you can check if all "runners" (jobs) are running]
Related
Summary
My gitlab-ci.yml has 3 stage for deploy an application to okd pod
Application running spring boot on tomcat:8
Sometimes, the cache.zip is not update after stage complete so that the next step can't run correctly
Steps to reproduce
My gitlab-ci run the following stage
Stage 1: run test compile ---> OK
Stage 2: package war file as output for deploy ---> Gitlab-ci log show success but the cache.zip has not war file (just sometimes cache.zip not have war file, sometimes it run correctly)
Stage 3: Deploy war file to pod ---> Because of war file not exists in cache.zip, script error -> failed
.gitlab-ci.yml
image: openshift/origin-cli
stages:
- build
- test
- staging
cache:
paths:
- .m2/repository
- target
- artifact
validate:jdk8:
stage: build
script:
- 'mvn test-compile'
only:
- master
image: maven:3.3.9-jdk-8
verify:jdk8:
stage: test
script:
- 'mvn verify'
- 'mvn package' # =====> this command generate war file
only:
- master
image: maven:3.3.9-jdk-8
staging:
script:
- "mkdir -p artifact"
- "cp ./target/*.war ./artifact/" # ======> Sometimes error at this line because of previous step not add war file into cache
- "oc start-build $APP"
- "rm -rf ./target/* && rm -rf ./artifact/*" # Remove war & class file, only cache m2 lib
stage: staging
variables:
APP: $CI_PROJECT_NAME
environment:
name: staging
url: http://$CI_PROJECT_NAME-staging.$OPENSHIFT_DOMAIN
only:
- master
Actual behavior
Sometimes cache not have war file after test stage complete (is this depends on war file size?)
Expected behavior
War file update into cache after test stage for staging stage deploy
Relevant logs and/or screenshots
ScreenShot
job log
Running with gitlab-runner 13.7.0 (943fc252)
on gitlab-runner-node1 y6awygsj
Preparing the "docker" executor
00:01
Using Docker executor with image openshift/origin-cli ...
Using locally found image version due to if-not-present pull policy
Using docker image sha256:7ebb6be01117a50344d63f77c385a13302afecd33480b97c36a518d4f5ebc25a for openshift/origin-cli with digest docker.io/openshift/origin-cli#sha256:509e052d0f2d531b666b7da9fa49c5558c76ce5d286456f0859c0a49b16d6bf2 ...
Preparing environment
00:00
Running on runner-y6awygsj-project-489-concurrent-0 via gitlab.runner.node1...
Getting source from Git repository
00:01
Fetching changes...
Reinitialized existing Git repository in /builds/my-project/.git/
Checking out b4c97428 as master...
Removing .m2/
Removing artifact/
Removing target/
Skipping Git submodules setup
Restoring cache
00:05
Checking cache for default-23...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache
Executing "step_script" stage of the job script
00:01
$ mkdir -p artifact
$ cp ./target/*.war ./artifact/
cp: cannot stat './target/*.war': No such file or directory
Cleaning up file based variables
00:00
ERROR: Job failed: exit code 1
Environment description
config.toml
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "gitlab-runner-node1"
url = "https://gitlab.mycompany.vn/"
token = "y6awygsj9zks18nU6PDt"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
dns = ["192.168.100.1"]
tls_verify = false
image = "alpine:latest"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/mnt/nfs/nfsshare-gitlab/cache:/cache"]
shm_size = 0
pull_policy = "if-not-present"
Used GitLab Runner version
Version: 13.7.0
Git revision: 943fc252
Git branch: 13-7-stable
GO version: go1.13.8
Built: 2020-12-21T13:47:06+0000
OS/Arch: linux/amd64
Possible fixes
Re-run test stage until cache has war file
Let's go step by step.
First, regarding how to manage the files between stages.
It's true that you could directly access to the files between jobs and stages if both run on the same environment, but that's not always the case (even if both runners are using the same nfs share directory) and you should use artifacts for that.
When you define an artifact within a job, you're specifying a list of files that are attached to the job when it succeeds, fails or always, depending on the configuration you have.
By default, all artifacts from previous stages are passed to each job, but in any case you can use dependencies to also define from which jobs you want to fetch artifacts from.
So basically you should use the following .gitlab-ci.yml
image: openshift/origin-cli
stages:
- build
- test
- staging
cache:
paths:
- .m2/repository
validate:jdk8:
stage: build
script:
- 'mvn test-compile'
only:
- master
image: maven:3.3.9-jdk-8
verify:jdk8:
stage: test
script:
- 'mvn verify' # =====> verify already includes: validate, compile, test and package
artifacts:
paths:
- target/[YOUR_APP_NAME].war
only:
- master
image: maven:3.3.9-jdk-8
staging:
dependencies:
- verify:jdk8
script:
- "mkdir -p artifact"
- "cp ./target/[YOUR_APP_NAME].war ./artifact/"
- "oc start-build $APP"
stage: staging
variables:
APP: $CI_PROJECT_NAME
environment:
name: staging
url: http://$CI_PROJECT_NAME-staging.$OPENSHIFT_DOMAIN
only:
- master
Also, notice that I deleted the mvn package instruction. I would recommend you to take a look into the Build Lifecycle Basics of Maven.
I have GitLab runner installed on a MacOS using homebrew. The runner configuration is located under ${HOME}/.gitlab-runner/config.toml and service configuration located under ${HOME}/Library/LaunchAgents/homebrew.mxcl.gitlab-runner.plist is the default.
Below is my gitlab-runner toml configuration file.
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "MY_RUNNER_NAME"
url = "https://gitlab.com/"
token = "MY_GITLAB_TOKEN"
executor = "shell"
shell = "bash"
environment = ["PATH=/usr/local/opt/openjdk#8/bin:/usr/local/opt/ruby#2.7/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin", "LC_ALL=en_US.UTF-8", "LANG=en_US.UTF-8"]
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
The runner is connected correctly to Gitlab.com, it executes the steps correctly, but it got stuck in the Uploading artifacts step until the build times out.
Below is my Uploading artifacts logs.
Uploading artifacts...
Runtime platform arch=amd64 os=darwin pid=16690 revision=775dd39d version=13.8.0
android/app/build/outputs/bundle/release/app-release.aab: found 1 matching files and directories
ERROR: Job failed: execution took longer than 1h0m0s seconds
As a debugging step, I tried to run the gitlab-runner artifacts-uploader locally to trace the behavior using this command gitlab-runner --debug --log-level debug artifacts-uploader --verbose --id MY_BUILD_ID --token MY_GITLAB_TOKEN --url https://gitlab.com/ --path android/app/build/outputs/bundle/release/app-release.aab --expire-in "1 week".
Below is my gitlab-runner artifacts-uploader logs.
Runtime platform arch=amd64 os=darwin pid=25259 revision=775dd39d version=13.8.0
android/app/build/outputs/bundle/release/app-release.aab: found 1 matching files and directories
Dialing: tcp gitlab.com:443 ...
It is obvious that the gitlab-runner artifacts-uploader got stuck connecting to gitlab.com:443, and now I am out of ideas on how to trace or solve this issue.
My Docker image is failing during build in GitLab CI and it fails silently without giving any errors to work with. I can build the image locally and no problem whatsoever so the problem is in CI environment. Something that is not obvious causes the build to fail. After doing some research about this I've learned the best thing to do to SSH into the CI server and "poke around" to find out what's happening. In particular I've learned that I can get a log of the last layer before the build fails to get insight into why it might be failing. However, GitLab doesn't support direct SSH connection into CI server. Supports only fixed SSH commands executed towards the server from the build environment (.gitlab-ci.yml) which isn't very helpful because I need to use SSH to access build layers of the image.
What are my other options as to how can I debug / analyze an image during build in CI ?
Any feedback much appreciated.
Dockerfile:
###########
# BUILDER #
###########
# base image
FROM node:11.12.0-alpine as builder
# set working directory
WORKDIR /usr/src/app
RUN apk add --no-cache --virtual .gyp python make g++
# install app dependencies
ENV PATH /usr/src/app/node_modules/.bin:$PATH
COPY package.json /usr/src/app/package.json
COPY package-lock.json /usr/src/app/package-lock.json
RUN npm install --no-optional
RUN npm install react-scripts#2.1.8 -g --silent --no-optional
# set environment variables
ARG REACT_APP_USERS_SERVICE_URL
ENV REACT_APP_USERS_SERVICE_URL $REACT_APP_USERS_SERVICE_URL
ARG NODE_ENV
ENV NODE_ENV $NODE_ENV
# create build
COPY . /usr/src/app
RUN npm run build
#########
# FINAL #
#########
# base image
FROM nginx:1.15.9-alpine
# update nginx conf
RUN rm -rf /etc/nginx/conf.d
COPY conf /etc/nginx
# copy static files
COPY --from=builder /usr/src/app/build /usr/share/nginx/html
# expose port
EXPOSE 80
# run nginx
CMD ["nginx", "-g", "daemon off;"]
.gitlab-ci.yml file:
...
...
after_script:
- bash ./docker-push.sh
- docker-compose down
docker-push.sh script that builds the image for pushing into ECR on AWS:
echo "building the client image ..."
docker -D build $CLIENT_REPO -t $CLIENT:$COMMIT -f Dockerfile-prod --build-arg REACT_APP_USERS_SERVICE_URL="" # this line is failing
if [ $? -ne 0 ]; then
echo "Failure. Exiting now..."
exit 1
fi
docker -D tag $CLIENT:$COMMIT $REPO/$CLIENT:$TAG
docker -D push $REPO/$CLIENT:$TAG
docker build $USERS_REPO -t $USERS:$COMMIT -f Dockerfile-$DOCKER_ENV
docker tag $USERS:$COMMIT $REPO/$USERS:$TAG
docker push $REPO/$USERS:$TAG
docker build $USERS_DB_REPO -t $USERS_DB:$COMMIT -f Dockerfile
docker tag $USERS_DB:$COMMIT $REPO/$USERS_DB:$TAG
docker push $REPO/$USERS_DB:$TAG
docker build $SWAGGER_REPO -t $SWAGGER:$COMMIT -f Dockerfile-$DOCKER_ENV
docker tag $SWAGGER:$COMMIT $REPO/$SWAGGER:$TAG
docker push $REPO/$SWAGGER:$TAG
job log from gitlab ci (relevant part only):
Login Succeeded
building the client image ...
time="2020-04-14T08:54:23Z" level=debug msg="Skipping excluded path: .dockerignore"
time="2020-04-14T08:54:23Z" level=debug msg="Skipping excluded path: Dockerfile"
time="2020-04-14T08:54:23Z" level=debug msg="Skipping excluded path: Dockerfile-prod"
time="2020-04-14T08:54:23Z" level=debug msg="Skipping excluded path: Dockerfile-stage"
time="2020-04-14T08:54:23Z" level=debug msg="Skipping excluded path: .dockerignore"
time="2020-04-14T08:54:23Z" level=debug msg="Skipping excluded path: Dockerfile-prod"
time="2020-04-14T08:54:23Z" level=debug msg="Skipping excluded path: Dockerfile"
time="2020-04-14T08:54:23Z" level=debug msg="Skipping excluded path: Dockerfile-stage"
Step 1/25 : FROM node:11.12.0-alpine as builder
---> 09084e4ff58d
Step 2/25 : WORKDIR /usr/src/app
---> Using cache
---> 9c6639a8a785
Step 3/25 : RUN apk add --no-cache --virtual .gyp python make g++
---> Using cache
---> 0d5320ee514b
Step 4/25 : ENV PATH /usr/src/app/node_modules/.bin:$PATH
---> Using cache
---> c041f8c64b34
Step 5/25 : COPY package.json /usr/src/app/package.json
---> 02d18d67a517
Step 6/25 : COPY package-lock.json /usr/src/app/package-lock.json
---> 2d94e8e8fb6c
Step 7/25 : RUN npm install --no-optional
---> Running in 59660215041e
> cypress#4.1.0 postinstall /usr/src/app/node_modules/cypress
> node index.js --exec install
Installing Cypress (version: 4.1.0)
[08:55:20] Downloading Cypress [started]
[08:55:20] Downloading Cypress 0% 0s [title changed]
[08:55:20] Downloading Cypress 2% 5s [title changed]
...
...
[08:55:39] Unzipping Cypress 9% 167s [title changed]
[08:55:39] Unzipping Cypress 100% 0s [title changed]
[08:55:39] Unzipped Cypress [title changed]
[08:55:39] Unzipped Cypress [completed]
[08:55:39] Finishing Installation [started]
[08:55:40] Finished Installation /root/.cache/Cypress/4.1.0 [title changed]
[08:55:40] Finished Installation /root/.cache/Cypress/4.1.0 [completed]
You can now open Cypress by running: node_modules/.bin/cypress open
https://on.cypress.io/installing-cypress
added 2034 packages from 768 contributors and audited 38602 packages in 77.201s
found 1073 vulnerabilities (1058 low, 14 moderate, 1 high)
run `npm audit fix` to fix them, or `npm audit` for details
Saving cache
00:02
Uploading artifacts for successful job
00:02
Job succeeded
I am trying to integrate CircleCI plugin with my spring-pet-clinic project. I was following the instruction on CircleCI web page. I have created .circleci folder inside my project root folder.
Inside .circleci I have added config.yml file and copy-pasted config from CircleCI page
My config was like this:
# Use the latest 2.1 version of CircleCI pipeline processing engine,
see https://circleci.com/docs/2.0/configuration-reference/
version: 2.1
# Use a package of configuration called an orb, see
https://circleci.com/docs/2.0/orb-intro/
orbs:
# Declare a dependency on the welcome-orb
welcome: circleci/welcome-orb#0.3.1
# Orchestrate or schedule a set of jobs, see
https://circleci.com/docs/2.0/workflows/
workflows:
# Name the workflow "Welcome"
Welcome:
# Run the welcome/run job in its own container
jobs:
- welcome/run
After I ran the project CircleCI has thrown an error. Especially this one: "Config Processing Error: Don't rerun"
$#!/bin/sh -eo pipefail
# No configuration was found in your project. Please refer to
https://circleci.com/docs/2.0/ to get started with your
configuration.
#
# -------
# Warning: This configuration was auto-generated to show you
the
message above.
# Don't rerun this job. Rerunning will have no effect.
false
Exited with code 1
Spin Up Environment looks like this
Build-agent version 1.0.10572-3ce00c85 (2019-04-
15T22:09:28+0000)
Docker Engine Version: 17.05.0-ce
Kernel Version: Linux b0a81c56acff 4.4.0-144-generic
#170~14.04.1-
Ubuntu SMP Mon Mar 18 15:02:05 UTC 2019 x86_64 Linux
Starting container bash:4.4.19
using image
bash#sha256:9f0a4aa3c9931bd5fdda51b1b2b74a0398a8eabeaf9519d807e010b9d9d41993
Using build environment variables
BASH_ENV=/tmp/.bash_env-5cbebf83d4b030000849b60f-0-build
CI=true
CIRCLECI=true
CIRCLE_BRANCH=master
CIRCLE_BUILD_NUM=5
CIRCLE_BUILD_URL=https://circleci.com/gh/sajmon2325/Spring-
Pet-
Clinic/5
CIRCLE_COMPARE_URL=
CIRCLE_JOB=Build Error
CIRCLE_NODE_INDEX=0
CIRCLE_NODE_TOTAL=1
CIRCLE_PREVIOUS_BUILD_NUM=4
CIRCLE_PROJECT_REPONAME=Spring-Pet-Clinic
CIRCLE_PROJECT_USERNAME=sajmon2325
CIRCLE_REPOSITORY_URL=git#github.com:sajmon2325/Spring-Pet-
Clinic.git
CIRCLE_SHA1=48f6db114b41c338e606de32d8648c64ba5119fd
CIRCLE_SHELL_ENV=/tmp/.bash_env-5cbebf83d4b030000849b60f-0-
build
CIRCLE_STAGE=Build Error
CIRCLE_USERNAME=sajmon2325
CIRCLE_WORKFLOW_ID=2789d93e-f1e4-4c81-93f1-846f7d38c107
CIRCLE_WORKFLOW_JOB_ID=670105ca-617e-445e-9b5e-6ac57f6af8da
CIRCLE_WORKFLOW_UPSTREAM_JOB_IDS=
CIRCLE_WORKFLOW_WORKSPACE_ID=2789d93e-f1e4-4c81-93f1-
846f7d38c107
CIRCLE_WORKING_DIRECTORY=~/project
Using environment variables from project settings and/or
contexts
CIRCLE_JOB=**REDACTED**
So at first I thought that I have only a skeleton of CircleCI configuration, that's why I have edited my config.yml file to look like this (the actual version)
# Java Maven CircleCI 2.0 configuration file
#
# Check https://circleci.com/docs/2.0/language-java/ for more details
#
version: 2
jobs:
build:
docker:
# specify the version you desire here
- image: circleci/openjdk:11-browsers-legacy
# Specify service dependencies here if necessary
# CircleCI maintains a library of pre-built images
# documented at https://circleci.com/docs/2.0/circleci-images/
# - image: circleci/postgres:9.4
working_directory: ~/repo
environment:
# Customize the JVM maximum heap limit
MAVEN_OPTS: -Xmx3200m
steps:
- checkout
# Download and cache dependencies
- restore_cache:
keys:
- v1-dependencies-{{ checksum "pom.xml" }}
# fallback to using the latest cache if no exact match
is
found
- v1-dependencies-
- run: mvn install -DskipTests
- run: mvn dependency:go-offline
- save_cache:
paths:
- ~/.m2
key: v1-dependencies-{{ checksum "pom.xml" }}
# run tests!
- run: mvn integration-test
But even this is not working. I still have the same error:
$#!/bin/sh -eo pipefail
# No configuration was found in your project. Please refer to https://circleci.com/docs/2.0/ to get started with your configuration.
#
# -------
# Warning: This configuration was auto-generated to show you the message above.
# Don't rerun this job. Rerunning will have no effect.
false
Exited with code 1
I just need to successfully integrate CircleCi plugin with my project. If you need to see my repo, here is the link: https://github.com/sajmon2325/Spring-Pet-Clinic.git
The problem is that .circleci is not in the root of the repository. It is currently in sfg-pet-clinic/, and the CircleCI build process won't find it there.
I have registred gitlab runner to the gitlab instance. My registered URL is as follows: http://azurestackgitlab1.southeastasia.cloudapp.azure.com/ci
However while running the build for that project; build is getting failed and giving the below message:
Running with gitlab-ci-multi-runner 1.11.1 (a67a225)
on java test (96d320b3)
WARNING: image is not supported by selected executor and shell
Using Shell executor...
Running on AzureStackPOCVM...
Cloning repository...
Cloning into '/home/gitlab-runner/builds/96d320b3/0/root/demoproject'...
fatal: unable to access 'http://gitlab-ci-token:xxxxxx#gitlab-ce.hxakzvpf0otezeojz3wqhme5wg.cx.internal.cloudapp.net/root/demoproject.git/': Could not resolve host: gitlab-ce.hxakzvpf0otezeojz3wqhme5wg.cx.internal.cloudapp.net
ERROR: Job failed: exit status 1
snippet of config.toml:
[[runners]]
name = "java test"
url = "http://azurestackgitlab1.southeastasia.cloudapp.azure.com/ci"
token = "96d320b33d3c69d706dad7f90df84e"
executor = "shell"
[runners.cache]
How to overcome this problem?
You will get this URL mentioned in etc/gitlab/gitlab.rb file. If you will change "hxakzvpf0otezeojz3wqhme5wg.cx.internal.cloudapp.net"
with your "http://azurestackgitlab1.southeastasia.cloudapp.azure.com"
It should work!
For more information you can refer : http://azurestackgitlab1.southeastasia.cloudapp.azure.com/help/administration/environment_variables.md