Gitlab ci with unit testing that connects to MySQL - go

I have built a Golang project, I want to deploy the app when successfully tested with GitLab ci, but when do some tests, it fails because cannot connect to MySQL.
I want to use the Golang image and MySQL image in one stage.
This is my current pipeline. On stage test, on before script fails (/bin/bash: line 130: mysql: command not found)
# To contribute improvements to CI/CD templates, please follow the Development guide
at:
# https://docs.gitlab.com/ee/development/cicd/templates.html
# This specific template is located at:
# https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Go.gitlab-ci.yml
image: golang:latest
services:
- mysql:latest
stages:
- test
- build
- deploy
variables:
MYSQL_DATABASE: "db"
MYSQL_USER: "user"
MYSQL_PASSWORD: "password"
MYSQL_ROOT_PASSWORD: "password"
format:
stage: test
variables:
# Configure mysql environment variables (https://hub.docker.com/_/mysql/)
MYSQL_DATABASE: $MYSQL_DATABASE
MYSQL_PASSWORD: $MYSQL_PASSWORD
MYSQL_ROOT_PASSWORD: $MYSQL_ROOT_PASSWORD
services:
- mysql:latest
before_script:
- mysql --version
script:
- go fmt $(go list ./... | grep -v /vendor/)
- go vet $(go list ./... | grep -v /vendor/)
- go test -race $(go list ./... | grep -v /vendor/)
compile:
stage: build
script:
- mkdir -p typing
- go build -o typing ./...
artifacts:
paths:
- typing
deploy:
image: google/cloud-sdk:alpine
stage: deploy
allow_failure: true
script:
- "which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )"
# Run ssh-agent (inside the build environment)
- eval $(ssh-agent -s)
# Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
- ssh-add <(echo "$SSH_PRIVATE_KEY" | base64 -d)
- mkdir -p ~/.ssh
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
- (echo "$SSH_PRIVATE_KEY" | base64 -d) > file
- echo "$SSH_PUBLIC_KEY" > file.pub
- chmod 600 file
- echo $SERVICE_ACCOUNT > file.json
- gcloud auth activate-service-account --key-file file.json
- gcloud compute scp typing/* --project="project-id" --zone="zone" vm-name:/home/ubuntu
- ssh -i file ubuntu#public-ip 'sudo ./kill.sh; sudo ./start.sh'
artifacts:
paths:
- typing
How can I achieve that?
Thanks in advance.

In the stage test, the job is based on the golang image as a result it does not come packaged with the MySQL client.
In order to reach the MySQL service you defined you need to install the client
If I am not mistaken the Golang image is based on debian, so something like this
before_script:
- apt get-update
- apt get-install -y default-mysql-client
- mysql --version

Related

Go get "get" unexpected EOF

Thank you for visiting here.
First of all, I apologize for my bad English, maybe a little wrong, hope you can help me.
Then I had a little problem when deploying a new CI/CD system on k8s platform (v1.23.5+1) with Gitlab runner (14.9.0) and dind (docker:dind)
When deploying CI to Golang apps with private repositories at https://gitlab.domain.com, (I did the go env -w GOPRIVATE configuration), I had a problem with the go mod tidy command. Specifically getting the unexpected EOF error. I've tried go mod tidy -v but it doesn't seem to give any more info.
I did a lot of work to figure out the problem. Specifically, I have done wget and git clone commands with my private repository and they are still able to download successfully. I tried adding a private repository at https://gitlab.com in go.mod, they can still be retrieved without any errors.
And actually, without using my new runner, I can still git clone and go mod tidy in another vps.
All of this leaves me wondering where am I actually getting the error? Is it my gitlab or my k8s gitlab runner
This is runner output
go: downloading gitlab.domain.com/nood/fountain v0.0.12
unexpected EOF
Cleaning up project directory and file based variables
ERROR: Job failed: command terminated with exit code 1
This is my .gitlab-ci.yml
image: docker:latest
stages:
- build
- deploy
variables:
GTV_ECR_REPOSITORY_URL: repo.domain.com
PROJECT: nood
APP_NAME: backend-super-system
APP_NAME_ECR: backend-super-system
IMAGE_TAG: $GTV_ECR_REPOSITORY_URL/$PROJECT/$APP_NAME_ECR
DOCKER_HOST: tcp://docker:2375/
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: ""
services:
- name: docker:dind
entrypoint: ["env", "-u", "DOCKER_HOST"]
command: ["dockerd-entrypoint.sh", "--tls=false"]
build:
stage: build
allow_failure: false
script:
- echo "Building image."
- docker pull $IMAGE_TAG || echo "Building runtime from scratch"
- >
docker build
--cache-from $IMAGE_TAG
-t $IMAGE_TAG --network host .
- docker push $IMAGE_TAG
Dockerfile
FROM golang:alpine3.15
LABEL maintainer="NoodExe <nood.pr#gmail.com>"
WORKDIR /app
ENV BIN_DIR=/app/bin
RUN apk add --no-cache gcc build-base git
ADD . .
RUN chmod +x scripts/env.sh scripts/build.sh \
&& ./scripts/env.sh \
&& ./scripts/build.sh
# stage 2
FROM alpine:latest
WORKDIR /app
ENV BIN_DIR=/app/bin
ENV SCRIPTS_DIR=/app/scripts
ENV DATA_DIR=/app/data
# Build Args
ARG LOG_DIR=/var/log/nood
# Create log directory
RUN mkdir -p ${BIN_DIR} \
mkdir -p ${SCRIPTS_DIR} \
mkdir -p ${DATA_DIR} \
mkdir -p ${LOG_DIR} \
&& apk update \
&& addgroup -S nood \
&& adduser -S nood -G nood \
&& chown nood:nood /app \
&& chown nood:nood ${LOG_DIR}
USER nood
COPY --chown=nood:nood --from=0 ${BIN_DIR} /app
COPY --chown=nood:nood --from=0 ${DATA_DIR} ${DATA_DIR}
COPY --chown=nood:nood --from=0 ${SCRIPTS_DIR} ${SCRIPTS_DIR}
RUN chmod +x ${SCRIPTS_DIR}/startup.sh
ENTRYPOINT ["/app/scripts/startup.sh"]
scripts/env.sh
#!/bin/sh
go env -w GOPRIVATE=gitlab.domain.com/*
git config --global --add url."https://nood_deploy:rvbsosecret_Hizt97zQSn#gitlab.domain.com".insteadOf "https://gitlab.domain.com"
scripts/build.sh
#!/bin/sh
grep -v "replace\s.*=>.*" go.mod > tmpfile && mv tmpfile go.mod
go mod tidy
set -e
BIN_DIR=${BIN_DIR:-/app/bin}
mkdir -p "$BIN_DIR"
files=`ls *.go`
echo "****************************************"
echo "******** building applications **********"
echo "****************************************"
for file in $files; do
echo building $file
go build -o "$BIN_DIR"/${file%.go} $file
done
Thank you for still being here :3
This is a known issue with installing go modules from gitlab in nested locations. The issue describes several workarounds/solutions. One solution is described as follows:
create a gitlab Personal Access Token with at least read_api and read_repository scopes.
create a .netrc file:
machine gitlab.com
login yourname#gitlab.com
password yourpersonalaccesstoken
use go get --insecure to get your module
do not use the .gitconfig insteadOf workaround
For self-hosted instances of GitLab, there is also the additional option of using the go proxy, which is what I do to resolve this problem.
For additional context, see this answer to What's the proper way to "go get" a private repository?

Cross using yml variables in the bash script

Below I have shown my .gitlab-ci.yml file:
variables:
timezone: "Europe/Vienna"
stages:
- style
- build
style:
stage: style
script:
- sudo docker run --rm -v $PWD:/code omercnet/pycodestyle
build:
stage: build
script:
blla blla
- sudo docker exec ${CI_PROJECT_PATH_SLUG} /bin/bash -c "./install.sh"
blla blla
Next, it is my install.sh script:
#!/bin/bash
blla blla
echo -e "\nmx_download_url: https://dl.bintray.com/random" >> /etc/random-installer/setup.yml
sed -i s+old text+new text+g etc/random-installer/setup.yml
blla blla
How I can use the value of the variable timezone from the yml file in the new text in the bash script?
timezone is visible as environment variable for the gitlab jobs, however install.sh scripts is running from another context - it's executed by docker. Docker itself has access to $timezone, however containers it's operating with don't.
To solve your problem you can explicitly pass it:
- sudo docker exec -e TIMEZONE=$timezone ${CI_PROJECT_PATH_SLUG} /bin/bash -c "./install.sh"
This will set environment variable $TIMEZONE for install.sh inside the container.

Gitlab runner CI/CD do not checkout and pull last commit

I want to have a CI/CD with gitlab-runner and docker swarm. I have problem when i deploy the commit will not checkout or checkout without changes, I wonder to know if problem is gitlab or docker or docker build. my .gitlab-ci.yml, look like:
stages:
- build
- deploy
build_image:
stage: build
image: docker:git
services:
- docker:dind
variables:
DOCKER_DRIVER: overlay
script:
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN registry.gitlab.com
- docker build -t registry.gitlab.com/insuretech1/backend:${CI_COMMIT_REF_SLUG} ./DockerFiles/Worker
- docker push registry.gitlab.com/insuretech1/backend:${CI_COMMIT_REF_SLUG}
only:
- branches
deploy_staging:
stage: deploy
image: rastasheep/ubuntu-sshd:latest
script:
# add the server as a known host
- ssh-keyscan 46.4.151.121 >> ~/.ssh/known_hosts
- chmod 600 ~/.ssh/known_hosts
# add ssh key stored in SSH_PRIVATE_KEY variable to the agent store
- eval $(ssh-agent -s)
- touch key.txt
- echo "$SSH_PRIVATE_KEY" >> key.txt
- chmod 600 key.txt
- ssh-add key.txt
# log into Docker registry
- ssh alireza#46.4.151.121 "docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN registry.gitlab.com"
# stop container, remove image.
- ssh alireza#46.4.151.121 "docker stop dockergitlab_${CI_COMMIT_REF_SLUG}" || true
- ssh alireza#46.4.151.121 "docker rm dockergitlab_${CI_COMMIT_REF_SLUG}" || true
- ssh alireza#46.4.151.121 "docker rmi registry.gitlab.com/insuretech1/backend:${CI_COMMIT_REF_SLUG}" || true
# start new container
- ssh alireza#46.4.151.121 "docker run --name dockergitlab_${CI_COMMIT_REF_SLUG} -d registry.gitlab.com/insuretech1/backend:${CI_COMMIT_REF_SLUG}"
only:
- branches
except:
- master
and I also I put my pipeline log below, that might help to describe more:
$ eval "$CI_PRE_CLONE_SCRIPT"
00:02
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/insuretech1/backend/.git/
Created fresh repository.
From https://gitlab.com/insuretech1/backend
* [new ref] refs/pipelines/124187268 -> refs/pipelines/124187268
* [new branch] develop -> origin/develop
Checking out 735209a2 as develop...
Skipping Git submodules setup
$ docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN registry.gitlab.com
03:43
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
the content of dockerfile which I use for my build
FROM debian:buster
MAINTAINER Alireza Rahmani Khalili "alirezarahmani#live.com"
ENV TERM xterm
RUN apt-get update --fix-missing && apt-get install -y --force-yes curl sudo vim
RUN apt-get install -y --force-yes wget apt-transport-https lsb-release ca-certificates
RUN wget -O /etc/apt/trusted.gpg.d/php.gpg https://packages.sury.org/php/apt.gpg
RUN echo "deb https://packages.sury.org/php/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/php.list
RUN echo "deb http://ftp.uk.debian.org/debian buster-backports main" >> /etc/apt/sources.list
RUN wget -O /etc/apt/trusted.gpg.d/php.gpg https://packages.sury.org/php/apt.gpg
RUN echo "deb https://packages.sury.org/php/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/php.list
RUN DEBIAN_FRONTEND="noninteractive" apt-get update && apt-get install -y --force-yes \
nginx \
php7.3 \
php7.3-cli \
php7.3-fpm \
php7.3-curl \
php7.3-json \
php7.3-mysql \
php7.3-sqlite \
php7.3-xml \
php7.3-intl \
php7.3-mbstring \
php7.3-xdebug \
php-memcached \
git \
openssh-server \
php7.3-gd \
zip \
php7.3-zip
# configure php-fpm
RUN sed -i 's/^;*clear_env = .*/clear_env = no/' /etc/php/7.3/fpm/pool.d/www.conf
RUN curl -sS https://getcomposer.org/installer | php && \
mv composer.phar /usr/local/bin/composer && chmod +x /usr/local/bin/composer
RUN mkdir /var/run/sshd
RUN echo 'root:root' | chpasswd
RUN sed -i 's/PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config
RUN sed 's#session\s*required\s*pam_loginuid.so#session optional pam_loginuid.so#g' -i /etc/pam.d/sshd
RUN echo "UseDNS no" >> /etc/ssh/sshd_config
RUN echo "KexAlgorithms diffie-hellman-group1-sha1" >> /etc/ssh/sshd_config
RUN echo "fastcgi_param PATH_TRANSLATED \$document_root\$fastcgi_script_name;" >> /etc/nginx/fastcgi_params
RUN mkdir /etc/nginx/ssl
RUN openssl ecparam -out /etc/nginx/ssl/nginx.key -name prime256v1 -genkey
RUN openssl req -new -batch -key /etc/nginx/ssl/nginx.key -out /etc/nginx/ssl/csr.pem
RUN openssl req -x509 -nodes -days 365 -key /etc/nginx/ssl/nginx.key -in /etc/nginx/ssl/csr.pem -out /etc/nginx/ssl/nginx.pem
RUN chmod 600 /etc/nginx/ssl/*
RUN ln -sf /dev/stdout /var/log/nginx/access.log
RUN ln -sf /dev/stderr /var/log/nginx/error.log
ADD docker-entrypoint.sh /usr/bin/docker-entrypoint
RUN chmod +x /usr/bin/docker-entrypoint
RUN sed -i 's/^user nginx;/user www-data;/' /etc/nginx/nginx.conf
RUN echo "apc.enable_cli=1" >> /etc/php/7.3/cli/php.ini
RUN echo "apc.shm_size=128M" >> /etc/php/7.3/fpm/conf.d/20-apcu.ini
RUN sed -i "s/\(max_execution_time *= *\).*/\1180/" /etc/php/7.3/fpm/php.ini
RUN sed -i "s/\(upload_max_filesize *= *\).*/\1100M/" /etc/php/7.3/fpm/php.ini
RUN sed -i "s/\(post_max_size *= *\).*/\1100M/" /etc/php/7.3/fpm/php.ini
RUN sed -i "s/\(^.*max_input_vars *= *\).*/max_input_vars = 10000/" /etc/php/7.3/fpm/php.ini
RUN sed -i "s/\(pm.max_children = 5\).*/\pm.max_children = 50/" /etc/php/7.3/fpm/pool.d/www.conf
RUN sed -i "s/\(pm.max_spare_servers = 3\).*/\pm.max_spare_servers = 10/" /etc/php/7.3/fpm/pool.d/www.conf
RUN echo "xdebug.default_enable=1" >> /etc/php/7.3/fpm/conf.d/20-xdebug.ini
RUN echo "xdebug.profiler_output_dir=/var/www/cachegrind/" >> /etc/php/7.3/fpm/conf.d/20-xdebug.ini
RUN echo "xdebug.profiler_enable_trigger=1" >> /etc/php/7.3/fpm/conf.d/20-xdebug.ini
RUN echo "xdebug.profiler_output_name= cachegrind.out" >> /etc/php/7.3/fpm/conf.d/20-xdebug.ini
RUN mkdir /root/.ssh/
ADD default.conf /etc/nginx/conf.d/default.conf
ADD default.conf /etc/nginx/sites-enabled/default
ADD default.conf /etc/nginx/sites-available/default
EXPOSE 22 443 80
WORKDIR /var/www/
ENTRYPOINT ["docker-entrypoint"]
CMD ["nginx", "-g", "daemon off;"]
and also content of my docker compose file which i use when I build in my ci/cd:
version: '3'
services:
worker:
image: registry.gitlab.com/insuretech1/backend:develop
ports:
- 0.0.0.0:80:80
depends_on:
- mysql
deploy:
mode: replicated
replicas: 3
# service resource management
resources:
# Hard limit - Docker does not allow to allocate more
limits:
cpus: '0.25'
memory: 512M
# Soft limit - Docker makes best effort to return to it
reservations:
cpus: '0.25'
memory: 256M
# service restart policy
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 3
window: 120s
# service update configuration
update_config:
parallelism: 1
delay: 10s
failure_action: continue
monitor: 60s
max_failure_ratio: 0.3
volumes:
- /var/www/backend:/var/www
mysql:
image: mariadb:10.4
ports:
- 0.0.0.0:3306:3306
environment:
MYSQL_ROOT_PASSWORD: root
volumes:
- /opt/mysql_data:/var/lib/mysql
deploy:
placement:
constraints: [node.role == manager]
redis:
image: redis
deploy:
placement:
constraints: [node.role == manager]
the issue is I can not see my last changes of my commit in my server (I mean i should manually git pull to fetch last changes), is there anything wrong?
first of all in your Dockerfile you should copy content of directory into docker container. that will help you keep git changes with your container, for example:
COPY . /var/www/
and other problem is in your docker compose file you have:
volumes:
- /var/www/backend:/var/www
this will override changes that git made on your container and that is why you are not able to see git changes.

Trigger after merge successful?

I have this .gitlab-ci.yml script --- it triggers if I push directly on the master. How can I trigger this if there's a successful merge on the master branch?
before_script:
- apt-get update -qq
- apt-get install -qq git
# Setup SSH deploy keys
- 'which ssh-agent || ( apt-get install -qq openssh-client )'
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
- mkdir -p ~/.ssh
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
deploy_staging:
type: deploy
environment:
name: staging
url: domain.com
script:
- ssh user#domain.com "cd /server/directory && git fetch && git pull origin master && exit"
only:
- master
Like the reference notes
test:
stage: test
script: ./test
only:
- merge_requests
see https://docs.gitlab.com/ee/ci/merge_request_pipelines/index.html

Can Travis CI cache docker images?

Is it possible to add a setting to cache my docker image anywhere in the travis configuration ? Mine is a bigger docker image and it takes a while for it to download.
Any suggestions ?
Simplest solution today (October 2019) is to add the following to .travis.yml:
cache:
directories:
- docker_images
before_install:
- docker load -i docker_images/images.tar || true
before_cache:
- docker save -o docker_images/images.tar $(docker images -a -q)
See Caching Docker Images on Build #5358
for the answer(s). For Docker 1.12 available now on Travis, it is recommended to manually cache the images. For the Docker 1.13, you could use its --cache-from when it is on Travis.
Save:
before_cache:
# Save tagged docker images
- >
mkdir -p $HOME/docker && docker images -a --filter='dangling=false' --format '{{.Repository}}:{{.Tag}} {{.ID}}'
| xargs -n 2 -t sh -c 'test -e $HOME/docker/$1.tar.gz || docker save $0 | gzip -2 > $HOME/docker/$1.tar.gz'
Load:
before_install:
# Load cached docker images
- if [[ -d $HOME/docker ]]; then ls $HOME/docker/*.tar.gz | xargs -I {file} sh -c "zcat {file} | docker load"; fi
Also need to declare a cache folder:
cache:
bundler: true
directories:
- $HOME/docker
Docker images are not recommended to be cached regard to Travis document here
https://docs.travis-ci.com/user/caching/#things-not-to-cache
I just found the following approach as discussed in this article.
services:
- docker
before_script:
- docker pull myorg/myimage || true
script:
- docker build --pull --cache-from myorg/myimage --tag myorg/myimage .
- docker run myorg/myimage
after_script:
- docker images
before_deploy:
- echo "$DOCKER_PASSWORD" | docker login -u "$DOCKER_USERNAME" --password-stdin
deploy:
provider: script
script: docker push myorg/myimage
on:
branch: master
This works for me:
Update the desired docker image name instead of <IMAGE_NAME_HERE> (3 places).
You can also use the same configuration for multiple images, docker save can handle multiple images, just make sure to pull them before trying to save them.
services:
- docker
cache:
directories:
- docker-cache
before_script:
- |
filename=docker-cache/saved_images.tar
if [[ -f "$filename" ]]; then docker load < "$filename"; fi
mkdir -p docker-cache
docker pull <IMAGE_NAME_HERE>
docker save -o "$filename" <IMAGE_NAME_HERE>
script:
- docker run <IMAGE_NAME_HERE>...

Resources