Say I'm working on a web project that runs gitlab-ci shell runner on my own ci server to build docker and deploy it to heroku, and I've gone through some docs from both gitlab and heroku like gitlab-ci: using docker build and heroku:Build and Deploy with Docker. Can I deploy the docker project without using heroku-docker plugin, which seems not so flexible to me? However I tried, the following approach build succeeded in deploying to heroku, but the app crash. Heroku logs says start script is missing in package.json, but since I'm deploying docker project, I couldn't do "start": "docker-compose up" there, could I?
#.gitlab-ci.yml
stages:
- deploy
before_script:
- npm install
- bower install
dev:
stage: deploy
script:
- docker-compose run nginx run-test
- gem install dpl
- dpl --provider=heroku --app=xixi-web-dev --api-key=$HEROKU_API_KEY
only:
- dev
# docker-compose.yml
app:
build: .
volumes:
- .:/code:ro
expose:
- "3000"
working_dir: /code
command: pm2 start app.dev.json5
nginx:
build: ./setup/nginx
restart: always
volumes:
- ./setup/nginx/sites-enabled:/etc/nginx/sites-enabled:ro
- ./dist:/var/app:ro
ports:
- "$PORT:80"
links:
- app
I don't want to use heroku docker plugin, because it seems less flexible, I can't create a app.json because I don't want to use an existing docker image for my app. Instead, I define custom Dockerfiles for app and nginx used in docker-compose.yml
Now it seems that heroku wouldn't detect my project as a docker project unless I deploy it by using heroku docker plugin, but as I mentioned above, I can't do that. Then is there any docs I'm missing on heroku or gitlab could help me out? Or do you have any idea that might be helpful? Thanks a lot!
OK, seems that heroku docker:release is required. I ended up installing heroku cli and heroku docker plugin on my CI server and use heroku docker:release --app app to release my app
Related
I use Laravel Vapor for deploying our microservices based on Laravel. This works very good so far, if the app with their dependencies is not too large. But if it is then it gets a little bit tricky.
Vapor provides a Docker runtime for this case where you are able to deploy apps up to 10GB size.
For local development we usually use Laradock.io because its easy and flexible.
That means if we deploy from our local environment it easy to enter the workspace container and and run the vapor deploy commands. After enabling Docker Client for the workspace container it works with the vapor Docker runtime properly.
But now we integrated the deployment process into Gitlab CI Pipeline. That works very well for our small services with Vapor PHP runtime.
But for the Docker runtime I desperate on the CI deployment.
The docker runtime needs an installed docker instance where vapor will be invoked. That means in the Gitlab-ci.yml I have to add an image with installed Docker and PHP to invoke the Vapor scripts.
So I created an docker image base on the laradock workspace container but the Gitlab-runner exits always with the error message no docker deamon is available.
This is the related part of my GitLab-CI yml (the image is only local available):
testing:
image:
name: lexitaldev/vapor-docker-deploy:latest
pull_policy: never
securityContext:
privileged: true
environment: testing
stage: deploy
only:
- test
script:
- composer install
- php vendor/bin/vapor deploy test
This is the specific output:
Error Output:
================
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
I've tried to use the standard 'laravelphp/vapor:php80' image and install docker over the script section as well.
before_script:
- apk add docker
- addgroup root docker
But nothing helped. It seems to be there is a problem with the docker.sock.
Did anybody managed to add Vapor Docker Runtime deployment to CI scripts?
Best,
Michael
I would like to tell you, that you only need to add the Service: dind, but after you do that, it will throw an error, related to the image that Gitlab create for your pipelines. So you need to create a runner with volumes, privileged flag, and tags.
I did it, using gitlab-runner on my machine.
sudo gitlab-runner register -n \
--url {{ your_url }} \
--registration-token {{your_token}} \
--executor docker \
--description "{{ Describe your runner }}" \
--docker-image "docker:20.10.12-alpine3.15" \
--docker-privileged \
--docker-volumes="/certs/client" \
--docker-volumes="cache" \
--docker-volumes="/var/run/docker.sock:/var/run/docker.sock"
--tag-list {{ a_tag_for_your_pipeline }}
Once you did that, you would need to use a docker stable version in your gitlab-ci.yml file. For some reason, it doesn't work when I was trying to use version 20 or latest
image: docker:stable
services:
- name: docker:stable:dind
before_script:
- echo $CI_JOB_TOKEN | docker login $CI_REGISTRY -u $CI_REGISTRY_USER --password-stdin
build:
tags:
- {{the tag you defined in your runner}}
variables:
IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG
script:
- echo $IMAGE_TAG
- docker build -t $CI_REGISTRY_IMAGE -f {{your Dockerfile}} .
- docker push $CI_REGISTRY_IMAGE
All the variables are previously defined in Gitlab, so don't worry, you can "copy & paste". Also, I added some advices that Gitlab mention on its documentation when you need to register your Docker container in Gitlab container.
Either I'm doing something wrong or Heroku is messing up. Heroku supports targeting a particular stage in a Dockerfile. I have a multistage Dockerfile but Heroku is not respecting the build.docker.release.target in my heroku.yml. For what it's worth, targeting works fine with docker-compose.yml.
I'm trying to keep dev and prod in the same Dockerfile. Essentially dev and prod are forked from base. I could flesh it out more, but the stages are:
FROM python:3.10.0-slim-buster AS venv
...
FROM python:3.10.0-slim-buster as base
...
FROM base AS dev
...
ENTRYPOINT ["entrypoint.dev.sh"]
FROM base AS prod
...
ENTRYPOINT ["entrypoint.prod.sh"]
My heroku.yml specifically targets the prod stage:
setup:
addons:
- plan: heroku-postgresql
as: DATABASE
build:
docker:
release:
dockerfile: image/app/Dockerfile
target: prod
web: image/app/Dockerfile
config:
DJANGO_ENV: production
release:
image: web
command:
- ./deployment-tasks.sh
run:
web: gunicorn server.wsgi:application --bind 0.0.0.0:$PORT --log-level debug --access-logfile - --error-logfile -
However, Heroku builds all the stages, seems like it just runs down the Dockerfile till the end. The Heroku build logs show that first dev follows base
and then prod follows dev
I would expect it to jump from base to prod, skipping dev.
Is this an issue on my side or Heroku's?
I haven't tested this with heroku.yml because I've moved to GitHub Actions but I believe the error was having prod come after dev. Apparently the --target flag in docker build means it will stop at that stage, so it will run everything before it.
I'm new to Elixir and Phoenix, and having to work in CI/CD environment I'm trying to figure out how to use Phoenix with Docker.
I've tried various tutorials and videos out there, many of them doesn't work, but those who do work, they have the same result.
Phoenix server doesn't seems to find some resources (the assets folder?).
But inside my Dockerfile I'm copying the entire app folder, and I can confirm that /assets is inside the container by attaching to it.
Dockerfile:
FROM elixir:alpine
RUN apk add --no-cache build-base git
WORKDIR /app
RUN mix local.hex --force && \
mix local.rebar --force
COPY . .
RUN mix do deps.get, deps.compile
CMD ["mix", "phx.server"]
Docker-compose
version: '3.6'
services:
db:
environment:
PGDATA: /var/lib/postgresql/data/pgdata
POSTGRES_PASSWORD: postgres
POSTGRES_USER: postgres
POSTGRES_HOST_AUTH_METHOD: trust
image: 'postgres:11-alpine'
restart: always
volumes:
- 'pgdata:/var/lib/postgresql/data'
web:
build: .
depends_on:
- db
environment:
MIX_ENV: dev
env_file:
- .env
ports:
- '4000:4000'
volumes:
- .:/app
volumes:
pgdata:
Steps I'm doing to create the containers and running the server:
docker-compose build
docker-compose run web mix ecto.create
docker-compose up
The database is created successfully in the db container.
What can be happening here?
Sorry if it's straightforward, I don't use Docker for a while and I still didn't understood Phoenix boilerplate completely.
If you know some good resources about Docker and CI/CD pipelines with Phoenix, I also appreciate so I can study it.
You also need to build the assets. npm install --prefix assets This needs to be done after after mix deps.get but can be done after the mix deps.compile which isn't really needed. You can start the server after mix deps.get and it will compile the deps and your app automatically.
I'm having a problem installing laravel through a dockerfile. I'm using docker-compose that pulls a dockerfile where I basically have this:
FROM php:7.3-apache-stretch
*some apt-get and install composer*
WORKDIR /var/www
RUN composer create-project --prefer-dist laravel/laravel app
CMD apachectl -D FOREGROUND
but when I access the container and I will see the files that should have been created with the composer I see that it is empty even though I have seen the command executed in the container build.
The container is working perfectly and even I can access it ... only files that do not even appear.
If I run the composer command manually after the container is created the files appear.
In your Dockerfile, you used WORKDIR /var/www and then RUN composer create-project ... which makes composer create files under /var/www on the container file system.
In your docker-compose.yml file you used to start your container:
version: '3.7'
services:
app:
container_name: "app"
build:
context: ./docker
dockerfile: Dockerfile-app
ports:
- "80"
- "443"
restart: always
depends_on:
- db
volumes:
- ".:/var/www"
You are declaring a volume that will be mounted on that same location /var/www in your container.
What happens is that the volume content will take the place of whatever you had on /var/www in the container file system.
I suggest you read carefully the documentation regarding docker volumes, and more specifically the part titled Populate a volume using a container.
Now to move on, ask yourself why you needed that volume in the first place. Is it necessary to change files at run time ?
If not, just add your files at build time:
FROM php:7.3-apache-stretch
*some apt-get and install composer*
WORKDIR /var/www
RUN composer create-project --prefer-dist laravel/laravel app
COPY . /var/www
CMD apachectl -D FOREGROUND
and remove the volume for /var/www.
EDIT
Developing with the help of a Docker container
During development, you change php files on your docker host (assumed to be you development computer) and need to frequently test the result by testing your app served by the webserver from the docker container.
It would be cumbersome to have to rebuild a Docker image every time you need to test your app. The solution is to mount a volume so that the container can serve the files from your development computer:
FROM php:7.3-apache-stretch
*some apt-get and install composer*
WORKDIR /var/www
CMD apachectl -D FOREGROUND
and start it with:
version: '3.7'
services:
app:
container_name: "app"
build:
context: ./docker
dockerfile: Dockerfile-app
ports:
- "80"
- "443"
restart: always
depends_on:
- db
volumes:
- ".:/var/www"
...
When you need to run some commands within that container, just use docker exec:
docker-compose exec app composer create-project --prefer-dist laravel/laravel app
Producing project artifacts
Since what you will be deploying is not a zip/tar archive containing your source code and configurations but a docker image, you need to build the docker image you will use at deployment time.
Dockerfile for production
For production use, you want to have a Docker image which holds all required files and does not need any docker volume, excepted for holding data produced by users (uploaded files, database files, etc)
FROM php:7.3-apache-stretch
*some apt-get and install composer*
WORKDIR /var/www
COPY . /var/www
CMD apachectl -D FOREGROUND
Notice that there is no RUN composer create-project --prefer-dist laravel/laravel app in this Dockerfile. This is because this command is to initialise your project and this is a development time task, not a deployment time task.
You will also need a place to host your docker images (a Docker registry). You can deploy your own registry as a Docker container using the official registry image, or use the one provided by companies:
Gitlab.com - Gitlab Registry (free)
Docker.com - hub.docker.com (1 private image free)
Google.com - Google Container Registry
...
So you need to build a docker image, and then push that image on your registry. Best practice is to automate those tasks with the help of continuous integration tools such as Jenkins, Gitlab CI, Travis CI, Circle CI, Google Cloud Build ...
Your CI job will run the following commands:
git clone <url of you git repo> my_app
cd my_app
git checkout <some git tag>
docker build -t <registry>/<my_app>:<version>
docker login <registry> --user=<registry user> --password=<registry password>
docker push <registry>/<my_app>:<version>
Deploying your Docker image
Start you container with:
version: '3.7'
services:
app:
container_name: "app"
image: <registry>/<my_app>:<version>
ports:
- "80"
- "443"
restart: always
depends_on:
- db
...
Notice here that the docker-compose file does not build any image. For production it is a better practice to refer to an already built docker image (which has been deployed earlier on a staging environment for validation).
I have managed to succesfully run laravel test build on Gitlab CI using Gitlab Runner on digitaocean (With help from tutorial HOW TO: LARAVEL TESTING ON GITLAB CI WITH DOCKER)
Now I am wondering how I can deploy it after succesfull test.
This is my deploy process on my staging env:
cd MY_PROJECT_ROOT_DIR
git reset --hard HEAD
git checkout master
git pull
composer install
composer update
php artisan migrate
php artisan db:seed
How I can manage to include this deploy after test is done?
My configuration of GitLab Runner is the same as those files on this repo
This is the content of my .gitlab-ci.yml file:
before_script:
- bash .gitlab-ci.sh
variables:
MYSQL_DATABASE: laravel
MYSQL_ROOT_PASSWORD: secret
phpunit:php-laravel-env:mysql:
image: woohuiren/php-laravel-env:latest
services:
- mysql:latest
script:
- php vendor/bin/phpunit --colors
How I should change this file in order to execute deploy script after test passed?
You need to use "stages"
Basically you would update your current test setup to include a stage.
stages:
- test
- deploy
phpunit:php-laravel-env:mysql:
stage: test
image: woohuiren/php-laravel-env:latest
services: ...
deploy_my_site:
stage: deploy
...
These stages will get run in sequence by GitLab but will stop if there is any error.
If you are using Forge you could use the deploy stage to trigger a script to curl the forge deploy hook.