I'm trying to deploy a java maven project on aws with Gitlab CI/CD.
This is my .gitlab-ci.yml
image: maven:3-jdk-8
services:
- docker:dind
stages:
- test
- build
- deploy
maven-test:
stage: test
script:
- echo "Test stage"
- mvn clean validate compile test -B
maven-build:
stage: build
script:
- echo "Build stage"
- mvn install -B -DskipTests
artifacts:
paths:
- ./target/*.jar
maven-deploy:
stage: deploy
script:
- echo "Deploy stage"
- scp -v -o StrictHostKeyChecking=no -I "mykey.pem" ./target/*.jar ubuntu#xxxxxxx.com:*.jar
when: manual
If I execute the scp command on a terminal in my pc then the jar is uploaded in aws ec2 instance while in gitlab I have errors and the jar is not uploaded.
This is my first approach with Gitlab CI and aws, so can someone explain step by step what I need to do to deploy the project in aws ec2 instance with Gitlab CI?
Thanks!
Since you have not posted much about your problem nor did you post the error I will just suggest a few things to look at:
From a GitLab perspective:
Are you sure that the "mykey.pem" is available within the repository when running that command(maven-deploy) on the the gitlab-runner.?
Also are you sure that you are using a docker gitlab-runner, if you are not then you can't use the image: directive and therefore it might not not have mvn/scp locally.
You might want to look into the dependencies directive and ensure you make that artifact available in next task. This should be done by default!
From an AWS perspective:
Make sure that the ubuntu target machine/server has port 22 exposed to the EC2 machine running the gitlab-runner.
Edit:
If the error you are receiving is with the pem files permissions then take a look at this resolution for AWS EC2 pem file issue. Another similar resolution is here.
Seems like if you put chmod 400 mykey.pem before the scp it might fix your problem.
Related
I'm trying to deploy & run a java spring boot application using github actions to a AWS Ec2 Instance. The application properties file of spring boot application points to environment variables where are present in the AWS Ec2 Instance. However, these environment variables are not available when the github action runs and so the execution of the jar fails with a null pointer exception.
What is the correct way to deploy a Spring boot (Non Docker Application) to Self hosted Ec2 Server? Can I do it without needing AWS Code Pipeline or AWS Elastic Beanstalk?
How do we read Ec2 instance environment variables while using github actions.
Thanks.
Sample Workflow file:
jobs:
build:
runs-on: [self-hosted]
steps:
- uses: actions/checkout#v3
- name: Set up JDK 11
uses: actions/setup-java#v3
with:
java-version: "11"
distribution: "temurin"
cache: maven
- name: Build with Maven
run: mvn clean -B package
deploy:
runs-on: [self-hosted]
needs: build
steps:
- name: Run Script file
working-directory: ./
run: |
chmod +x ./script.sh
./script.sh
shell: bash
// script.sh - Try to print the env variables inside ec2.
#!/bin/bash
whoami
printenv
I use Laravel Vapor for deploying our microservices based on Laravel. This works very good so far, if the app with their dependencies is not too large. But if it is then it gets a little bit tricky.
Vapor provides a Docker runtime for this case where you are able to deploy apps up to 10GB size.
For local development we usually use Laradock.io because its easy and flexible.
That means if we deploy from our local environment it easy to enter the workspace container and and run the vapor deploy commands. After enabling Docker Client for the workspace container it works with the vapor Docker runtime properly.
But now we integrated the deployment process into Gitlab CI Pipeline. That works very well for our small services with Vapor PHP runtime.
But for the Docker runtime I desperate on the CI deployment.
The docker runtime needs an installed docker instance where vapor will be invoked. That means in the Gitlab-ci.yml I have to add an image with installed Docker and PHP to invoke the Vapor scripts.
So I created an docker image base on the laradock workspace container but the Gitlab-runner exits always with the error message no docker deamon is available.
This is the related part of my GitLab-CI yml (the image is only local available):
testing:
image:
name: lexitaldev/vapor-docker-deploy:latest
pull_policy: never
securityContext:
privileged: true
environment: testing
stage: deploy
only:
- test
script:
- composer install
- php vendor/bin/vapor deploy test
This is the specific output:
Error Output:
================
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
I've tried to use the standard 'laravelphp/vapor:php80' image and install docker over the script section as well.
before_script:
- apk add docker
- addgroup root docker
But nothing helped. It seems to be there is a problem with the docker.sock.
Did anybody managed to add Vapor Docker Runtime deployment to CI scripts?
Best,
Michael
I would like to tell you, that you only need to add the Service: dind, but after you do that, it will throw an error, related to the image that Gitlab create for your pipelines. So you need to create a runner with volumes, privileged flag, and tags.
I did it, using gitlab-runner on my machine.
sudo gitlab-runner register -n \
--url {{ your_url }} \
--registration-token {{your_token}} \
--executor docker \
--description "{{ Describe your runner }}" \
--docker-image "docker:20.10.12-alpine3.15" \
--docker-privileged \
--docker-volumes="/certs/client" \
--docker-volumes="cache" \
--docker-volumes="/var/run/docker.sock:/var/run/docker.sock"
--tag-list {{ a_tag_for_your_pipeline }}
Once you did that, you would need to use a docker stable version in your gitlab-ci.yml file. For some reason, it doesn't work when I was trying to use version 20 or latest
image: docker:stable
services:
- name: docker:stable:dind
before_script:
- echo $CI_JOB_TOKEN | docker login $CI_REGISTRY -u $CI_REGISTRY_USER --password-stdin
build:
tags:
- {{the tag you defined in your runner}}
variables:
IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG
script:
- echo $IMAGE_TAG
- docker build -t $CI_REGISTRY_IMAGE -f {{your Dockerfile}} .
- docker push $CI_REGISTRY_IMAGE
All the variables are previously defined in Gitlab, so don't worry, you can "copy & paste". Also, I added some advices that Gitlab mention on its documentation when you need to register your Docker container in Gitlab container.
This my .travis.yml file. I am trying to automate deployment to aws-codedeploy.
language: node_js
node_js:
- 7.10.0
services:
- mongodb
env:
- PORT=6655 IP="localhost" NODE_ENV="test"
script:
- npm start &
- sleep 25
- npm test
deploy:
provider: codedeploy
access_key_id:
secure: $Access_Key_Id
secret_access_key:
secure: $Access_Key_Secret
revision_type: github
application: Blog
deployment_group: Ayush-Bahuguna
region: us-east-2
after_deploy:
- "./build.sh"
Here build.sh is a shell script that generates the build files
cd /var/www/cms
sudo yarn install
npm run build-prod
And here is .gitignore file
node_modules/
client/dashboard/dist/
client/blog/dist/
The issue is that, even though travis-ci build succeeds, and after_deploy runs successfully, no build files are generated on the aws ec2 instance where my project is hosted.
Are you able to see any deployment created on your AWS CodeDeploy console? And are your able to see the deployment status? If there is a deployment created, but failed, you can try to see the reason why it failed. Even though the deployment succeeded, it doesn't equal to all instances are deployed depends on the deployment configuration: http://docs.aws.amazon.com/codedeploy/latest/userguide/deployment-configurations.html.
Thanks,
Binbin
I'm trying to use bitbucket pipelines to do deploys in heroku and I created a simple web application using spring-boot.
I'm wondering if any of you have ideas about what is need it in "bitbucket-pipelines.yml" file? This is my first time using heroku with bitbucket pipeline and I'm very lost.
My file looks like this one:
image: maven:3.3.9
clone:
depth: full
pipelines:
default:
- step:
script: # Modify the commands below to build your repository.
- mvn -B clean install -P heroku # -B batch mode makes Maven less verbose
- git push https://heroku:$HEROKU_API_KEY#git.heroku.com/$HEROKU_APP_NAME.git HEAD
I would recommend the following steps:
Follow Step 1 mentioned in Deploy to Heroku in Bitbucket Pipelines' documentation.
Please edit your "bitbucket-pipelines.yml" file with the following content:
image: maven:3.3.9
clone:
depth: full
pipelines:
default:
- step:
script:
- git push https://heroku:$HEROKU_API_KEY#git.heroku.com/$HEROKU_APP_NAME.git HEAD
- mvn clean package
- kill -9 $(lsof -t -i:<your_app_port_number> -sTCP:LISTEN)
- java -jar target/<your-app-name>.jar &
NOTE:
Replace < your_app_port_number > and < your-app-name > with appropriate values.
Remember to check your "bitbucket-pipelines.yml" file with the online validator.
Here is my bitbucket-pipelines.yml where spring profile is specified.
image: maven:3.3.9
clone:
depth: full
pipelines:
default:
- step:
name: Deploy to Heroku
deployment: test # set to test, staging or production
script:
- mvn -B clean package -Dspring.profiles.active=prod # -B batch mode makes Maven less verbose
- git push https://heroku:$HEROKU_API_KEY#git.heroku.com/$HEROKU_APP_NAME.git HEAD
I have a Profile with:
web: java -Dserver.port=$PORT $JAVA_OPTS -Dspring.profiles.active=prod -jar target/my-api-1.0.0-SNAPSHOT.jar
Make sure you have configured $HEROKU_API_KEY and $HEROKU_APP_NAME on BitBucket like:
I am using a CI tool called Drone(drone.io). So i really want to do some integration tests with it. What i want is Drone to start my application container on some port on the drone host and then I would be able to run integration tests against it. For example in .drone.yml file:
build:
image: python3.5-uwsgi
pull: true
auth_config:
username: some_user
password: some_password
email: email
commands:
- pip install --user --no-cache-dir -r requirements.txt
- python manage.py integration_test -h 127.0.0.1:5000
# this should send various requests to 127.0.0.1:5000
# to test my application's behaviour
compose:
my_application:
# build and run a container based on dockerfile in local repo on port 5000
publish:
deploy:
Drone 0.4 can't start service from your Dockerfile, if you want start docker container, you should build it before, outside this build, and push to dockerhub or your own registry and put this in compose section, see http://readme.drone.io/usage/services/#images:bfc9941b6b6fd7b4ef09dd0ccd08af0c
You can also start your application in build, nohup python manage.py server -h 127.0.0.1:5000 & before you running your integration tests. Make sure that your application is started and listening 5000 port, before you run integration_test.
I recommend you use drone 0.5 with pipelines, you can build docker image and push that to registry before build, and use that as service inside your build.