CI/CD using Github Actions and AWS EC2 instance - amazon-ec2

I have a dockerised fastapi app whith depends on mysql and redis which are all configured in docker-compose.yml. Want to implement a CI/CD using github actions and AWS EC2 instance. My EC2 instance has docker and docker-compose installed. Here are my questions.
What do I do to run the tests that depends on the test db?
How do I implement CD from github actions and AWS EC2 instance?
I might not be clear so please ask some questions for clarification. Thank you.
name: backend-api
on:
push:
branches: ["main"]
pull_request:
branches: ["main"]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v3
- name: Set up Python 3.10
uses: actions/setup-python#v3
with:
python-version: "3.10"
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install flake8 pytest
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
- name: Lint with flake8
run: |
# stop the build if there are Python syntax errors or undefined names
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
- name: Create .env file for configuration settings.
uses: SpicyPizza/create-envfile#v1.3
with:
envkey_APP_ENV: ${{secrets.APP_ENV}}
envkey_APP_HOST: ${{secrets.APP_HOST}}
envkey_MYSQL_USER: ${{secrets.APP_ENV}}
envkey_PROD_BASE_URL: ${{secrets.MYSQL_USER}}
envkey_DEV_BASE_URL: ${{secrets.APP_ENV}}
envkey_MYSQL_ROOT_PASSWORD: ${{secrets.MYSQL_ROOT_PASSWORD}}
envkey_MYSQL_DATABASE: ${{secrets.MYSQL_DATABASE}}
envkey_PRODUCTION_DB_URI: ${{secrets.PRODUCTION_DB_URI}}
envkey_TEST_DB_URI: ${{secrets.TEST_DB_URI}}
envkey_BASE_URL: ${{secrets.BASE_URL}}
envkey_WALLET_PROVIDER_ACCESS_TOKEN: ${{secrets.WALLET_PROVIDER_ACCESS_TOKEN}}
envkey_S3_BUCKET_NAME: ${{secrets.S3_BUCKET_NAME}}
envkey_S3_ACCESS_SECRET: ${{secrets.S3_ACCESS_SECRET}}
envkey_S3_ACCESS_KEY: ${{secrets.S3_ACCESS_KEY}}
envkey_S3_BUCKET_REGION: ${{secrets.S3_BUCKET_REGION}}
envkey_JWT_SECRET_KEY: ${{secrets.JWT_SECRET_KEY}}
envkey_ETHERSCAN_API_URL: ${{secrets.ETHERSCAN_API_URL}}
envkey_BLOCKCHAIN_API_URL: ${{secrets.BLOCKCHAIN_API_URL}}
envkey_WALLET_PROVIDER_BASE_URL: ${{secrets.WALLET_PROVIDER_BASE_URL}}
envkey_STRATEGY_PROVIDER_BASE_URL: ${{secrets.STRATEGY_PROVIDER_BASE_URL}}
envkey_INDEX_PROVIDER_BASE_URL: ${{secrets.INDEX_PROVIDER_BASE_URL}}
- name: Running Tests with pytest
run: |
pytest
Deploy:
needs: build
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Git pull
env:
AWS_EC2_PEM: ${{ secrets.AWS_EC2_PEM }}
AWS_EC2_PUBLIC_IP: ${{ secrets.AWS_EC2_PUBLIC_IP }}
AWS_EC2_USERNAME: ${{ secrets.AWS_EC2_USERNAME }}
run: |
pwd
echo "$AWS_EC2_PEM" > private_key && chmod 600 private_key
ssh -o StrictHostKeyChecking=no -i private_key ${AWS_EC2_USERNAME}#${AWS_EC2_PUBLIC_IP}
git checkout main &&
git fetch --all &&
git reset --hard origin/main &&
git pull origin main &&
touch .env
docker-compose up -d --build

Related

How to bump version with GitHub Actions

I want to bump the version by running my shell script. I am manually triggering the GitHub Actions with inputs. I am using GitHub API to get the latest version and I want to run my shell script with the version taken from GitHub API.
I am able to achieve everything. It's just that I am not able to get the version output.
This is my GitHub Workflow:
#############################################################################
# GitHub Action to bump release version
#
#############################################################################
name: "Bump version and Update Milestone"
on:
workflow_dispatch:
inputs:
version:
description: 'New Version'
required: true
default: 'warning'
permissions:
contents: write
pull-requests: write
jobs:
printInputs:
runs-on: ubuntu-latest
steps:
- run: |
echo "Latest Version: ${{ github.event.inputs.version }}"
bump:
name: Bump version
runs-on: ubuntu-latest
steps:
- name: Checkout the latest code
uses: actions/checkout#v2
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Get the version
id: get_version
run: |
VERSION=$(curl -s https://api.github.com/repos/checkstyle/checkstyle/releases/latest \
| jq ".tag_name")
echo VERSION="$VERSION"
- name: Modify File
run: |
./.ci/bump-version.sh ${{ steps.get_version.VERSION }}
- name: Push commit
run: |
git config --global user.name 'github-actions[bot]'
git config --global user.email 'github-actions[bot]#users.noreply.github.com'
git commit -am "minor: Bump release version to ${{ steps.get_version.VERSION }}"
git push
- name: GitHub Milestone
run: |
.ci/update-github-milestone.sh
This is my shell script:
#!/bin/bash
set -e
VERSION=$1
echo VERSION="$VERSION"
echo "bump version in pom.xml"
mvn versions:set -DnewVersion="$VERSION" && mvn versions:commit
And these is my tested actions where I want help:
Action link: https://github.com/Rahulkhinchi03/checkstyle/runs/7785606554?check_suite_focus=true
To transfer an output from one step to another, you need to use a workflow command:
echo "::set-output name=VERSION::$VERSION"
and then reference it like this in a later step via the outputs object:
run: |
./.ci/bump-version.sh ${{ steps.get_version.outputs.VERSION }}

How to assign environment variables from a file in GitHub actions?

I'm currently working on a Django project. I have create a GitHub action to run python manage.py test command everytime I push the code to main branch.
The problem here is, I have many environment variables in my project. I can't set env variables as GitHub secrets for each variables.
I have a env.dev file in my repository. What I need to do is, everytime I push the code, it needs to assign environment variables by reading it from the env.dev file.
Is there any way to do this?
This is my django.yml file used for GitHub actions.
name: Django CI
on:
pull_request:
branches: [ "main"]
jobs:
build:
runs-on: ubuntu-latest
strategy:
max-parallel: 4
matrix:
python-version: [3.9]
steps:
- uses: actions/checkout#v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python#v3
with:
python-version: ${{ matrix.python-version }}
- name: Install Dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Run Tests
run: |
python manage.py test
As noted in "How do I use an env file with GitHub Actions?", you do have actions like Dotenv Action which do read a .env file from the root of this repo and provides environment variables to build steps.
But those would not be GitHub secret though.
There is an API to create secrets, which means, if your .env content does not change too often, you can script the secret creation step.

GitActions workflow: cache is not passing between jobs

What I am trying to do is basically an assembly line: every job must build/maniuplate the provided files, the cache, that is passed to the following job.
This is my workflow:
jobs:
Job1_-_Prepearing_enviornment:
runs-on: ubuntu-latest
steps:
- name: Get date for artifacts
id: date
run: echo "::set-output name=date::$(date +'%Y%m%d_%H%M%S')"
- name: Extract branch name
shell: bash
run: echo "##[set-output name=branch;]$(echo ${GITHUB_REF#refs/heads/})"
id: extract_branch
- name: Generating build ID
run: echo "${{ steps.extract_branch.outputs.branch }}-${{ steps.date.outputs.date }}"
outputs:
buildID: "${{ steps.extract_branch.outputs.branch }}-${{ steps.date.outputs.date }}"
# In the last 3 steps I am generating a build specific key in order to not be confused with other running workflows
Job2_-_Downloading_Sources:
runs-on: ubuntu-latest
needs: [Job1_-_Prepearing_enviornment]
steps:
# Here I download a global cache if it's existing (I am generating the global cache in a job not copypasted here)
- uses: actions/download-artifact#v2
with:
name: global-cache
continue-on-error: true
- uses: actions/checkout#v3
- name: Downloading sources
run: |
git pull
git submodule init
git submodule update
sudo add-apt-repository ppa:alexlarsson/flatpak
sudo apt-get update
sudo apt install flatpak flatpak-builder p7zip-full
sudo flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo
sudo flatpak install -y org.kde.Sdk//5.15-21.08 org.kde.Platform//5.15-21.08 io.qt.qtwebengine.BaseApp/x86_64/5.15-21.08 org.freedesktop.Sdk.Extension.llvm13
sudo flatpak-builder --download-only --user --repo=local retrodeck-flatpak net.retrodeck.retrodeck.yml
# Here I am using the key generated in the Job1 to save the cache
- name: Generating build specific cache
id: intenral-cache
uses: actions/cache#v2
with:
path: /home/runner/work/RetroDECK
key: ${{ needs.Job1_-_Prepearing_enviornment.outputs.buildID }}
Job3_-_Building_part_1:
runs-on: ubuntu-latest
needs: [Job1_-_Prepearing_enviornment, Job2_-_Downloading_Sources]
steps:
# Here I am supposed to get the internal cache calling its key but this process is not working and the build stops
- name: cache-cooker
id: internal-cache
uses: actions/cache#v2
with:
path: /home/runner/work/RetroDECK
key: ${{ needs.Job1_-_Prepearing_enviornment.outputs.buildID }}
- name: Initializing enviornment
run: |
git submodule init
git submodule update
sudo add-apt-repository ppa:alexlarsson/flatpak
sudo apt-get update
sudo apt install flatpak flatpak-builder p7zip-full
sudo flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo
sudo flatpak install -y org.kde.Sdk//5.15-21.08 org.kde.Platform//5.15-21.08 io.qt.qtwebengine.BaseApp/x86_64/5.15-21.08 org.freedesktop.Sdk.Extension.llvm13
- name: Bulding part 1 - Until melonds
run: |
cd /home/runner/work/RetroDECK/RetroDECK
sudo flatpak-builder --build-only --stop-at=melonds --user --repo=local retrodeck-flatpak net.retrodeck.retrodeck.yml
As you can see from the comments in the code, the job3 cannot get the cache and I get this output from the GitAction log:
Run actions/cache#v2
with:
path: /home/runner/work/RetroDECK
key: cooker-20220502_115338
Cache not found for input keys: cooker-20220502_115338
Then the job fails because it cannot find the needed files.
I am trying to use the Artifacts instead of the cache but it's too slow by zipping, uploading, downloading and unzipping, so maybe the cache it's a more straightforward way.
What am I doing wrong?
How can I fix this?
Thanks in advance.

Using github secrets in another non-workflow yaml file

Is it possible to access a github secret in a yaml file that's not a workflow or an action yaml file?
For example, I've saved in github the environment secret INFURA_RINKEBY_WSS and I attempt to access it in the following yaml config file for my program.
type: EndpointList
endpoints:
- type: RPCEndpoint
chain_id: 1
network: rinkeby
provider: Infura
url: ${{ secrets.INFURA_RINKEBY_WSS}}
explorer: https://etherscan.io
However, the INFURA_RINKEBY_WSS environment variable I've set in github isn't accessed yet by my yaml config file.
The following is my main.yaml github workflow:
name: Report to eth/usd on rinkeby w/ pytelliot
on: push
jobs:
build:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python-version: ["3.9"]
steps:
- uses: actions/checkout#v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python#v2
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install telliot-feed-examples
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
- name: Move pre-made pytelliot configs to home directory
run: |
cp -r ./config ~/
- name: report :)
run: telliot-examples --legacy-id 1 report --submit-once
env:
PK: ${{ secrets.PK }}
INFURA_RINKEBY_WSS: ${{ secrets.INFURA_RINKEBY_WSS }}
Thanks!

Is it possible to use GitHub secrets inside my shell file?

This is my simple Action on my GitHub repo:
name: CI
on:
push:
branches: [ main ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Get /my_account/my_infra
run: |
sudo mkdir /my_account
sudo chmod -R 777 /my_account
cd /my_account
git clone https://github.com/my_account/my_infra
- name: Get /my_account/my_repo
run: |
cd /my_account
git clone https://github.com/my_account/my_repo
- name: Run my build script
run: |
cd /my_account/my_infra
./build.sh /my_account/my_repo
Since GitHub does not provide a way to reuse actions across multiple similar repos, I came up with the idea of creating a base repo, then download that base alongside the current repo, then run a custom shell script from that base repo, passing my current repo as a parameter.
This works perfect. This way I can reuse my base repo across many similar repositories. And I can reuse near 500 lines of build script instead of repeating myself for 50 repositors (which means 25000 lines of CI/CD code).
However, now I need to access some resources (like login into my docker hub account) to pull and push stuff.
Is it possible to use GitHub secrects in my build.sh?
When you set env in your workflow, doc here, they are set as environment variables in your containerised workflow.
This means that if you set a secret in your repository, can be found under settings=> secrets and then assign it to an env in your workflow, they can then be accessed in your build.sh
example:
name: CI
on:
push:
branches: [ main ]
env:
super_secret: ${{ secrets.my_secret }}
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Get /my_account/my_infra
run: |
sudo mkdir /my_account
sudo chmod -R 777 /my_account
cd /my_account
git clone https://github.com/my_account/my_infra
- name: Get /my_account/my_repo
run: |
cd /my_account
git clone https://github.com/my_account/my_repo
- name: Run my build script
run: |
cd /my_account/my_infra
./build.sh /my_account/my_repo
In this case your build.sh can do something like:
#!/bin/bash
npm run build $super_secret
Yes, you just need to assign them to a variable, like
env:
ACCESS_TOKEN: ${{ secrets.ACCESS_TOKEN }}
run: build.sh
Then you can refer to ACCESS_TOKEN variable in the shell script.

Resources