i'm setting up the Github Action, AWS EC2, CodeDeploy. All the configuration seems working well. But excepts one thing. I can not understand and how can i solve it. If someone have experiences about this please help me.
I'm using:
EC2 Rhel 8
Node project (VueJs framework)
This is my cicd.yml file
on:
push:
branches:
- paymentV2
name: Deploy VueJS to Amazon ECS
#on: [push]
jobs:
deploy:
name: Deploy
runs-on: ubuntu-latest
strategy:
matrix:
node-version: ['12.x']
appname: ['staging-aws-codedeploy']
deploy-group: ['staging']
repo: ['project/MyProject']
steps:
- uses: actions/checkout#v2
# Configure AWS credentials
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials#v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ap-southeast-1
# Deploy to AWS
- name: Deploy to AWS
run: |
aws deploy create-deployment \
--application-name ${{ matrix.appname }} \
--deployment-config-name CodeDeployDefault.OneAtATime \
--deployment-group-name ${{ matrix.deploy-group }} \
--file-exists-behavior OVERWRITE \
--description "GitHub Deployment for the ${{ matrix.appname }}-${{ github.sha }}" \
--github-location repository=${{ matrix.repo }},commitId=${{ github.sha }}
This is my appspec.yml
version: 0.0
os: linux
files:
- source: /
destination: /var/www/MyProject
hooks:
ApplicationStart:
- location: scripts/application_start.sh
timeout: 300
runas: root
#scripts/application_start.sh
#cd /var/www/MyProject
#npm run build
This is the log from Github action & CodeDeploy AWS
I've tried editing the Vision.vue file and created the pull request on Github. Everything was working well. But one thing i'm confusing is why the modified file is existed. Please refer the image below
=> What am i expected is the modified file shouldn't have existed. I thought that Github should be automatically run git pull to get all new source code.
I've some more research and found out --file-exists-behavior with OVERWRITE but it seems not working as i want.
https://docs.aws.amazon.com/cli/latest/reference/deploy/create-deployment.html
==> Once again, i have no experience about CD by Github action & CodeDeploy. Everyone please help me and advice me the right thing. Thank you so much.
After a period of learning, I understood that appspect and buildspec.yaml were just the way to build and deploy, but for the pull code, I used webhook (aws codebuild, AWS Code Pipeline, Github webhook) or schedule (crontab). And i've decided to user crontab for my project, scheduling to pull new source code every hours. Hope this sharing can help anyone. Tks
Related
I am new to deploying web apps.
I recently finished working on a project I wanted to deploy to Azure. I used VS to publish it an Azure App Service with a GitHub CI/CD Workflow but when I try to access it I am getting the following page:
I have absolutely no clue why. Everything in the process went smoothly but the page is not showing. I could really use some help!
I searched around for this error but I couldn't find anything. I also tried different deployment strategies to no avail.
EDIT: turns out that my project files are not getting copied to the app service.
EDIT 2: I got the app running with a zip deploy but Id still like to use GitHub Actions to automate it
I don't know how you create the .yml file. I generate it by default, you can check below. It it will deploy failed and don't worry.
Then you will find the .yml file generated under your github repo. Please find it and copy the below script and paste there. Don't forget replace the app-name, publish-profile and so on.
Then it will auto deploy again, now it should be deployed successfully. If not, try to modify anything, and deploy again for testing. You can see I add Azure Web App-01 in the first line.
The script I have tested and it works well like below.
Then we will find the publish file under the wwwroot. The scm site is :
https://app_name.scm.azurewebsites.net/newui
Then we should add startup command in azure portal.
Then the application should works well in your side. 👍
It not works in my side, after investigating the reason,I found it missing the connectionstring and other configurations, so it not works in my side.
.yml script
name: Build and deploy ASP.Net Core app to Azure Web App-01 - WebApplication920221114163713
on:
push:
branches:
- main
env:
AZURE_WEBAPP_NAME: WebApplication920221114163713
AZURE_WEBAPP_PACKAGE_PATH: .
CONFIGURATION: Release
DOTNET_CORE_VERSION: 6.0.x
WORKING_DIRECTORY: EasyRank
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Set up .NET Core
uses: actions/setup-dotnet#v1
with:
dotnet-version: '6.0.x'
include-prerelease: true
- name: Build with dotnet
run: dotnet build "${{ env.WORKING_DIRECTORY }}" --configuration Release
- name: dotnet publish
run: dotnet publish "${{ env.WORKING_DIRECTORY }}" --configuration ${{ env.CONFIGURATION }} --no-build -o ${{env.DOTNET_ROOT}}/myapp
- name: Upload artifact for deployment job
uses: actions/upload-artifact#v2
with:
name: .net-app
path: ${{env.DOTNET_ROOT}}/myapp
deploy:
runs-on: ubuntu-latest
needs: build
environment:
name: 'Production'
url: ${{ steps.deploy-to-webapp.outputs.webapp-url }}
steps:
- name: Download artifact from build job
uses: actions/download-artifact#v2
with:
name: .net-app
- name: Deploy to Azure Web App
id: deploy-to-webapp
uses: azure/webapps-deploy#v2
with:
app-name: 'WebApplication920221114163713'
slot-name: 'Production'
publish-profile: ${{ secrets.AZUREAP***0A789F0B31 }}
package: .
Is that possible to deploy the Laravel web application to shared hosting using GitHub Action & GitHub FTP Deploy? If possible how should I change the.github\workflows\master.yml?
on:
push:
branches:
- master
name: 🚀 Deploy website on push
jobs:
web-deploy:
name: 🎉 Deploy
runs-on: ubuntu-latest
steps:
- name: 🚚 Get latest code
uses: actions/checkout#v2
- name: 📂 Sync files
uses: SamKirkland/FTP-Deploy-Action#4.2.0
with:
server: ${{ secrets.ftp_server }}
username: ${{ secrets.ftp_username }}
password: ${{ secrets.ftp_password }}
server-dir: /
Looks like you're very close but are missing 2 important steps: set up a temporary PHP environment, and use that environment to install your dependencies (Composer).
GitHub Actions Setup
This guide assumes you have a working Laravel installation, a GitHub account, and a shared hosting account that you can access via FTP using a username/password.
I found this video https://www.youtube.com/watch?v=UNWIXYSZfZY helpful to get a basic understanding of how to deploy a simple application. To make this answer helpful to a wider range of people, I'll give a quick outline of my setup. There really aren't any Laravel specific steps.
Workflow directory set up
Create the directories .github\workflows at the root of your project. In the workflows directory, create a yml file named after the branch you want to push to your shared hosting account. Ex. master.yml, staging.yml, development.yml etc. If you only have a single branch then just create one file. The name is important and should match the name of the branch.
Design your workflow
This is very dependent on your project but assuming you have a basic Laravel application without the need for additional components such as Node, then this is a basic GitHub Action that works for me on a variety of projects.
A basic action file consists of 2 sections, the workflow, and the jobs. A workflow triggers the jobs.
Workflow
Lines 1-4 say this will run each time we push to the master branch.
on:
push:
branches:
- master
Line 5 is the name of this workflow and will show up on your Actions page. Set this to something descriptive.
name: 🚀 Deploy website on push (Master)
Setting up jobs
In this action, there are 5 jobs. Some take parameters, others don't. I'm not going to explain all the details here but have linked to the corresponding repositories if you need details.
Checkout your code so the workflow has access to it,
https://github.com/actions/checkout
name: 🚚 Get latest code
uses: actions/checkout#v2
Sets up a temporary PHP environment so you can run things like
Composer, https://github.com/shivammathur/setup-php. Make sure to set your PHP version here otherwise you could run into issues when installing Composer packages with an unexpected PHP version.
name: Setup PHP
uses: shivammathur/setup-php#v2
with:
php-version: 7.2
Caches your dependencies for faster deploys,
https://github.com/actions/cache
name: Cache Composer packages
id: composer-cache
uses: actions/cache#v2
with:
path: vendor
key: ${{ runner.os }}-php-${{ hashFiles('**/composer.lock') }}
restore-keys: |
${{ runner.os }}-php-
Install your dependencies from composer.json and composer.lock
files.
name: Install dependencies
run: composer install --prefer-dist --no-progress
Deploys your code to your remote shared hosting site,
https://github.com/SamKirkland/FTP-Deploy-Action. Note the use of ${{ secrets.ftp_username }} and ${{ secrets.ftp_password }}. These are set up in your repository's secrets section. See https://docs.github.com/en/actions/security-guides/encrypted-secrets
name: 📂 Sync files
uses: SamKirkland/FTP-Deploy-Action#4.0.0
with:
server: name_of_server.com
username: ${{ secrets.ftp_username }}
password: ${{ secrets.ftp_password }}
server-dir: public_html/
Final file
on:
push:
branches:
- master
name: 🚀 Deploy website on push (Master)
jobs:
web-deploy:
name: 🎉 Deploy
runs-on: ubuntu-latest
steps:
- name: 🚚 Get latest code
uses: actions/checkout#v2
- name: Setup PHP
uses: shivammathur/setup-php#v2
with:
php-version: 7.2
- name: Cache Composer packages
id: composer-cache
uses: actions/cache#v2
with:
path: vendor
key: ${{ runner.os }}-php-${{ hashFiles('**/composer.lock') }}
restore-keys: |
${{ runner.os }}-php-
- name: Install dependencies
run: composer install --prefer-dist --no-progress
- name: 📂 Sync files
uses: SamKirkland/FTP-Deploy-Action#4.0.0
with:
server: name_of_server.com
username: ${{ secrets.ftp_username }}
password: ${{ secrets.ftp_password }}
server-dir: public_html/
Running the workflow
Check-in .github\workflows\master.yml, and others if appropriate,
into your GitHub repository. Without these files checked in nothing
will happen when you push a change to the branch.
Go to your Actions tab and ensure the workflow shows up there.
Push a change to your branch and watch the Actions tab. Click into
the running action to see details about the run.
Fix any errors that show up in the console.
Finally, you mentioned in a comment something about NPM. If you have Node as a component in your project you can simply run two extra steps that will bundle your assets and will get deployed along with the rest of the code.
Good luck!
Hello there and thank you for reading my question, its my first one here.
I am working with CI/CD pipelines for a year now and I think they are pretty nice and convinient for developing Websites and Stuff. But in the last months I have more and more problems creating fast, efficient and smart pipelines without redundant dependency installs or similar. So I want to use as less computation ressources as possible while still have fast builds. I want to parallelize steps and use theire artifacts in another final step. For example the following GitHub Actions workflow:
My goal with this workflow is to just build a VueJS Single Page App and deploy it to the IBM Cloud. For that I need to install the npm dependencies and build the Vue App and also install the IBM Cloud CLI. After these two steps are finished the builded App should be pushed to the IBM Cloud.
I could just simply run all steps sequentially like this:
name: Deploy
on:
push:
branches: [main]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout Code
uses: actions/checkout#v2
- name: Use Node.js 10.X
uses: actions/setup-node#v1
with:
node-version: '10.x'
- name: Cache Node Modules
uses: actions/cache#v2
env:
cache-name: cache-node-modules
with:
# npm cache files are stored in `~/.npm` on Linux/macOS
path: ~/.npm
key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-build-${{ env.cache-name }}-
${{ runner.os }}-build-
${{ runner.os }}-
- name: Install Dependencies
run: npm ci
- name: Build Page
run: npm run build
- name: Install IBM Cloud CLI
run: curl -fsSL https://clis.cloud.ibm.com/install/linux | sh
shell: bash
- name: Install Cloud Foundry CLI
run: ibmcloud cf install
shell: bash
- name: Authenticate with IBM Cloud CLI
run: ibmcloud login --apikey "${{ secrets.IBM_CLOUD_API_KEY }}" --no-region -g Default
shell: bash
- name: Target a Cloud Foundry org and space
run: ibmcloud target --cf-api "${{ secrets.IBM_CLOUD_CF_API }}" -o "${{ secrets.IBM_CLOUD_CF_ORG }}" -s "${{ secrets.IBM_CLOUD_CF_SPACE }}"
shell: bash
- name: Deploy to Cloud Foundry
run: ibmcloud cf push
shell: bash
But in my opinion this is very ugly and can be improved. So I tried to split the job into 3 parts: build, predeploy and deploy. The build job installs and builds the Vue App. The Predeploy job install the IBM CLI. These two jobs doesn't depend on each other so they can be parallized. But the last job, deploy, depends on both so I added the needs: [build, predeploy] value to it. So I have the following workflow to archive this:
### This will not work!
name: Deploy
on:
push:
branches: [main]
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout Code
uses: actions/checkout#v2
- name: Use Node.js 10.X
uses: actions/setup-node#v1
with:
node-version: '10.x'
- name: Cache Node Modules
uses: actions/cache#v2
env:
cache-name: cache-node-modules
with:
# npm cache files are stored in `~/.npm` on Linux/macOS
path: ~/.npm
key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-build-${{ env.cache-name }}-
${{ runner.os }}-build-
${{ runner.os }}-
- name: Install Dependencies
run: npm ci
- name: Build Page
run: npm run build
predeploy:
runs-on: ubuntu-latest
defaults:
run:
shell: bash
steps:
- name: Install IBM Cloud CLI
run: curl -fsSL https://clis.cloud.ibm.com/install/linux | sh
- name: Install Cloud Foundry CLI
run: ibmcloud cf install
- name: Authenticate with IBM Cloud CLI
run: ibmcloud login --apikey "${{ secrets.IBM_CLOUD_API_KEY }}" --no-region -g Default
- name: Target a Cloud Foundry org and space
run: ibmcloud target --cf-api "${{ secrets.IBM_CLOUD_CF_API }}" -o "${{ secrets.IBM_CLOUD_CF_ORG }}" -s "${{ secrets.IBM_CLOUD_CF_SPACE }}"
deploy:
needs: [build, predeploy]
runs-on: ubuntu-latest
steps:
- name: Deploy to Cloud Foundry
# Error: 'ibmcloud: command not found'
run: ibmcloud cf push
shell: bash
Which looks on the GUI like:
[![My GitHub Workflow on the GUI][1]][1]
But this workflow will error since the last job doesn't share the same environment as the other jobs. I am aware that I could use the up/download Artifact feature of GitHub Actions but this seems to me like using a lot of resources. But I dont want to use a lot of ressources for my pipeline, I dont need a lot of different virtual environments or build matrixes. (I know they are very good for large projects, but they seem a little overkill for my little site)
So here are my two final Questions:
Why is parallelism in CI/CD often complication and not straight forward?
How can I improve my current pipeline with parallelism and without redundant executions?
I am glad about every helpful advice or link. Thank you. :)
[1]: https://i.stack.imgur.com/qEqLs.png
I think your original workflow was already pretty efficient. As you mentioned, different jobs are executed on different runners and sometime the additional complexity and effort put into the synchronization/logic between workflows outweighs the benefits of parallelism. In your case I don't think it would make much sense to run your jobs in parallel.
For your first question, I don't think it's an issue specific to CI/CD pipelines. I am getting a bit out of scope here but you have similar issues in any code that does work in parallel or as a matter of fact in any work in general that is done in parallel anywhere. Being factories, teams, code, CI pipelines, as soon as the work is split up, there will be some sort of mechanism to manage the allocation of work and track its progress. Which will make it more complex.
Why GH workflows might seem less straightforward than other systems seem to be a better question and I think it comes does to how long it has been around. It's a pretty recent addition to github and as new features are progressively being added it gets easier and easier to work with.
Regarding other optimizations for your workflow, I would recommend trying to avoid redoing the same work every time the workflow run if it's not needed. You already do this with the cache action for npm. But you could, for example build a docker image, or even better an action ,with your IBM CLI in it and remove the pre-deploy stage entirely. Simply having:
name: Deploy
on:
push:
branches: [main]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout Code
uses: actions/checkout#v2
- name: Use Node.js 10.X
uses: actions/setup-node#v1
with:
node-version: '10.x'
- name: Cache Node Modules
uses: actions/cache#v2
env:
cache-name: cache-node-modules
with:
# npm cache files are stored in `~/.npm` on Linux/macOS
path: ~/.npm
key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-build-${{ env.cache-name }}-
${{ runner.os }}-build-
${{ runner.os }}-
- name: Install Dependencies
run: npm ci
- name: Build Page
run: npm run build
- name: Deploy to Cloud Foundry
uses: my-action:v1
with:
api-key: ${{ secrets.IBM_CLOUD_API_KEY }}
cf-api: ${{ secrets.IBM_CLOUD_CF_API }}
cf-org: ${{ secrets.IBM_CLOUD_CF_ORG }}
cf-space: ${{ secrets.IBM_CLOUD_CF_SPACE }}
I am currently completing a migration from travis CI to Github Actions CI however my build keeps failing when the container tries to connect to my test DB.
Its a very generic error
UNKNOWN_CODE_PLEASE_REPORT: An internal error has occurred. Please retry or report your issues.
at Handshake.Object.<anonymous>.Sequence._packetToError (node_modules/***/lib/protocol/sequences/Sequence.js:47:14)
at Handshake.Object.<anonymous>.Handshake.ErrorPacket (node_modules/***/lib/protocol/sequences/Handshake.js:123:18)
at Protocol.Object.<anonymous>.Protocol._parsePacket (node_modules/***/lib/protocol/Protocol.js:291:23)
at Parser._parsePacket (node_modules/***/lib/protocol/Parser.js:433:10)
at Parser.write (node_modules/***/lib/protocol/Parser.js:43:10)
at Protocol.Object.<anonymous>.Protocol.write (node_modules/***/lib/protocol/Protocol.js:38:16)
at Socket.<anonymous> (node_modules/***/lib/Connection.js:88:28)
at Socket.<anonymous> (node_modules/***/lib/Connection.js:526:10)
Additional Notes: My test DB is hosted on Azure (mysql server) and my tests execute fine on local and previously on travis. Project details: NodeJs project running TypeOrm for MySql.
This is what the current base version of my yml looks like:
on:
pull_request:
branches: [master, develop]
defaults:
run:
working-directory: ./Server
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Use Node.js
with:
node_version: 14.x
uses: actions/setup-node#v1
- name: Run tests
run: |
npm install
npx jest --coverage --config ./jest.config.js
env:
TEST_DB_NAME: ${{ secrets.TEST_DB_NAME }}
TEST_DB_PORT: ${{ secrets.TEST_DB_PORT }}
TEST_DB_TYPE: ${{ secrets.TEST_DB_TYPE }}
TEST_HOST: ${{ secrets.TEST_HOST }}
TEST_PASSWORD: ${{ secrets.TEST_PASSWORD }}
TEST_PORT: ${{ secrets.TEST_PORT }}
ACCESS_SECRET_KEY: ${{ secrets.ACCESS_SECRET_KEY }} ```
Answering my own question:
I was missing an extra env key/variable related to an external AP. This was causing the failure.
There are a lot of similar issues already floating around:
Install private github package from package.json on Github Actions
Download private module from Github Package Registry via Yarn within a Github Action? Publishing works, but installing is met with '401 Unauthorized'
Installing private package from Github Package registry using Yarn fails with not authorized
However, our issue seems different, because:
yarn install runs fine on a local machine
the issue is only when using Github Actions
yarn install succeeds on GH Actions if we delete yarn.lock
Has anyone run into this before? Specifically with it not working with a yarn.lock file?
In case it matters, here's the setup:
build.yml:
steps:
- uses: actions/checkout#v2
- uses: actions/setup-node#v1
with:
node-version: '10.x'
registry-url: 'https://npm.pkg.github.com'
- name: Install
run: yarn install
env:
# GITHUB_TOKEN can't access packages hosted in private repos,
# even within the same organisation
NODE_AUTH_TOKEN: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
- name: Build
run: yarn build
- name: Test
run: yarn test --forbid-only
We also have a .npmrc file for local installs:
#<org>:registry=https://npm.pkg.github.com
But no .yarnrc file.
I'm create a file .npmrc and .yarnrc.
Type:
name: Test
on: push
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
node-version: [12.x]
steps:
- uses: actions/checkout#v2
- name: Node ${{ matrix.node-version }}
uses: actions/setup-node#v1
with:
node-version: ${{ matrix.node-version }}
- name: Create NPMRC
run: |
echo "//npm.pkg.github.com/:_authToken=${{ secrets.PACKAGES_TOKEN }}" >> ~/.npmrc
echo "#you-scope:registry=https://npm.pkg.github.com" >> ~/.npmrc
echo 'registry "https://registry.yarnpkg.com"' >> ~/.yarnrc
- run: yarn install
Replace #you-scope for you user of github or of your org in github in LowerCase.
Create a PACKAGES_TOKEN secrete token of your github access for this repository.
We managed to solve this by explicitly duplicating the .npmrc config in the build.yml config:
- uses: actions/setup-node#v1
with:
node-version: '10.x'
registry-url: 'https://npm.pkg.github.com'
# These following two lines are the key:
always-auth: true
scope: '#reedsy'