I'm trying to run a local server for my project CI/CD Pipeline. When I start the server I got a "Connection refused" on it.
My project is a fastAPI application and I'm trying to run a integration tests on PR to validate the app before merge the code. I tried to start my app directly (gunicorn), building a docker image and starting it... I tried a lot of things. Then, I tried to run a simple server instead of my app and... got the same error!
This is my simple server workflow:
on:
push:
branches:
- "develop"
jobs:
mylocalserver:
name: Run server
runs-on: ubuntu-latest
steps:
- name: Setup python
uses: actions/setup-python#v3
with:
python-version: 3.9
- name: Run server in background
run: |
python -V
python -m http.server 3000 &> /dev/null &
sudo lsof -i -P -n | grep LISTEN
curl http://localhost:3000
- name: Run server with console
run: |
python -m http.server 3000
Output:
If I run my app with console (no daemon mode in gunicorn), the server start and log to console in workflow with success:
But this way I cannot run nothing after this (and I have to cancel workflow). Some idea? Thank you!
Maybe not the best answer, but for now runnig the job into a container works (only add a container label in question example). Example for my fastAPI app:
on:
pull_request:
branches:
- 'main'
- 'develop'
jobs:
run-on-pr:
runs-on: ubuntu-latest
container: ubuntu
services:
mongodb:
image: mongo
ports:
- 27017:27017
steps:
- name: Setup git
run: |
apt-get update; apt-get install -y git
- name: Git checkout
uses: actions/checkout#v3
with:
path: api
- name: Setup python
uses: actions/setup-python#v4
with:
python-version: 3.9
- name: Install pip
run: |
apt-get update; apt-get install -y python3-pip
- name: Build and ENV
run: |
cd api
cp .env_example_docker .env
pip3 install -r requirements.txt
- name: Run fastAPI
run: |
cd api
gunicorn -D -k uvicorn.workers.UvicornWorker -c ./gunicorn_conf.py app.main:app
env:
MONGO_URL: "mongodb://mongodb:27017"
- name: Install curl
run: |
apt-get update; apt-get install -y curl
- name: Run curl
run: |
curl http://localhost:3000
This works, but I have to install all in container (git, pip). I will try a solution without using the container label and if I found anything I can post here.
Related
I have added Machine Learning Model and preceding with CI pipeline in Github, however while executing workflow, I am getting error for Missing required root key on. Below is my yml file code.
ci_pipeline:
on:
push:
branches:
- main
steps:
- uses: actions/checkout#v1
with:
fetch-depth: 0
- name: Set up Python 3.9
uses: actions/setup-python#v1
with:
python-version: 3.9
- name: Install dependencies
run: |
python -m pip install --upgrade pip
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
- name: Format
run: |
black app.py
- name: Lint
run: |
pylint --disable=R,C app.py
- name: Test
run: |
python -m pytest -vv test.py
I was trying to understand the CI/CD flow from this tutorial by Indian Pythonista and after following the tutorial and setting the Heroku app name and token in the Github Secrets I ran the yaml file below.
name: Python application
on: [push]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- run: |
git fetch --prune --unshallow
- name: Set up Python 3.8
uses: actions/setup-python#v1
with:
python-version: 3.8
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install flake8 pytest
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
- name: Lint with flake8
run: |
# stop the build if there are Python syntax errors or undefined names
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
- name: Test with pytest
run: |
pip install pytest
export PYTHONPATH=src
pytest
- name: Deploy to Heroku
env:
HEROKU_API_TOKEN: ${{ secrets.HEROKU_API_TOKEN }}
HEROKU_APP_NAME: ${{ secrets.HEROKU_APP_LINKNAME }}
if: github.ref == 'refs/heads/master' && job.status == 'success'
run: |
git remote add heroku https://heroku:$HEROKU_API_TOKEN#git.heroku.com/$HEROKU_APP_NAME.git
git push heroku HEAD:master -f
The build looked like the screenshot below:
I do not exactly know what should I call this error as well, since I am new to the topic. Due to this, the flask app doesn't get updated on the Heroku app link as well.
If I push to master, it works perfectly, all environment variables available and I have a successful deploy to Heroku.
Problem: If I push to the dev branch, it can't see the environment variables for the deploy.
$ dpl --provider=heroku --app=$HEROKU_DEV_APP --api-key=$HEROKU_API_KEY
invalid option "--api-key="
ERROR: Job failed: exit code 1
Environment settings:
.gitlab-ci.yml:
stages:
- build
- test
- deploy
build:
stage: build
image: maven:3.6.3-jdk-14
script:
- mvn clean package
tags:
- docker
test:
stage: test
image: maven:3.6.3-jdk-14
script:
- mvn test
tags:
- docker
deploy_dev:
stage: deploy
image: ruby:2.3
script:
- apt-get update -qy
- apt-get install -y ruby-dev
- gem install dpl
- dpl --provider=heroku --app=$HEROKU_DEV_APP --api-key=$HEROKU_API_KEY
environment:
name: prod
url: https://.....herokuapp.com/
only:
- dev
tags:
- docker
deploy_prod:
stage: deploy
image: ruby:2.3
script:
- apt-get update -qy
- apt-get install -y ruby-dev
- gem install dpl
- dpl --provider=heroku --app=$HEROKU_PROD_APP --api-key=$HEROKU_API_KEY
environment:
name: prod
url: https://.....herokuapp.com/
when: manual
only:
- master
tags:
- docker
This is because your Heroku api key variable is set as protected.
Protected variables are visible only by protected branches and protected tags. That is why it works for you on master but not on dev.
More information: https://gitlab.com/help/ci/variables/README#protect-a-custom-variable and https://gitlab.com/help/user/project/protected_branches.md
Your options are: either remove protected flag, or introduce another unprotected variable with another api key for your non-protected branches which would be less sensitive.
I am planning to migrate my existing CI to CircleCI 2 in favor of the docker. Once I tried to trigger a build it's shown something like this:
I also checked the steps as indicated in the below alert here https://circleci.com/sunset1-0/
Service alert: Your project references CircleCI 1.0 or it has no
configuration. CircleCI 1.0 and projects without configuration files
are no longer supported. You must update your project to use CircleCI
2.0 configuration to continue. Learn more.
Is there anything I missed out?
Below is my .circleci/config.yml
version: 2 # use CircleCI 2.0
jobs: # a collection of steps
build: # runs not using Workflows must have a `build` job as entry point
parallelism: 3 # run three instances of this job in parallel
docker: # run the steps with Docker
- image: circleci/ruby:2.4.2-jessie-node # ...with this image as the primary container; this is where all `steps` will run
environment: # environment variables for primary container
BUNDLE_JOBS: 3
BUNDLE_RETRY: 3
BUNDLE_PATH: vendor/bundle
RAILS_ENV: test
- image: mysql:5.7 # database image
environment: # environment variables for database
MYSQL_ALLOW_EMPTY_PASSWORD: 'yes'
steps: # a collection of executable commands
- checkout # special step to check out source code to working directory
- run:
name: setup
command: |
curl -sL https://deb.nodesource.com/setup_10.x | bash \
&& curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - \
&& echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
- run:
name: Dependencies
command: |
apt-get update && \
DEBIAN_FRONTEND=noninteractive apt-get install -y \
build-essential mysql-client nodejs yarn && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
# Which version of bundler?
- run:
name: Which bundler?
command: bundle -v
# Restore bundle cache
# Read about caching dependencies: https://circleci.com/docs/2.0/caching/
- restore_cache:
keys:
- rails-demo-bundle-v2-{{ checksum "Gemfile.lock" }}
- rails-demo-bundle-v2-
- run: # Install Ruby dependencies
name: Bundle Install
command: bundle check --path vendor/bundle || bundle install --deployment
# Store bundle cache for Ruby dependencies
- save_cache:
key: rails-demo-bundle-v2-{{ checksum "Gemfile.lock" }}
paths:
- vendor/bundle
# Only necessary if app uses webpacker or yarn in some other way
- restore_cache:
keys:
- rails-demo-yarn-{{ checksum "yarn.lock" }}
- rails-demo-yarn-
- run:
name: Yarn Install
command: yarn install --cache-folder ~/.cache/yarn
# Store yarn / webpacker cache
- save_cache:
key: rails-demo-yarn-{{ checksum "yarn.lock" }}
paths:
- ~/.cache/yarn
- run:
name: Wait for DB
command: dockerize -wait tcp://localhost:5432 -timeout 1m
- run:
name: Database setup
command: bin/rails db:schema:load --trace
- run:
name: Run rspec in parallel
command: |
bundle exec rspec --profile 10 \
--format RspecJunitFormatter \
--out test_results/rspec.xml \
--format progress \
$(circleci tests glob "spec/**/*_spec.rb" | circleci tests split --split-by=timings)
# Save test results for timing analysis
- store_test_results: # Upload test results for display in Test Summary: https://circleci.com/docs/2.0/collect-test-data/
path: test_results
# See https://circleci.com/docs/2.0/deployment-integrations/ for example deploy configs
In order for the build work, your config.yml file must live inside a .circleci folder at the root of your project directory.
You post a very good description. Thanks for the code. The version config looks fine. However, you wrote that your config currently lives at .circle/config.yml (Notice the missing ci at end of circle)
Move the config.yml to .circleci/config.yml and the build should proceed further. (right now the build is failing because it has no configuration.
E
I have created a CircleCI config which will run my PHPUnit tests against my laravel application and that is working 100% however I am now trying to add a workflow to then SSH and deploy my app to an AWS EC2 server and I am getting the following errors:
Your config file has errors and may not run correctly:
2 schema violations found
required key [jobs] not found
required key [version] not found
However I cannot see an issue with my CircleCI config file, have I made a mistake somewhere?
version: 2
jobs:
build:
docker:
- image: circleci/php:7.1-browsers
working_directory: ~/laravel
steps:
- checkout
- run:
name: Download NodeJS v6
command: curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash -
- run:
name: Install SQLite and NodeJS 6
command: sudo apt-get install -y libsqlite3-dev nodejs
- run:
name: Setup Laravel testing environment variables for CircleCI test
command: cp .env.circleci .env
- run:
name: Update composer to latest version
command: composer self-update
- restore_cache:
keys:
- composer-v1-{{ checksum "composer.json" }}
- composer-v1-
- run: composer install -n --prefer-dist --ignore-platform-reqs
- save_cache:
key: composer-v1-{{ checksum "composer.json" }}
paths:
- vendor
- restore_cache:
key: dependency-cache-{{ checksum "package.json" }}
- run:
name: Install NodeJS Packages
command: npm install
- save_cache:
key: dependency-cache-{{ checksum "package.json" }}
paths:
- ./node_modules
- run:
name: Create SQLite Database
command: touch database/database.sqlite
- run:
name: Migrate Laravel Database
command: php artisan migrate --database=sqlite --force
- run:
name: Run NPM
command: npm run production
# Run Laravel Server for front-end tests
- run:
name: Run Laravel Server
command: php artisan serve
background: true
- run:
name: Run PHPUnit Tests
command: vendor/bin/phpunit
deploy:
machine:
enabled: true
steps:
- run:
name: Deploy Over SSH
command: |
ssh $SSH_USER#$SSH_HOST "cd /var/www/html"
workflows:
version: 2
build-and-deploy:
jobs:
- build
- deploy:
requires:
- build
filters:
branches:
only: master
Any help is appreciated, thank you!
CircleCI has documentation for AWS deployment. Look here https://circleci.com/docs/1.0/continuous-deployment-with-aws-codedeploy/
I think your problem is with SSH authorization for AWS. You can try it locally and make sure that your authorization is successfully, and then do the same thing with your AWS.