How to get GitLab CI variables into the Sinatra app to query Google Cloud Datastore? - ruby

Short:
I want to be able to deploy a Sinatra app without hardcoded secrets and variables using Google Cloud App Engine. Deploying to App Engine while ensuring the environment variables get loaded from the Google Cloud Datastore was the suggested option. I hoped to find something similar to the simplicity of Heroku, but this is not the case.
Question:
How would I get the $SERVICE_ACCOUNT environment information (that I've set in the GitLab CI / CD section) as the credentials value in foobar.rb?
app.yaml:
image: google/cloud-sdk:latest
before_script:
- apt-get install -y ruby-full build-essential
- gem install bundler
- bundle config --global silence_root_warning 1
- bundle install
after_script:
- rm /tmp/$CI_PIPELINE_ID.json
deploy_production:
stage: deploy
environment: Production
only:
- production
script:
- echo $SERVICE_ACCOUNT > /tmp/$CI_PIPELINE_ID.json
- gcloud auth activate-service-account --key-file /tmp/$CI_PIPELINE_ID.json
- gcloud --quiet --project $PROJECT_ID app deploy app.yaml
foobar.rb:
...
require "google/cloud/datastore"
class App < Sinatra::Application
if settings.production?
data_store = Google::Cloud::Datastore.new(
project: 'foobar',
credentials: "???"
)
query = data_store.query "kindoffoobar"
results = data_store.run query
# Set each result as an env variable
results[0].properties.to_h.each do |key, value|
ENV[key]= value
end
end
... rest of app

Related

Gitlab pipeline error With CD/CI for AWS ec2 debian instance: This job is stuck because you don't have any active runners online

I want to create a CI/CD pipeline between gitlab and aws ec2 deployment.
My repository is nodejs/express web server project.
And I created a gitlab-ci.yaml
image: node:latest
cache:
paths:
- node_modules/
stages:
- build
- test
- staging
- openMr
- production
before_script:
- apt-get update -qq && apt-get install
Build:
stage: build
tags:
- node
before_script:
- yarn config set cache-folder .yarn
- yarn install
script:
- npm run build
Test:
stage: test
tags:
- node
before_script:
- yarn config set cache-folder .yarn
- yarn install --frozen-lockfile
script:
- npm run test
Deploy to Production:
stage: production
tags:
- node
before_script:
- mkdir -p ~/.ssh
- echo -e "$SSH_PRIVATE_KEY" > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
script:
- bash ./gitlab-deploy/.gitlab-deploy.prod.sh
environment:
name: production
url: http://ec2-url.compute.amazonaws.com:81
When I push a new commit pipeline failed on build step. And I get a warning as :
This job is stuck because you don't have any active runners online or
available with any of these tags assigned to them: node
I checked my runner on gitlab settings/CI/CD
After that I checkked server
admin#ip-111.222.222.111:~$ gitlab-runner
statusRuntime platform arch=amd64 os=linux pid=18787 revision=98daeee0 version=14.7.0
FATAL: The --user is not supported for non-root users
You need to remove the tag node from your jobs. Runner tags are used to define which runner should pick up your jobs (https://docs.gitlab.com/ee/ci/runners/configure_runners.html#use-tags-to-control-which-jobs-a-runner-can-run). As there is no runner available which supports the tag node, your job gets stuck.
It doesn't look like your pipeline has any special requirements so you can just remove the tag so it can be picked up by every runner.
The runner that can be seen in your screenshot supports the tag shop_service_runner. So another option would be to change the tag node to shop_service_runner which would lead to this runner (and every runner with the same tags) being able to pick up this job.

Travis doesn't deploy on heroku. Invalid credentials

I'm trying to set up automatic deploy on heroku with travis. I get this error when travis try to deploy:
API request failed.
Message: Invalid credentials provided.
Reference:
failed to deploy
This is my travis file:
jobs:
include:
- language: python
python:
- "3.6"
install:
- pip install -r Deployment/requirements.txt
script:
- python -c "print ('Testing some script')"
branches:
only:
- master
- develop
before_deploy:
- cd Deployment
deploy:
- provider: heroku
skip_cleanup: true
api_key:
secure: b3AVdCtJ2e/+Gu1...
app:
master: motorent-deploy
develop: motorent-apitest
- language: android
dist: trusty
env:
global:
- ANDROID_API_LEVEL=29
- ANDROID_BUILD_TOOLS_VERSION=29.0.3
- extra-google-google_play_services
- extra-google-m2repository
- extra-android-m2repository
- addon-google_apis-google-$ANDROID_API_LEVEL
android:
licenses:
- 'android-sdk-preview-license-.+'
- 'android-sdk-license-.+'
- 'google-gdk-license-.+'
components:
- tools
- platform-tools
- android-$ANDROID_API_LEVEL
- build-tools-$ANDROID_BUILD_TOOLS_VERSION
- extra-google-google_play_services
- extra-google-m2repository
- extra-android-m2repository
- addon-google_apis-google-$ANDROID_API_LEVEL
addons:
apt:
packages:
ant
before_install:
- touch $HOME/.android/repositories.cfg
- yes | sdkmanager "platforms;android-29"
- yes | sdkmanager "build-tools;29.0.3"
before_script:
- cd AndroidApp
- chmod +x gradlew
script:
- ./gradlew build check
As you can see I have two differents projects in the same repository, but it's not important, because the android test works well. What doesn't work is the deploy of Flask project. The solutions that I have found talk about the need to encrypt the api_key. I have tested it with Travis encrypt $(heroku auth:token) but it doesn't work either.
I've been trying to find the error for a long time but I don't know what it can be.
I had the same error.
Here are steps that I performed to fix it.
Firstly I tried the command: heroku auth:token
but the output was:
› Warning: token will expire 06/06/2021
› Use heroku authorizations:create to generate a long-term token
Then I tried the command: heroku authorizations:create
One line from the output contained Token: <created_heroku_auth_token>
I took the value of it (<created_heroku_auth_token>)
and I went to
https://travis-ci.org/github/<my_github_user>/<my_repo>/settings
where I created new environment variable:
HEROKU_AUTH_TOKEN with value of my <created_heroku_auth_token>
Then in my .travis.yml I changed value of api_key to:
api_key: $HEROKU_AUTH_TOKEN
After pushing this change, the deployment to heroku went fine.

Travis-Ci after_deploy script is not working, and displaying success

This my .travis.yml file. I am trying to automate deployment to aws-codedeploy.
language: node_js
node_js:
- 7.10.0
services:
- mongodb
env:
- PORT=6655 IP="localhost" NODE_ENV="test"
script:
- npm start &
- sleep 25
- npm test
deploy:
provider: codedeploy
access_key_id:
secure: $Access_Key_Id
secret_access_key:
secure: $Access_Key_Secret
revision_type: github
application: Blog
deployment_group: Ayush-Bahuguna
region: us-east-2
after_deploy:
- "./build.sh"
Here build.sh is a shell script that generates the build files
cd /var/www/cms
sudo yarn install
npm run build-prod
And here is .gitignore file
node_modules/
client/dashboard/dist/
client/blog/dist/
The issue is that, even though travis-ci build succeeds, and after_deploy runs successfully, no build files are generated on the aws ec2 instance where my project is hosted.
Are you able to see any deployment created on your AWS CodeDeploy console? And are your able to see the deployment status? If there is a deployment created, but failed, you can try to see the reason why it failed. Even though the deployment succeeded, it doesn't equal to all instances are deployed depends on the deployment configuration: http://docs.aws.amazon.com/codedeploy/latest/userguide/deployment-configurations.html.
Thanks,
Binbin

How to input heroku credentials in Travis Ruby on Rails

Am working Ruby on Rails site, and I have implemented Travis CI with it and pushed to to GitHub, so as to Test my build before pushing to Heroku.
When Travis parsed my github source code, I get an error asking me to input my Heroku Credentials before Travis can push to Heroku.
What I want to do
How do I pass in my heroku credentials to Travis as requested without the >risk of putting it on version control?
Here is the Build on Travis: https://travis-ci.org/AfolabiOlaoluwa/LawVille/jobs/166099588
.travis.yml
language: ruby
rvm:
- 2.2.4
env:
global:
- secure: {{ I have my travis encrypted key here }}
- secure: {{ I another travis encrypted key here }}
- DB=sqlite
- DB=mysql
- DB=postgresql
- secure: {{ I have another travis encrypted key here }}
deploy:
provider: heroku
api_key:
secure: {{ I have HEROKU API KEY encrypted by travis here }}
script:
- RAILS_ENV=test bundle exec rake db:migrate --trace
- bundle exec rake db:test:prepare
before_script:
- mysql -e 'create database strano_test'
- psql -c 'create database strano_test' -U postgres
after_success:
- gem install heroku
- yes | ruby ./config/initializers/travis_deployer.rb
- git remote add heroku git#heroku.com:lawville.git
- heroku keys:clear
- yes | heroku keys:add
- git push heroku master
The answer from #andresk above is very useful, but encrypting the api-key only didnt work for me as you could see from our comments/correspondence.
What worked for me was, running setup heroku --org --force from the root of my app inside terminal and I followed the prompt to get my Travis file having the correct deploy script.
Note: the --force command is to force script written on the already existing .travis.yml file.
After responding to the prompts, my deploy script became:
deploy:
provider: heroku
api_key:
secure: encrypted_key_was_written_here
app: lawville # being app name on heroku which am deploying to.
on:
repo: AfolabiOlaoluwa/LawVille # GitHub Repo
That's what worked for me.
You don't need to manually push to heroku on after_success. Just having the deploy with your encrypted credentials is enough to automatically deploy after the build. So try removing the after_success commands and everything should work.
For more information, check here
If you need to store your credentials to be used on Travis, you can add secure environment to Travis. Go to More options > Settings there you can add the credentials as environment variables. Just make sure Display value in build log is set as off

deploy docker to heroku without using heroku docker plugin

Say I'm working on a web project that runs gitlab-ci shell runner on my own ci server to build docker and deploy it to heroku, and I've gone through some docs from both gitlab and heroku like gitlab-ci: using docker build and heroku:Build and Deploy with Docker. Can I deploy the docker project without using heroku-docker plugin, which seems not so flexible to me? However I tried, the following approach build succeeded in deploying to heroku, but the app crash. Heroku logs says start script is missing in package.json, but since I'm deploying docker project, I couldn't do "start": "docker-compose up" there, could I?
#.gitlab-ci.yml
stages:
- deploy
before_script:
- npm install
- bower install
dev:
stage: deploy
script:
- docker-compose run nginx run-test
- gem install dpl
- dpl --provider=heroku --app=xixi-web-dev --api-key=$HEROKU_API_KEY
only:
- dev
# docker-compose.yml
app:
build: .
volumes:
- .:/code:ro
expose:
- "3000"
working_dir: /code
command: pm2 start app.dev.json5
nginx:
build: ./setup/nginx
restart: always
volumes:
- ./setup/nginx/sites-enabled:/etc/nginx/sites-enabled:ro
- ./dist:/var/app:ro
ports:
- "$PORT:80"
links:
- app
I don't want to use heroku docker plugin, because it seems less flexible, I can't create a app.json because I don't want to use an existing docker image for my app. Instead, I define custom Dockerfiles for app and nginx used in docker-compose.yml
Now it seems that heroku wouldn't detect my project as a docker project unless I deploy it by using heroku docker plugin, but as I mentioned above, I can't do that. Then is there any docs I'm missing on heroku or gitlab could help me out? Or do you have any idea that might be helpful? Thanks a lot!
OK, seems that heroku docker:release is required. I ended up installing heroku cli and heroku docker plugin on my CI server and use heroku docker:release --app app to release my app

Resources