How can I host concurrent environments of a single application of on AppHarbor?
We are currently in development so a single environment of DEBUG is sufficient, with builds triggered from the develop branch.
Soon we will need a PROD environment, triggered by changes to the master branch (or perhaps manually).
How can I configure AppHarbor such that both the DEBUG & PROD environments are accessible at any given time for a single application?
With hostnames such as:
http://debug-myapp.apphb.com
http://prod-myapp.apphb.com
For now you will have to create two applications, one for your debug environment and one for your production environment. You can also set up the tracking branch for each application. Here is a link where we describe how to manage environments.
Related
I have a general question about good practices and lets say way of work between docker and IDE.
Right now i am learning docker and docker compose, and i must admit that i like the idea of containers! Ive deployed my whole spring boot microservices architecture on containers, and everything is working really well!
The thing is, that in every place of properties when i am declaring localhost address, i was forced to change localhost to custom container names, for example localhost:8888 --> naming-server:8888. It is okay for running in containers, but obviously when i am trying to run this on IDE, it will fail. I like working/optimizing/debugging microservices in IDE, but i dont want rebuilding image and returning whole docker-compose every time i made a tiny small change.
What does it look like in real dev?
Regards!
In my day job there are at least four environments my code can run in: my desktop development environment, a developer-oriented container environment, and pre-production and production container environments. All four of these environments can have different values for things like host names. That means they must be configurable in some way.
If you've hard-coded localhost as a hostname in your application source code, it will not run in any environment other than your development system, and it needs to be changed to a configuration option.
From a pure-Docker point of view, making these configurable via environment variables is easiest (and Spring can set property values from environment variables). Spring also has the notion of a profile, which in principle matches the concept of having different settings for different environments, but injecting a whole profile configuration can be a little more complex at deployment time.
The other practice I've found helpful is to have the environment variable settings default to reasonable things for developers. The pre-production and production deployments are all heavily scripted and so there's a reasonably strong guarantee that they will have all of the correct environment variables set. If $PGHOST defaults to localhost that's right for a non-Docker developer, and all of the container-based setups can set an appropriate value for their environment at deploy time.
Even though our actual deployment system is based on containers (via Kubernetes) I do my day-to-day development in a mostly non-Docker environment. I can run an individual microservice by launching it from a shell prompt, possibly with setting some environment variables, and services have unit tests that can run just on the checked-out source tree, without needing any Docker at all. A second step is to build an image and deploy it into the development environment, and our CI system runs integration tests against the images it builds.
This is a bit theoretical, but I'll try to explain my setup as much as I can:
1 server (instance) with a self-hosted gitlab
1 server (instance) for development
1 server (instance) for production
Let's say in my gitlab I have a ReactJs project and I configured my gitlab-ci.yml as following:
job deploy_dev Upon pushing to dev branch, the updates will be copied with rsync to /var/www/html/${CI_PROJECT_NAME} (As a deployment to dev server)
The runner that picks up the job deploy_dev is a shared runner installed on that same dev server that I deploy to and it picks up jobs with the tag reactjs
The question is:
If I want to deploy to production what is the best practice should I follow?
I managed to come up with a couple of options that I thought of but I don't know which one is the best practice (if any). Here is what I came up with:
Modify gitlab-ci.yml adding a job deploy_prod with the same tag reactjs but the script should rsync with the production server's /var/www/html/${CI_PROJECT_NAME} using SSH?
Set up another runner on production server and let it pick up the jobs with tags reactjs-prod and modify gitlab-ci.yml to have deploy_prod with the tag reactjs-prod?
You have a better way other than the 2 mentioned above?
Last question (related):
Where is the best place to install my runners? Is what I'm doing (Having my runners on my dev server) actually ok?
Please if you can explain to me the best way (that you would choose) with the reasons as in pros or cons I would be very grateful.
The best practice is to separate your CI/CD infrastructure from the infrastructure where you host your apps.
This is done to minimize the number of variables which can lead to problems with either your applications or your runners.
Consider the following scenarios when you have a runner on the same machine where you host your application: (The below scenarios can happen even if the runner and app are running in separate Docker containers. The underlying machine is still a single point of failure.)
The runner executes a CPU/RAM heavy job and takes up most of the resources on the machine. Your application starts experiencing performance problems.
The Gitlab runner crashes and puts the host machine in an inoperable state. (docker panic or whatever).
Your production app stops functioning.
Your app brakes the host machine (Doesn't matter how. It can happen), your CI/CD stops working and you can not deploy a fix to production.
Consider having a separate runner machine (or machines. Gitlab runner can scale horizontally), that is used to run your deployment jobs to both dev and production servers.
I agree with #cecunami's answer.
As an example, in our Org we have a dedicated VM only for the runner, which is explicitly monitored by our teams.
Since first creating the machine, the CPU, RAM and storage demand has grown massively, thus why the infrastructure is to be separated.
I was just checking on how pipelines work in Heroku. I want the staging and production apps to be the same except that they should access different API endpoints.
How could I achieve that?
Heroku encourages getting configuration from the environment:
A single app always runs in multiple environments, including at least on your development machine and in production on Heroku. An open-source app might be deployed to hundreds of different environments.
Although these environments might all run the same code, they usually have environment-specific configurations. For example, an app’s staging and production environments might use different Amazon S3 buckets, meaning they also need different credentials for those buckets.
An app’s environment-specific configuration should be stored in environment variables (not in the app’s source code). This lets you modify each environment’s configuration in isolation, and prevents secure credentials from being stored in version control. Learn more about storing config in the environment.
On a traditional host or when working locally, you often set environment variables in your .bashrc file. On Heroku, you use config vars.
In this instance you might use an environment variable called API_BASE that gets set to the base URL of your staging API on your staging instance and to the base URL of your production API in production.
Exactly how you read those values depends on the technology you're using, but if you look for "environment variables" in your language's documentation you should be able to get started.
We have two environments, DEV and PROD. Each environment has 3 nodes:
DEV
devapp01 (Tomcat)
devapp02 (Tomcat; identical to 01 and load-balanced)
devdb01 (MySQL)
PROD
app01 (Tomcat)
app02 (Tomcat; identical to 01 and load-balanced)
db01 (MySQL)
The Tomcat instances serve WARs that are produced by a CI build.
We need the software stack on all DEV machines to be configured identically to those on PROD. We have set up a simple Chef server to manage configs on all nodes, and have created recipes for app and DB servers.
On the Chef server, we currently have an auto-update feature that runs every 30 minutes to check all nodes and make sure they're in synch with their respective recipes. The way our in-house "chef" (a sys admin) set everything up, there's a part of a recipe that does an existential check on Tomcat's webapps directory to determine whether an update should be performed or not. In other words, if Tomcat's webapp's directory has a WAR inside of it already, then when the auto-update runs every 30 mins, since webapps isn't empty, Chef won't go out to the CI server and pull in the new WAR.
To combat this, our chef cooked up a "clean slate" recipe that will first delete exploded WARs from Tomcat's webapps dir. So as long as that recipe executes first, TOMCAT_HOME/webapps will be executed before the Tomcat check; it then removes itself from the Chef runlist. His reasoning for this removal was that - in production - if we are
always deleting Tomcat's webapps dir then we will be redeploying prod nodes every 30 minutes.
So on DEV, we do want every CI build to produce a new WAR being deployed to our Tomcat instances (devapp01/02). In PROD, we want to manually kick off the deployment, which according to how this chef has configured everything, involves manually adding the clean slate recipe so that the CI server can deploy the new WAR.
I'm wondering how other people/teams have used CI and Chef in conjunction in the past, and if they ran into similar issues. My concrete question is: how can we accomplish letting CI drive all the DEV deployments, but still make PROD deployments a manual process?
You should take a look at CloudMunch for this as this is one of the scenarios that has been envisaged already there.
CloudMunch integrates natively into chef server and provides a manual workflow to allow deployment into staging or production environments.
Disclaimer: I work at CloudMunch.
The easiest way is to have separate Chef organizations (or even servers) for dev and prod. This gives you strong separation by default. You CI pushes to dev, prod pushes are done purposefully.
Chef Inc is also working on Policyfiles, which are an interesting replacement for Environments that shouldn't suffer from the same metadata/cookbook version leakage problems.
My office is growing and ive been tasked to build out the IT for our web development.
Whats the best tool/setup for doing web development in a group setting? The requirements are a centralized code repository, a location to test development code on, and finally a way to push tagged code out to a staging server. What im thinking is svn/redmine for code repo, each user has an account on a central development machine to allow for ssh access(eclipse over ssh) and their own virtual host on the dev server which gives everyone a centralized development sandbox. Code is written and tested on this dev box then checked back into svn and later tagged and pushed out to the staging server. Yeah? Thoughts comments or recommendations?
*Also, in a dev environment what is the best way to handle databases? Is it wise to pull from the production database? Also should each developer have his/her own db or work off a master db?
**We are building a magento application and also have some custom backoffice tools that run on cakePHP.
Although this subject is off-topic in StackOverflow and flagged so then you need to concentrate on following areas:
VERSION-CONTROL
GIT has all the glory and you don't need your own box for this as https://bitbucket.org/ offers unlimited data and private/public repos and you can set your codebase there. http://github.com is also powerful and de facto most popular version-control oriented tool out there although it comes for a small price
so your master branches live in your version control and your devs will checkout frpom there and commit to it as well
your deployment tools will deploy data to your live and staging environments from your master
ENVIRONMENTS
usually three are used LIVE, STAGE, DEV
LIVE is well live and only approved code gets deployed there
STAGE is pre-live environment and should be exact replica environment according to LIVE so all things can be tested there by merchant
DEV is cool to have exact replica but can as well be on developers local env and is ment for loose testing and experimenting
DATABASES AND DEPLOYMENT
mysql databases are pain in the ass to sync so you better have a script for it that syncs from live to others and prevent syncing from other environments to LIVE. This limitation also requires that all the configuration and content will be added from LIVE only and only then synced down the line. Every change to schema or permanent setting should be handled by update scripts (As we are talking MAGENTO CE , MAGENTO EE has migration built in)
for deployment I also suggest you to build a fabric or capistrano script that resets dev and staging environments, handles database reset and pull from LIVE DB, and imports code from central repository.
it's also a good idea to target the following everyday tasks:
clients needs to reset the stage for it's tests
project manager, developer or testers need to test so spawning a test clone should be oneclick action (take current db and code and make it live in some subfolder for specific test only) as well as deleting the test
3rd party devs might need access to specific test or dev environment (this is actual with magento as in average there are at least 10 external extensions installed in every magento store)