I understand how to create and deploy code to a staging environment on Heroku before deploying it to production. I understand that it is good to see if the code will "blow up" in a very similar environment than the one in production.
However, I can't figure out any tangible ways or mechanisms to determine if the application is broken on the staging environment.
In other words, I don't understand how having a running application on the staging environment is supposed to give me the confidence to deploy my application to production.
Hence my questions:
What additional steps are there to do on the staging environment on Heroku?
Are the integration tests supposed to be run on the staging environment?
If I note that the application is running on the staging environment, is it good enough?
A staging environment is intended just for that - to stage, or to "rehearse" an app version, before you promote it to production.
Heroku pipelines enable you to deploy a staging app in an environment as identical as possible to your production app, to reduce "but it worked in my dev environment" problems.
What you do with your app on the staging environment is up to you. You can consider using various simulation tools to simulate live users who would be accessing your production app. You might have migrations that you need to perform on your production app's data, so you could run them first in your staging app. You can have testers working with your staging app to verify everything works as intended before promoting to production.
Bottom line: Whatever you can do with your production app, you should be able to do with your staging app. Therefore, you should strive to achieve a deployment flow whereby BEFORE promoting a version to production, whatever scenarios may occur on production should first be tested on staging. How you achieve that is up to you.
What additional steps are there to do on the staging environment on Heroku?
Visually catch any broken CSS + HTML (which is a very difficult task with currently testing tools)
You might have other members in your team (SEO specialist, Marketing Manager, etc) which wants to inspect text, meta tags, etc. Having a staging environment will just make their life much easier as they can navigate and probe, propose improvements, etc
In your development machine you and other team members may have a different environment than you have running on production (different DB OS versions and third party services (.i.e Elasticsearch, Redis, etc) just to name a few.
You want to test deployment itself (you might have a series of tasks that are triggered during deployment)
The list of benefits and advantages of having a staging environment as close as possible as your production environment goes on...
Are the integration tests supposed to be run on the staging environment?
Normally you run your automated tests in a CI server (not on staging). Staging is just there to visually test and maybe to catch an error not covered yet by your test scenarios
If I note that the application is running on the staging environment, is it good enough?
Good enough is difficult to say (your staging environment may not be a perfect copy of your production environment), but if it is well tested, it should be very close to what you expect to see running in production
Related
In Google's latest docs, they say to test Go 1.12+ apps locally, one should just go build.
However, this doesn't take into account all the routing etc that would happen in the app engine utilizing the app.yaml config file.
I see that the dev_appserver.py is still included in the sdk. But it doesn't seem to work in Windows 10.
How does one test their Go App Engine App locally with the app.yaml. ie: as an actual emulated app engine app.
Thank you!
On one hand, if your application consists of just the default service I would recommend to follow #cerise-limón comment suggestion. In general, it is recommended for the routing logic of the application to be handled within the code. Although I'm not a Go programmer, for single service applications that use static_files and static_dir there shouldn't be any problems when testing the application locally. You might also deploy the new version without promoting traffic to it in order to test it as explained here.
On the other hand, if your application is distributed across multiple services and the routing is managed through the dispatch.yaml configuration file you might follow two approaches:
Test each service locally one by one. This could be the way to go if each service has a single responsibility/functionality that could be tested in isolation from the other services. In fact, with this kind of architecture the testing procedure would be more or less the same as for single service applications.
Run all services locally at once and build your own routing layer. This option would allow to test applications where services needs to reach one another in order to fulfill the requests made to them.
Another approach that is widely used is to have a separate project for development purposes where you could just deploy the application and observe it's behavior in the App Engine environment. As for applications with highly coupled services it would be the easiest option. But it largely depends on your budget.
Let's assume I have staging and production versions of my application. I want staging version should be accessible only to specific people like developers, testers. Do we have that kind of configuration on Heroku like deployed address can be only accessed by users have some key.
There is nothing like this in the Heroku product at this time. You would need to handle this as part of your application in some way.
My company has decided to transition to a micro/service based architecture.
We have been doing a bunch of research for the last couple of months on exactly what the architecture of this thing is going to look like.
So far, we've settled on:
Dotnet core for service development (although being language agnostic is somewhat of an end goal)
Kafka for message brokering
Docker
Kubernetes
Ansible
We have a pretty basic proof of concept working, which seems to have ticked all the right boxes with the management team, and is an absolute joy to work with.
My next task is to investigate options for how the development workflow is actually going to work. They are already used to working in a CI/CD manner, with some of their newer products using Jenkins/Octopus Deploy.
My question is: Do any of you have any firm recommendations for setting up a CI/CD pipeline when deploying to a Kubernetes cluster?
A list of must haves is:
Multiple environments i.e. Integration, Test, UAT, Staging, Production.
A means through which different business units can uniquely handle deployments to different environments (development can only push to integration, tester into test, etc). This one is probably their biggest ask - they are used to working with Octopus, and they love the way it handles this.
The ability to roll back / deploy at the click of a button (or with as few steps as possible).
We would be deploying to our own servers initially.
I've spent the past couple of days looking in to options, of which there are many.
So far, Jenkins Pipeline seems like it could be a great start. Spinnakar also seems like a solid choice. I did read a bit into Fabric8, and while it offers much of what I'm asking, it seems a bit like overkill.
If you want to use Jenkins, Pipelines are indeed the way to go. Our setup does pretty much what you want, so let me explain how we set it up.
We use a Jenkins agent that has docker and kubectl installed. This agent first builds the docker container and pushes it to our docker registry. It will then call kubectl in various stages to deploy to our testing, acceptance and production clusters.
Different business units: in a Pipeline you can use an input step to ask whether the Pipeline should proceed or not. You can specify who may press the button, so this is how you could solve the deployment to different clusters. (Ideally, when you get to CD, people will realize that pressing the button several times per day is silly and they'll just automate the entire deployment.)
Rollback: we rely on Kubernetes's rollback system for this.
Credentials: we provision the different Kubernetes credentials using Ansible directly to this Jenkins agent.
To reduce code duplication, we introduced a shared Jenkins Pipeline library, so each (micro)service talks to all Kubernetes clusters in a standardized way.
Note that we use plain Jenkins, Docker and Kubernetes. There is likely tons of software to further ease this process, so let's leave that open for other answers.
We're working on an open source project called Jenkins X which is a proposed sub project of the Jenkins foundation aimed at automating CI/CD on Kubernetes using Jenkins pipelines and GitOps for promotion across environments.
If you want to see how to automate CI/CD with multiple environments on Kubernetes using GitOps for promotion between environments and Preview Environments on Pull Requests you might wanna check out my recent talk on Jenkins X at DevOxx UK where I do a live demo of this on GKE.
I am deploying a packaged liberty server into Bluemix that contains my application.
I want to update my application but before I do so, I'm wondering what's the best way to backup what I have currently up and running? If my update is bad, I would like to restore the previous version of my app.
In other words, what is the best practice or recommended way to update a web application running on a Liberty server in Bluemix. Do I simply keep a backup of the zip I pushed to Bluemix and restore it if something goes wrong? Or is there management capability provided by Bluemix for backup and restore?
It's understood that manual backup of the pushed zip is an acceptable strategy. Additionally, I found the Bluemix documentation Blue-green deployments to be a reasonable solution, as it's a deployment technique that utilizes continuous delivery and allows clients to rollback their app in the case of any issues.
The Cloud Foundry article Using Blue-Green Deployment to Reduce Downtime and Risk succinctly explains the deployment steps (since Bluemix is based on Cloud Foundry, the steps are similar to the Example: Using the cf map-route command steps in the previously cited Bluemix documentation).
I agree with Ryan's recommendation to use the blue/green approach, though the term may be unfamiliar to those new to cloud server deployments. Martin Fowler summarizes the problem it addresses in BlueGreenDeployment:
One of the challenges with automating deployment is the cut-over
itself, taking software from the final stage of testing to live
production. You usually need to do this quickly in order to minimize
downtime. The blue-green deployment approach does this by ensuring you
have two production environments, as identical as possible. At any
time one of them, let's say blue for the example, is live. As you
prepare a new release of your software you do your final stage of
testing in the green environment. Once the software is working in the
green environment, you switch the router so that all incoming requests
go to the green environment - the blue one is now idle.
Solving this problem is one of the main benefits of PaaS.
That said, for historical context, it's worth noting this blue/green strategy isn't new to cloud computing. Allow me to elaborate on one of the "old" ways of handling this problem:
Let's assume I have a website hosted on a dedicated server, myexample.com. My public-facing server's IP address ("blue") would be represented in the DNS "#" entry or as a CNAME alias; another server ("green") would host the newer version of the application. To test the new application in a public-facing manner without impacting the live production environment, I simply update /etc/hosts to map the top-level domain name to the green server's IP address. For example:
129.42.208.183 www.myexample.com myexample.com
Once I flush the local DNS entries and close all browsers, all requests will be directed to the green pre-production environment. Once I've confirmed all works as expected, I update the DNS entry for the live environment (myexample.com in this case). Assuming the DNS has a reasonably short TTL value like 300 seconds, I update the A record value if by IP or CNAME record value if by alias and the change will be propagated to DNS servers in minutes. To confirm the propagation of the new DNS values, I comment out the aforementioned /etc/hosts change, flush the local DNS entries, then run traceroute. Assuming it correctly resolves locally, I perform a final double-check all is well in the rest of the world with the free online DNS checker (e.g., whatsmydns.net).
The above assumes an update to the public-facing content server (e.g., an Apache server connecting to a database or application server); the switch over from pre-production to production is more involved if the update applies to a central database or similar transactional data server. If it's not too disruptive for site visitors, I disable login and drop all active sessions, effectively rendering the site read-only. Then I go about updating the backend server in much the same manner as previously described, i.e., switching a pre-production green front end to reference a replication in the pre-production green backend, test, then when everything checks out, switch the green front end to blue and re-enable login. Voila.
The good news is that with Bluemix, the same strategy above applies, but is simplified since there's no need to fuss with DNS entries or separate servers.
Instead, you create two applications, one that is live ("blue") and one that is pre-production ("green"). Instead of changing your site's DNS entries and waiting for the update to propagate around the world, you can update your pre-production application (cf push Green pushes the new code to your pre-production application), test it with its own URL (Green.ng.mybluemix.net), and once you're confident it's production-ready, add the application to the routing table (cf map-route Green ng.mybluemix.net -n Blue), at which point both applications "blue" and "green" will receive incoming requests. You can then take the previous application version offline by unmapping it (cf unmap-route Blue ng.mybluemix.net -n Blue).
Site visitors will experience no service disruption and unlike the "old" way I outlined previously, the deployment team (a) won't have to bite their nails waiting for DNS entries to propagate around the world before knowing if something doesn't work and (b) can immediately revert to the previous known working production version if a serious problem is discovered post-deployment.
You should be using some sort of source control, such as Git or SVN. Bluemix is nicely integrated with IBM DevOps Services (IDS) which can leverage git or an external Github repo to manage your project. When you open your app's dashboard, you should see a link in the upper right-hand corner that says "ADD GIT". That will automatically create a git repo for your project in IDS.
Using an SCM tool, you can manage versions of your code with relative ease. IDS provides you with an ability to deploy directly to Bluemix as part of your build pipeline.
After you have your code managed as above, then you can think about green/blue deployments, etc. as recommended above.
I have an app running happily on Heroku, but it's registered as a 'development' app, and I can't for the life of me find any formal definition of what a 'production' app is, despite the apps dashboard and the status page making a very clear distinction between the two.
I have come across this explanation of the status of the two, which suggests that the difference is implicit (based on usage) rather than explicit (based on some configuration / setup):
Production issues are those that affect running, stable, production applications that have at least two web dynos and use a production-grade database (or no database at all). Includes dynos, database, HTTP caching, other platform components (DelayedJob workers, scheduler, etc.), and routing.
Development issues are those that affect the health of deployment workflow and tools. Includes deployment (git push, gem installation, slug compilation, etc.), general git activity, command line gem/API (scaling up/down, changing configuration, etc.), and related services (rake, console, db push/pull with TAPS, etc.). Development also includes issues specific to the operation of non-production applications such as unidling free 1-dyno apps and the operation of development databases.
Even these explanations reference a mysterious difference between development and production databases, although there is no corresponding explanation of the difference anywhere. Is the $9pcm 'Basic' Postgres plan a 'production' database?
[UPDATE]
There is now a 'Run Production Check' link on the app dashboard within your Heroku account that shows the steps to determine your app status. Screenshot attached:
I've been trying to find the same answer to your question.
So far from what I can glean Shared / Dev / Starter databases via Heroku Postgresql Database plans are not considered production and only Crane and higher are consider production grade database plans.
The Heroku Postgres production tier starts with Crane and extends through to the Mecha plan. Shared, Dev and Starter plans are not production databases.
https://devcenter.heroku.com/articles/maximizing-availability
In addition, it seems that Heroku has a plugin to check to see if your app meets their guidelines:
https://github.com/heroku/heroku-production-check
I'm not sure if you scale up and down web dynos from 2 (during the day) and 1 (at night) with a Crane database if that would be considered "production" in Heroku's eyes.
Hope that helps!