We will be migrating from our old servers to new ones and now the question has come up: should we keep 3 servers:
testing
acceptance (also known as staging)
production
Or should we lump the first together, and just get two servers. The software we deploy is not very complicated, it's small stand-alone programs that move and process various data-files at scheduled intervals.
In the year+ i have been working here I was never in the situation I actually required separate testing and acceptance environments. Usually the programs have been unit tested and integration tested (on a separate build-server) before they go for a final test on the acceptance environment.
However at my previous place of work I do remember missing the additional server. We would be doing final tests before the release but this would then mean we could not test or showcase a branch with new features.
Such a situation is less likely at my current place of work because our team is very small: 1 dev and 2 testers.
So under what circumstances should we have more then one server for testing and acceptance?
I'm thinking:
When the software you are making only fully functions on the server and running it locally is impossible or of limited value
When the team is large enough for different people to work on different versions of the same product
When testing takes up all the resources on one server.
Am i missing something in this list?
What was the reason for having both environments in the first place?
Reasons for having both environments could also be:
You are required to keep an environment running the same version as
Production, so you can investigate/reproduce production issues - and
thus have a potential pipeline for testing hotfixes.
Maybe one of the environments are dedicated for e.g. performance tests or whatever.
Or perhaps your end users need a place where they can be educated,
while not disturbing ongoing development and testing.
That was my 10 cents.
Related
I hear these terms together and wonder what is the difference? How are they related to continuous builds and continuous deployments?
Continuous integration / continuous builds is all about getting developers to commit code to a source code repository little and often (and get the latest version from the repository, so any further changes are based on other developers recent changes). This reduces the time wasted on merge resolution, as it's easier to merge in this case.
The process is best automated using a build server, which can also run any Unit Tests. Feedback is then provided to the developers in the case of a build / test failure, so that any issues can be fixed quickly.
Continuous deployment involves the automated deployment of the build artefacts from the build process onto the test and production environments. To mitigate the risk involved with this, people often use feature toggles to separate the release (in a controlled way) from the deployment.
Continuous delivery is less about the technology and more about the organisations approach to software delivery (although is does make heavy use of automation).
DevOps is a much larger area that generally emphasises breaking down barriers between developers and operations teams, and getting them to collaborate in a way where they benefit from combined skills. More automation of environment provisioning, build deployment, monitoring (and reacting automatically to problems and scalability), and in some cases software defined networks will come out of this in a company. In some organisations, dedicated DevOps team(s) has been created.
Continuous Delivery (CD) is a concept that was first described in the 2010 book co-authored by Jez Humble and David Farley, both of ThoughtWorks.
Continuous Integration and Continuous Delivery often get confused with one another, but there are some key differences:
CI can be done by one dev where CD requires team collaboration
CD cannot be done without CI
CD is a linear journey where CI is a continuous feedback (build) loop moving CD forward
With CD you are always ready to push to prod
CI allows you to check your code into repo multiple times so you can detect your issues early on
Here is a quote from Martin Fowler:
"Continuous Integration is a software development practice where
members of a team integrate their work frequently, usually each person
integrates at least daily - leading to multiple integrations per day.
Each integration is verified by an automated build (including test) to
detect integration errors as quickly as possible. Many teams find that
this approach leads to significantly reduced integration problems and
allows a team to develop cohesive software more rapidly."
The main difference between Continuous Delivery and Continuous Deployment, is automation. You automate the deployment side of things. This works well if you are pushing to production multiple times a day or for a variety of other reasons.
As for DevOps, that's a whole other ball of wax. People often think DevOps is a role or a tool, but it's really a culture. You don't "do" DevOps. Here's a quote from Mike Kavis that I like quite a bit:
"DevOps is a culture shift or a movement that encourages great
communication and collaboration (aka teamwork) to foster building
better-quality software more quickly with more reliability."
There's probably some ambiguity in how the continuous xxx phrases are used by different people, but I think this blog post sums it up pretty well.
http://blog.assembla.com/assemblablog/tabid/12618/bid/92411/Continuous-Delivery-vs-Continuous-Deployment-vs-Continuous-Integration-Wait-huh.aspx
DevOps is more of an overarching idea than a specific practice, a bit like Agile is the idea, and unit testing is a practice.
I'm having some problems with the "development environment should be as close as possible to the production environment".
(Production machine's operating system is Linux.)
My understanding of development steps (roughly):
code, compile, test/run, repeat
"Normally" I would go through these on my own machine, then push the code to CI for testing, and possibly deploy. The CI would be responsible for running the tests in an environment that matches production, this way if the tests pass, it's safe to assume that the code works in production as well.
The problem of a larger environment
☑ Database - of some kind.
☑ Job Processing Pool - for some long-running background tasks.
☑ User Account Management - used by other systems as well.
☑ Centralized Logging - for sanity.
☑ Forward Proxy - to tie individual http-accessible services under the same url but different paths.
☐ And possible other services or collections of services.
Solutions?
All on my own machine? No way in hell.
All on a virtual machine? Maybe, but security-wise if this setup was supposed to mirror the prod.env., and the prod.env. was like this, well.. that might not be such a good idea in case of a breach.
Divide by responsibility and set them up on multiple virtual machines? Who's gonna manage all those machines? I think it's possible to do better than this.
Use containers such as Docker, or slap similar together by yourself? Sounds good: (Possibly:) very fast iteration cycles, separation of concern, some security by separation, and easy reproducibility.
For the sake of simplicity, let's say that our containerization tooling of choice is Docker, and we are not going to build one ourselves with libvirt / lxc tooling / direct kernel calls.
So Docker it is, possibly with CoreOS or Project Atomic. So now there is a container for an application (or multiple applications) that has been separated from the rest of the system, and can be brought up nearly identically anywhere.
Solution number 1: Production environment is pretty and elegant.
Problem number 1: This is not development environment.
The development environment
Whatever the choice to not having to sprinkle the production environment into my own machine, the problem remains the same:
Even though the production environment is correctly set up, I have to run the compilation and testing somewhere, before being able to deploy (be it to another testing round by CI or whatever).
How do I solve this?
Can it really be that the proper way to solve this is by writing code on my own machine, having it synchronized/directly visible in a virtualized-mirrored-production-like environment, which automates running of the tests?
What happens when I don't want to run all the tests, but only the portion that I'm writing right now? Do I edit the automated compilation process every time? What about remote debugging, since multiple systems must be orchestrated to run in the correct way, and debugging must attach in-between to one of the programs. Not to mention the speed of "code, test" cycle, which would be _very_ slow.
This sounds helluvalot like CI, but multiple developers can't all use the same CI and modify it, so they probably have to have this setup on their own machines.
I was also thinking that the developers could each use a completely virtualized os that contained all the development tools and was mirrored environment-wise with the production, but that would force veteran users to adopt the tooling of the virtual development environment, which doesn't sound such a good idea.
So I've been agonizing over embracing a deployment/configuration management tool like Chef or Puppet for a good long while. Not because I have any hesitation about them in general, but because I don't think they are a good fit for our specific scenario.
As far as I can see, these types of tools are targeted at frequent/wide-scale deployments, where you need to roll out software to 10s-1000s of systems. In our environment, we have a collection of ~25 different web services spread across half a dozen runtimes, with 1-8 deployments of each in production currently. Our big deployment problem is that each of the services has a different deployment story, and it's entirely manual, so it tends to be time consuming and error prone. Another wrinkle is that different instances in production may be different versions of the software, so we may need to concurrently support multiple deployment stories for a single service.
So I feel like we need something more like Ant/Maven/Rake, which is customized for each service. However, my experience with those is they are generally focused on local operations, and specific to a given language/runtime.
Is there a runtime-agnostic framework for describing and orchestrating building/testing/deployment in the manner I'm interested in?
I'm sure if I hit them long enough, I could get Rake or Puppet to do these for me, but I'm looking for something built for this purpose.
(Oh, and to make things worse, everything runs on Windows)
Thanks!
Here's another alternative you might want to consider: kwatee (I'm affiliated) is a free lightweight deploiement tool which besides having a web management interface can also integrate with ant (or maven or anything else with python CLI) to automate build & deploiement on dev/test environments for instance.
One of the nice things is the web configuration interface which make it pretty easy to quickly configure your deploiment stories, i.e. which software/version goes on which server. It's often necessary to setup different parameters in configuration files depending on the target server. For that you can "templatize" your packages using kwatee variable (similar to environment variables) which are configured with different values for each server.
Software must be registered in Kwatee's repository in the form of a folder of files, or an archive (zip, tar, tar.gz, bzip2, war) or a single file (e.g. an exe). Msi's are not supported. To deploy on windows kwatee needs the servers to have either telnet/ftp or ssh/scp (there are free tools out there).
I work for a small dotcom which will soon be launching a reasonably-complicated Windows program. We have uncovered a number of "WTF?" type scenarios that have turned up as the program has been passed around to the various not-technical-types that we've been unable to replicate.
One of the biggest problems we're facing is that of testing: there are a total of three programmers -- only one working on this particular project, me -- no testers, and a handful of assorted other staff (sales, etc). We are also geographically isolated. The "testing lab" consists of a handful of VMWare and VPC images running sort-of fresh installs of Windows XP and Vista, which runs on my personal computer. The non-technical types try to be helpful when problems arise, we have trained them on how to most effectively report problems, and the software itself sports a wide array of diagnostic features, but since they aren't computer nerds like us their reporting is only so useful, and arranging remote control sessions to dig into the guts of their computers is time-consuming.
I am looking for resources that allow us to amplify our testing abilities without having to put together an actual lab and hire beta testers. My boss mentioned rental VPS services and asked me to look in to them, however they are still largely very much self-service and I was wondering if there were any better ways. How have you, or any other companies in a similar situation handled this sort of thing?
EDIT: According to the lingo, our goal here is to expand our systems testing capacity via an elastic computing platform such as Amazon EC2. At this point I am not sure suggestions of beefing up our unit/integration testing are going to help very much as we are consistently hitting walls at the systems testing phase. Has anyone attempted to do this kind of software testing on a cloud-type service like EC2?
Tom
The first question I would be asking is if you have any automated testing being done?
By this I mean mainly mean unit and integration testing. If not then I think you need to immediately look into unit testing, firstly as part of your build processes, and second via automated runs on servers. Even with a UI based application, it should be possible to find software that can automate the actions of a user and tell you when a test has failed.
Apart from the tests you as developers can think of, every time a user finds a bug, you should be able to create a test for that bug, reproduce it with the test, fix it, and then add the test to the automated tests. This way if that bug is ever re-introduced your automated tests will find it before the users do. Plus you have the confidence that your application has been tested for every known issue before the user sees it and without someone having to sit there for days or weeks manually trying to do it.
I believe logging application activity and error/exception details is the most useful strategy to communicate technical details about problems on the customer side. You can add a feature to automatically mail you logs or let the customer do it manually.
The question is, what exactly do you mean to test? Are you only interested in error-free operation or are you also concerned how the software is accepted at the customer side (usability)?
For technical errors, write a log and manually test in different scenarios in different OS installations. If you could add unit tests, it could also help. But I suppose the issue is that it works on your machine but doesn't work somewhere else.
You could also debug remotely by using IDE features like "Attach to remote process" etc. I'm not sure how to do it if you're not in the same office, likely you need to build a VPN.
If it's about usability, organize workshops. New people will be working with your application, and you will be video and audio recording them. Then analyze the problems they encountered in a team "after-flight" sessions. Talk to users, ask what they didn't like and act on it.
Theoretically, you could also built this activity logging into the application. You'll need to have a clear idea though what to log and how to interpret the data.
We know this is good to have, but I find myself justifying it to my employer. Please pitch in on why a development team needs a build server.
There are multiple reasons to use build servers. In no particular order and off the top of my head:
You simplify the developers' workflow and reduce the chance of mistakes. Your build server can take care of multiple steps such as checking out latest code, having required software installed, etc. There's no chance of a developer having some stray DLLs on their machine that can cause the build to pass or fail seemingly at random.
Your build server can replicate your target environment (operating system, etc.) and there's less of a chance of something working on developers' desktops and breaking in production.
While it's a good practice for developers to test everything they check in, sometimes they just don't. Then it's good to have the build server there to catch test errors and let the team know the product is broken.
Centralized builds provide easy access to code metrics -- which tests passed, which failed, how often, how well is your code covered by your tests, etc. Having a solid understanding of the quality state of the codebase reduces maintenance and testing costs by providing timely feedback that allows errors to be fixed quickly and easily.
Product deployment is simplified -- the developer or QA doesn't have to remember multiple manual steps. It can be easily automated.
The link between developers and QA is simplified. QA personnel can go to a known location to grab latest, propertly versioned builds.
It's easy to set up builds for release branches, providing an extra safety net for products in their release stage, when making code changes must be done with extra care.
To avoid the "but it works on my box" issue.
Have a consistent, known environment where the software is built to avoid dependencies on local dev boxes.
You can use a virtual server to avoid (much) extra cost if you need to.
ASAP knowledge on what unit tests are currently working and which do not; furthermore, you'll also know if a once passing unit tests starts to fail.
This should sum up why it is critical to have a build server:
http://www.codinghorror.com/blog/2006/10/the-build-server-your-projects-heart-monitor.html
It's a continuous quality test dashboard; it shows you statistics about how the quality of your software is doing, and it shows them to you now. (JUnit, Cobertura)
It makes sure developers aren't hamstrung by other developers breaking the build, and encourages developers to write better code. (FindBugs, PMD)
It saves you time and money throughout the year by getting better code from developers the first time - less money on testing and retesting - and by getting more code from the same developers, because they're less likely to trip each other up.
Two main reasons that non technical people can relate to:
It improves the productivity of the dev team because problems are identified earlier.
It makes the state of the project very obvious. I've shown my management the build status dashboard an now they look at it all the time.
One more thing. Something like Hudson is very simple to set up - you might want to simply run it somewhere in a corner for a while and then show it later.
This is my principal argument:
all official releases must be build in a controlled environment. No exception.
simply because you never know how the developers create their personal releases.
You also don't need to talk about build server as in "blade that costs an arm a a leg". The first build server I set up was a desktop machine that sat unplugged in a corner. It served us very well for more than 3 years.
One you have your build machine, you can start adding some features (Hudson is great) and implement everything that the other posters mentioned.
Once your build machine becomes indispensable to your organization (and everyone sees its benefits), you will be able to ask for a shiny new blade if you wish :-)
The simplest thing you can do to convince your your employer to have a build server is to tell them that they will be able to release faster and with better quality.
Faster releases come from the immediate feedback about quality of the build. If someone breaks the build, he or she can fix the broken build immediately thus avoiding a delay in the build and release schedule. Without a build server the team will have to spend time trying to find what and when happened and how to fix it.
Better quality is achieved by the build server running bug detection tools automatically every time someone check is changes into a version control system. You don't mention what is the main development language in your organization, but such tools, advanced but commercial and simple but free, exist practically for all languages. Lint, FxCop, FindBugs and PMD come to mind.
You may also check this presentation on benefits of continuous integration for a more extensive discussion.