Question
Can anyone explain why it would be better to choose the puppet or chef vagrant provisioners, rather than the shell provisioner?
Background
I'm in the process of getting started with Vagrant. One of the things I'm having trouble with is deciding which provisioner to use. So far, I've had some success using the shell provisioner, but it has been more work than I expected to get it to run reliably.
At the moment, I'm not familar with ruby, puppet or chef, but I'm happy to learn any or all of them if I have to. My early experience playing with puppet and chef is that if someone else has a recipe that does exactly what you want, it works really well, but doing something non-standard means falling back coding up solution in ruby.
I'm aware of articles comparing puppet and chef, and I'm less worried about which of them to use, rather than knowing when and why I should use them at all.
Full disclosure: I'm a Puppet Labs employee. But I chose Puppet as a product over 2 years before joining them.
I would recommend that you use Puppet or Chef over shell if your configurations are going to a) have any degree of complexity and b) going to change over time - or you expect your installation environment itself to change in a way that might alter the way your deployment performs. Your scripts may be very good, but ultimately, unless you are following terrific programming practices around them, testing and QA'ing them, etc they are going to fail at some point.
There's an entire body of literate around DevOps discussing this notion, but it comes down to the principle of "technical debt" - we tend to do things the easy way now, and thus perceive them as simpler, at the cost of increasing complexity and difficulty later.
One of Puppet's strengths is its deterministic nature - the manifest you write must be able to be programmatically transformed by Puppet into a model of the server you are building. This is perceived by people as being more "difficult" but I would argue that the difficulty is lessened if you average it out along the curve of your technology's lifecycle. In other words, Puppet forces you to do your thinking now, but then deploy to scale with ease, rather than thinking later and re-engineering as you go. Pay in cash now, rather than by credit, with interest, later.
If you're purely pulling down other peoples' manifests, you're going to run into trouble at some point - although we would like it not to be so, working with Puppet today that's certainly the case, because they are writing them to address the general case, and not your particular system. Many general-purpose manifests become useful only when you reach a better understanding of Puppet.
So rather than start there, I'd work my way through the excellent Learning Puppet guide to start to grasp the basics. Puppet's learning curve is steep, but it levels off after a short while.
There are other reasons to use other provisioners or tools, but I'd surely argue that you are better with Puppet or Chef than trying to ensure that your shell scripts are doing exactly what you think they are supposed to do, for as long as you need to spawn new environments.
Ah, with the freedom of choice comes the complication of choosing what is right for you.
Chef Solo - Chef solo is most ideal if you’re just getting started with chef or a chef server is simply too heavy for your situation. Chef solo allows you to embed all your cookbooks within your project as well, which is nice for projects which want to keep track of their cookbooks within the same repository. Chef solo runs standalone – it requires no chef server or any other server to talk to; it simply runs by itself on the VM.
Chef Server - Chef server is useful for companies or individuals which manage many projects, since it allows you to share cookbooks across multiple projects. The cookbooks themselves are stored on the server, and the client downloads the cookbooks upon running.
Puppet - The Puppet provisioner runs stand-alone Puppet manifests that are stored on the server and downloaded to the client VM when it is created. The provisioner does not require a Puppet server and runs on the VM itself.
Puppet Server - The Puppet Server provisioner connects to a Puppet server and configures your client VM using node configuration on that server.
Other tools, shell scripts, etc. - Do you use something other than that which is built into Vagrant?
Provisioners are simply subclasses of Vagrant::Provisioners::Base, meaning you can easily build your own, should the need arise.
You can also check out the documentation, docs.vagrantup.com/v2
I would choose the Shell provisioner, then let the shell script clone your puppet/chef repository from github or bitbucket. The script can setup a ssh key to allow automated git clone. The benefits are most cloud providers support this as well so you can use the same script.This blog is explains git, puppet and vagrant well, one man and the cloud blog
Related
I'm currently looking for a provisioning-solution to deploy, configure and customize VMs in a vSphere / vCenter environment automatically. By doing that, I would like to adopt some changes to each VM individually (e.g. by using different licence keys for different software products).
During my research, I found that vagrant in combination with the vagrant-vcenter plugin with the help of puppet and git (by having different branches for different kinds of VMs) is doing exactly what i want to achieve:
https://github.com/gosddc/vagrant-vcenter
Unfortunatly, this plugin is immature and still in a beta-state.
Does anyone of you know a suitable alternative (could also be commercial)?
I searched for the same since few months. There is no other way to do that currently except using the plugin and modifying it according to your requirements. There is no other commercial solution currently in place. I used the both the plugins vagrant-vcenter and vagrant-vcloud and I was able to do the basic provisioning stuff without issues. You will not have any issues until you want to customize a lot during deployment. Some of the customization you can achieve through scripts that you can run using puppet after the VM is provisioned on vCenter.
There are several libraries out there to interact with vCeneter. The one I use is https://github.com/rlane/rbvmomi. The code base is somewhat old, but so is vCenter. The Vagrant implementation is great for dev, but has several issues going to full blown staging or production environment. For the latter, a library to interact with the underlying API of vCenter such as rbvmomi is preferred.
I'm having some problems with the "development environment should be as close as possible to the production environment".
(Production machine's operating system is Linux.)
My understanding of development steps (roughly):
code, compile, test/run, repeat
"Normally" I would go through these on my own machine, then push the code to CI for testing, and possibly deploy. The CI would be responsible for running the tests in an environment that matches production, this way if the tests pass, it's safe to assume that the code works in production as well.
The problem of a larger environment
☑ Database - of some kind.
☑ Job Processing Pool - for some long-running background tasks.
☑ User Account Management - used by other systems as well.
☑ Centralized Logging - for sanity.
☑ Forward Proxy - to tie individual http-accessible services under the same url but different paths.
☐ And possible other services or collections of services.
Solutions?
All on my own machine? No way in hell.
All on a virtual machine? Maybe, but security-wise if this setup was supposed to mirror the prod.env., and the prod.env. was like this, well.. that might not be such a good idea in case of a breach.
Divide by responsibility and set them up on multiple virtual machines? Who's gonna manage all those machines? I think it's possible to do better than this.
Use containers such as Docker, or slap similar together by yourself? Sounds good: (Possibly:) very fast iteration cycles, separation of concern, some security by separation, and easy reproducibility.
For the sake of simplicity, let's say that our containerization tooling of choice is Docker, and we are not going to build one ourselves with libvirt / lxc tooling / direct kernel calls.
So Docker it is, possibly with CoreOS or Project Atomic. So now there is a container for an application (or multiple applications) that has been separated from the rest of the system, and can be brought up nearly identically anywhere.
Solution number 1: Production environment is pretty and elegant.
Problem number 1: This is not development environment.
The development environment
Whatever the choice to not having to sprinkle the production environment into my own machine, the problem remains the same:
Even though the production environment is correctly set up, I have to run the compilation and testing somewhere, before being able to deploy (be it to another testing round by CI or whatever).
How do I solve this?
Can it really be that the proper way to solve this is by writing code on my own machine, having it synchronized/directly visible in a virtualized-mirrored-production-like environment, which automates running of the tests?
What happens when I don't want to run all the tests, but only the portion that I'm writing right now? Do I edit the automated compilation process every time? What about remote debugging, since multiple systems must be orchestrated to run in the correct way, and debugging must attach in-between to one of the programs. Not to mention the speed of "code, test" cycle, which would be _very_ slow.
This sounds helluvalot like CI, but multiple developers can't all use the same CI and modify it, so they probably have to have this setup on their own machines.
I was also thinking that the developers could each use a completely virtualized os that contained all the development tools and was mirrored environment-wise with the production, but that would force veteran users to adopt the tooling of the virtual development environment, which doesn't sound such a good idea.
I am looking for some modern Capistrano 2.x / Deprec recipes for setting up toosl ike MongoDB, Xapian and Xapian-fu, and other tools (Redis, Memcache/d, etc)
Anyone know if Deprec is still active enough to have some recipes for some of these modern tools
Thanks
Charles
I would suggest looking into using Chef or Puppet to manage the tool/system configuration and continue to use Capistrano for app deployment.
There are modules to integrate Capistrano with either Chef or Puppet, eg:
capistrano-chef
capistrano-puppet
The Chef community is growing quickly but Puppet has been around for longer. The community contributed Puppet recipes tend to have broader O/S support, but Chef is catching up.
Either could be a great choice but discussing the merits of each for this question would lead to extended debate :).
A few perspectives:
Puppet vs Chef - Fight!
Why We Chose Chef
deprec.org and the deprec-users discussion group both seem rather dormant at the moment.
Imagine you're going to manage a number of servers with a number of different services that's used by a number of people. Now say you want to reconfigure or replace some software on one of those servers. Obviously you don't want to work on servers that are in production.
If this was a code change, as a developer, I would make the change on my local development machine, test it locally and commit the change to a version control system. The changes could then be deployed in a staging environment, tested further and finally deployed in a production environment. It would also be easy for me to roll back, if necessary.
Generally, or specifically, how do you achieve this in system administration?
(The first thing that comes to mind is to use virtual machines and put virtual machine images in version control, but I'm sure there is a lot of literature and clever solutions I'm not presently aware of.)
Use chef or puppet to enforce machine configurations, and place their cookbooks and recipes under version control. Yes vms would make things easier but even physical server provioning can be controlled by kickstart or preeseed which can again be version controlled.
I have a custom build and deployment script which work over SSH and deploy to servers (on running MacOS). The bash script does a lot of simple things like copying files, backing up the old ones and applying the correct SQL scripts for a forward moving database. But there are some advanced things like starting a remote SQL upgrade procedure which can be disconnected from and once the deployment script is started again it only goes forward if the SQL script has been applied completely (in short there is some flow control happening and bash is not really ideal for such stuff)
The script is already huge and is a mess since bash is not meant for such kind of detailed logic. Can you recommend some tools, libraries which would make things easier.
For what you tell us, I think you need a deployment tool, rather than a configuration management tool.
To simplify, I'll distinguish the two like this:
A deployment tool is a 'push' tool: When you press the button, the required actions are run to make the deployment. It's a one-step process (it can have multiple actions, but it's launched once).
A configuration management tool is usually a 'pull' tool, where your servers periodically check if their configuration is exactly as the CM server tells them to be - and apply changes, if needed. You configure your servers once, and after that the system assures that all is as it should be. It is also a great tool to easily clone systems.
For deployment tools, I personally know Fabric, a great Python tool. But there is also Capistrano in the Ruby world. I don't know of any others.
For CM tools, Puppet and Chef seem to be the preferred choice of people nowadays. Cfengine is an older tool, which had some problems (I don't know if that has changed).
Here are my recommendations:
Puppet
Chef
cfengine
These are all free (as in beer) and allow you to do what you're wanting. They will require you to adapt your current bash script into modules to fit their design/framework. It's a bit of work, but in the long run it tends to be better since the frameworks take care of error checking, converging configurations and a lot of other things you'd have to manually insert into your own code were you doing this yourself.
I've also used Opsware previously for this sort of thing, but that costs a fair bit of cash and, for what you're trying to do, does not offer significantly more benefit.
In some cases moving from a bash-script to an complete solution is not as straightforward as many cloudservices claim.
With 'dont try new things when your on a deadline' in mind:
it could also be a good timing to refactor your bashscripts.
I have done automated, repeatable deployments in the past using PaaS or just using GIT/SVN hooks using deployogi (which is bash) : https://github.com/coderofsalvation/deployogi
I understand your situation, but Im not sure whether its fair to say that the bash-language implies 'a mess' and 'complex'.
Every language allows to hide complexity no?
I guess code (in whatever language) gets overly complex when time does not allow us to refactor :)
PaaS is great. But always needed? I think not.