I have been learning some new technologies for automatization of workflow. On some projects I saw people using Rakefiles. I have learned that they are used to automate the deployment/workflow etc.
On others, I saw people using Ansible.
Maybe I am misunderstanding one of those technologies, but how do they differ?
Rake is a tool most often used for administrative tasks that can be scripted for an application whether that be database migrations, data loading, export/import of data, clearing caches, etc. Ansible is a provisioning tool it can configure a server, and do application deployment. You could conceivably use rake to do some of what Ansible does and vice versa, but if you were to try to recreate Ansible in a rake utility you'd be re-inventing the wheel. The closer equivalent to Anisble in the ruby/rails world is Capistrano. However, Ansible is a more generically applicable tool and Capistrano is more a specialty tool for rails applications (IMO).
Related
I am aware of the process to install WAS 8.5.5.x and 9.0.x versions using IM response file(s) but would like to know best practices and recommendations to perform WAS installation and upgrade on more than one server, to avoid manual errors and reduce time.
I am open to use to Ansible, Puppet or any other orchestration tools as well, but would like to know possible options if we are not allowed to use these tools.
Ultimate goal is to automate most of the setup/upgrade steps, if not all of them since when dealing with bunch of servers.
Thanks
Assuming you are referring to WebSphere Application Server traditional, take a look at the approaches described here, https://www.ibm.com/support/knowledgecenter/SSEQTP_9.0.0/com.ibm.websphere.installation.base.doc/ae/tins_enterprise_install.html, especially if you are working with larger scale deployments.
Consider creating master images and distributing them in a swinging profile-type setup. They make it easier and faster to install and apply updates since you only need to create images once and distribute many times. You have consistency across systems too.
You can then automate with your preferred automation technology.
We use ansible, simple and effectively.
True, you must of course develop a playbook that will be able to do all this.
So I've been agonizing over embracing a deployment/configuration management tool like Chef or Puppet for a good long while. Not because I have any hesitation about them in general, but because I don't think they are a good fit for our specific scenario.
As far as I can see, these types of tools are targeted at frequent/wide-scale deployments, where you need to roll out software to 10s-1000s of systems. In our environment, we have a collection of ~25 different web services spread across half a dozen runtimes, with 1-8 deployments of each in production currently. Our big deployment problem is that each of the services has a different deployment story, and it's entirely manual, so it tends to be time consuming and error prone. Another wrinkle is that different instances in production may be different versions of the software, so we may need to concurrently support multiple deployment stories for a single service.
So I feel like we need something more like Ant/Maven/Rake, which is customized for each service. However, my experience with those is they are generally focused on local operations, and specific to a given language/runtime.
Is there a runtime-agnostic framework for describing and orchestrating building/testing/deployment in the manner I'm interested in?
I'm sure if I hit them long enough, I could get Rake or Puppet to do these for me, but I'm looking for something built for this purpose.
(Oh, and to make things worse, everything runs on Windows)
Thanks!
Here's another alternative you might want to consider: kwatee (I'm affiliated) is a free lightweight deploiement tool which besides having a web management interface can also integrate with ant (or maven or anything else with python CLI) to automate build & deploiement on dev/test environments for instance.
One of the nice things is the web configuration interface which make it pretty easy to quickly configure your deploiment stories, i.e. which software/version goes on which server. It's often necessary to setup different parameters in configuration files depending on the target server. For that you can "templatize" your packages using kwatee variable (similar to environment variables) which are configured with different values for each server.
Software must be registered in Kwatee's repository in the form of a folder of files, or an archive (zip, tar, tar.gz, bzip2, war) or a single file (e.g. an exe). Msi's are not supported. To deploy on windows kwatee needs the servers to have either telnet/ftp or ssh/scp (there are free tools out there).
I am looking for some modern Capistrano 2.x / Deprec recipes for setting up toosl ike MongoDB, Xapian and Xapian-fu, and other tools (Redis, Memcache/d, etc)
Anyone know if Deprec is still active enough to have some recipes for some of these modern tools
Thanks
Charles
I would suggest looking into using Chef or Puppet to manage the tool/system configuration and continue to use Capistrano for app deployment.
There are modules to integrate Capistrano with either Chef or Puppet, eg:
capistrano-chef
capistrano-puppet
The Chef community is growing quickly but Puppet has been around for longer. The community contributed Puppet recipes tend to have broader O/S support, but Chef is catching up.
Either could be a great choice but discussing the merits of each for this question would lead to extended debate :).
A few perspectives:
Puppet vs Chef - Fight!
Why We Chose Chef
deprec.org and the deprec-users discussion group both seem rather dormant at the moment.
I have a custom build and deployment script which work over SSH and deploy to servers (on running MacOS). The bash script does a lot of simple things like copying files, backing up the old ones and applying the correct SQL scripts for a forward moving database. But there are some advanced things like starting a remote SQL upgrade procedure which can be disconnected from and once the deployment script is started again it only goes forward if the SQL script has been applied completely (in short there is some flow control happening and bash is not really ideal for such stuff)
The script is already huge and is a mess since bash is not meant for such kind of detailed logic. Can you recommend some tools, libraries which would make things easier.
For what you tell us, I think you need a deployment tool, rather than a configuration management tool.
To simplify, I'll distinguish the two like this:
A deployment tool is a 'push' tool: When you press the button, the required actions are run to make the deployment. It's a one-step process (it can have multiple actions, but it's launched once).
A configuration management tool is usually a 'pull' tool, where your servers periodically check if their configuration is exactly as the CM server tells them to be - and apply changes, if needed. You configure your servers once, and after that the system assures that all is as it should be. It is also a great tool to easily clone systems.
For deployment tools, I personally know Fabric, a great Python tool. But there is also Capistrano in the Ruby world. I don't know of any others.
For CM tools, Puppet and Chef seem to be the preferred choice of people nowadays. Cfengine is an older tool, which had some problems (I don't know if that has changed).
Here are my recommendations:
Puppet
Chef
cfengine
These are all free (as in beer) and allow you to do what you're wanting. They will require you to adapt your current bash script into modules to fit their design/framework. It's a bit of work, but in the long run it tends to be better since the frameworks take care of error checking, converging configurations and a lot of other things you'd have to manually insert into your own code were you doing this yourself.
I've also used Opsware previously for this sort of thing, but that costs a fair bit of cash and, for what you're trying to do, does not offer significantly more benefit.
In some cases moving from a bash-script to an complete solution is not as straightforward as many cloudservices claim.
With 'dont try new things when your on a deadline' in mind:
it could also be a good timing to refactor your bashscripts.
I have done automated, repeatable deployments in the past using PaaS or just using GIT/SVN hooks using deployogi (which is bash) : https://github.com/coderofsalvation/deployogi
I understand your situation, but Im not sure whether its fair to say that the bash-language implies 'a mess' and 'complex'.
Every language allows to hide complexity no?
I guess code (in whatever language) gets overly complex when time does not allow us to refactor :)
PaaS is great. But always needed? I think not.
Question
Can anyone explain why it would be better to choose the puppet or chef vagrant provisioners, rather than the shell provisioner?
Background
I'm in the process of getting started with Vagrant. One of the things I'm having trouble with is deciding which provisioner to use. So far, I've had some success using the shell provisioner, but it has been more work than I expected to get it to run reliably.
At the moment, I'm not familar with ruby, puppet or chef, but I'm happy to learn any or all of them if I have to. My early experience playing with puppet and chef is that if someone else has a recipe that does exactly what you want, it works really well, but doing something non-standard means falling back coding up solution in ruby.
I'm aware of articles comparing puppet and chef, and I'm less worried about which of them to use, rather than knowing when and why I should use them at all.
Full disclosure: I'm a Puppet Labs employee. But I chose Puppet as a product over 2 years before joining them.
I would recommend that you use Puppet or Chef over shell if your configurations are going to a) have any degree of complexity and b) going to change over time - or you expect your installation environment itself to change in a way that might alter the way your deployment performs. Your scripts may be very good, but ultimately, unless you are following terrific programming practices around them, testing and QA'ing them, etc they are going to fail at some point.
There's an entire body of literate around DevOps discussing this notion, but it comes down to the principle of "technical debt" - we tend to do things the easy way now, and thus perceive them as simpler, at the cost of increasing complexity and difficulty later.
One of Puppet's strengths is its deterministic nature - the manifest you write must be able to be programmatically transformed by Puppet into a model of the server you are building. This is perceived by people as being more "difficult" but I would argue that the difficulty is lessened if you average it out along the curve of your technology's lifecycle. In other words, Puppet forces you to do your thinking now, but then deploy to scale with ease, rather than thinking later and re-engineering as you go. Pay in cash now, rather than by credit, with interest, later.
If you're purely pulling down other peoples' manifests, you're going to run into trouble at some point - although we would like it not to be so, working with Puppet today that's certainly the case, because they are writing them to address the general case, and not your particular system. Many general-purpose manifests become useful only when you reach a better understanding of Puppet.
So rather than start there, I'd work my way through the excellent Learning Puppet guide to start to grasp the basics. Puppet's learning curve is steep, but it levels off after a short while.
There are other reasons to use other provisioners or tools, but I'd surely argue that you are better with Puppet or Chef than trying to ensure that your shell scripts are doing exactly what you think they are supposed to do, for as long as you need to spawn new environments.
Ah, with the freedom of choice comes the complication of choosing what is right for you.
Chef Solo - Chef solo is most ideal if you’re just getting started with chef or a chef server is simply too heavy for your situation. Chef solo allows you to embed all your cookbooks within your project as well, which is nice for projects which want to keep track of their cookbooks within the same repository. Chef solo runs standalone – it requires no chef server or any other server to talk to; it simply runs by itself on the VM.
Chef Server - Chef server is useful for companies or individuals which manage many projects, since it allows you to share cookbooks across multiple projects. The cookbooks themselves are stored on the server, and the client downloads the cookbooks upon running.
Puppet - The Puppet provisioner runs stand-alone Puppet manifests that are stored on the server and downloaded to the client VM when it is created. The provisioner does not require a Puppet server and runs on the VM itself.
Puppet Server - The Puppet Server provisioner connects to a Puppet server and configures your client VM using node configuration on that server.
Other tools, shell scripts, etc. - Do you use something other than that which is built into Vagrant?
Provisioners are simply subclasses of Vagrant::Provisioners::Base, meaning you can easily build your own, should the need arise.
You can also check out the documentation, docs.vagrantup.com/v2
I would choose the Shell provisioner, then let the shell script clone your puppet/chef repository from github or bitbucket. The script can setup a ssh key to allow automated git clone. The benefits are most cloud providers support this as well so you can use the same script.This blog is explains git, puppet and vagrant well, one man and the cloud blog