Key-pair management for build server EC2 - amazon-ec2

I am working on bringing a bit of devops sanity to my environment. Until now, we've all been building releases on our local systems and then scp`ing these builds to the target EC2 instances. This is not ideal for these reasons:
I want to build on Linux, not on BSD, if I my target environment is Linux.
I don't want any local environment idiosyncrasies to get in the way, and I don't want to be in the business of policing developers' local environments.
Transferring builds between EC2 VMs is instantaneous, but takes potentially long time transferring from our locals systems, which are frequently on a slow public WiFi.
I have an devops AMI, that can be launched by a developer, which has all the tooling to build the product distro. The next step is for the developer to transfer that distro to another EC2 "target" EC2 instance, where it can be unpacked and run. And it's this scp hop from the build to the target instance that I am struggling with. My solutions, as I see them are:
Make each developer copy their private key to the build machine. Then, assuming that the target instance was started by the same developer, sch will be happy.
Have a shared key-pair, call it "devops". This way, I can pre-install the private key on the build AMI, and, so long as developers launch both build and target instances with those keys, everything will be transparent.
Both have their shortcomings, so I wonder what's the best practice here.

Related

How to create a customizable environment that can be rapidly distributed to a local machine?

I am looking for a way to be able to do the following:
Create an instance of Windows with installed prerequisites and configuration
An isolated environment would be recommended (As in it will not modify the existing configuration on local machine only in that VM-like environment)
Ability to use the internet within that environment
Using it sort of like a "check-point" (Start working on it, doing something wrong and being able to start once again from the instance that we created)
Ability to share the environment
Possibility of creating multiple different environments
Low disk usage if possible
Fast deployment of environment on local machine
I have looked into Docker which seems pretty good for what I need, but I want to investigate other options as well because it requires Windows 10 x64 Enterprise
.
Something that works on Windows 7/Server/8/8.1 would be nice
I would also love to get arguments on why X option is better than Y option.
Thanks in advance!
If you want a completely separate environment, creating a Virtual Machine will be worth considering.
There are products from VMware and Oracle to create your virtual machine. I have been using Oracle Virtualbox (Oracle's virtual machine software) for some time now and find it pretty useful.
With a virtual machine it addresses all your concerns:
Create an instance of Windows with installed prerequisites and
configuration - A virtual machine will run on top of your installed OS without making
any modifications in current installation
An isolated environment would be recommended (As in it will not
modify the existing configuration on local machine only in that
VM-like environment) - It runs completely isolated like a separate
machine.
Ability to use the internet within that environment - You can use
internet inside of a virtual machine
Using it sort of like a "check-point" (Start working on it, doing
something wrong and being able to start once again from the instance
that we created) - You can take a snapshot and save the state. Next time when you start the VM it will be started from this state only.
Ability to share the environment - Export a created VM and it can be
reused.
Possibility of creating multiple different environments - You can run
multiple VMs on your machine. Configure the disk usage and RAM
accordingly.
Low disk usage if possible - Configurable while creating a virtual
machine.
Fast deployment of environment on local machine - Yes, you'll need
the .iso image of your Operating System

Vagrant work flow

I read about vagrant today, I have observed few points in it.
Vagrant creates a identical development environment for everyone in the team.
A modern, complex application probably uses a number of external dependencies, like various database servers, message queue etc. Installing these on every developer's machine individually for a local development environment can be quite a difficult. Using a virtual machine image, we can distribute a per-configured environment easily.
Have a 5 minute setup for any new developer who joins our team to get up and running. This should be as easy as running a single command: "vagrant up". No installation, no troubleshooting. It should “just work” ™
I have some queries on above statements,
How can we setup an environment for new member who joins our team?
In case new developer makes some changes in his virtual setup, how will it be reflected or deployed to your virtual setup?
Will you have to do the setup again? Or will it work as similar to SVN, namely, if one developer will change it will automatically affect the other machine also?
Vagrant is a good tool for manage teamwork and create a good startup flow..
You can create a git repository with your Vagrantfile and your provisioning script.. a new developer clone your repo and start it with "vagrant up" command.. And he is ready for work with complete env.
My configuration of Vagrant is a git repository.. I use it with normale VCS.
VCS :) git commit, pull, push ecc how a normal project..

Whats the best fit for continious delivery, Vagrant or Puppet,

At present I have to deploy the latest version of my application to a QA environment which is a painful and completely manual task. Im looking at possibly using Puppet or Vagrant as a way to manage my infrastructure and spin up some virtual machines with all the components I need so I can run acceptance tests against the latest version of my software. Im looking for something that integrates with my Bamboo and will teardown these machines when im finished with them.
Theres alot of talk about Vagrant, but it seems to me that you need to first prepare a VM before using Vagrant. Is one better than the other or is anyone using both as part of there CI environment?.
Why or when you can use "and":
Puppet is a configuration management tool - forcing consistency and predictability across your cluster. Usually used in Staging/Production set ups where autoscaling of machines is used.
Vagrant is a simple framework for setting up virtual machines from a list of pre-configured "clean" installations. The first use case is a "playground" on your development machine.
Once Vagrant sets up the virtual machine it can provision it using Puppet to install all the prerequisites (e.g. your app) (just like it does in the production machines) and then you can test it.

How can I integrate a virtual machine into my automated unit tests in Visual Studio?

I've got some legacy software that I'd like to involve in an automated unit test (for testing network protocol compatibility) and because this software is old and runs in an outdated environment I'd like to encapsulate it in a virtual machine. What is the best way to control a virtual machine from a Visual Studio unit test? Once I have the vm configured and have saved the state appropriately, I will need to be able to start and stop the vm and possibly launch some programs inside the vm on command.
One consideration I do have is that I'd like for developers not to have to download the vm image if they aren't planning to run this test. The unit test may therefore have to also handle downloading the latest vm image from some location. Our convention is to tag long running tests with a special description so developers will be able to exclude this test during active development.
The virtual machine platforms provide a scripting API that let you control VMs from the command line. The VMware server docs and a video on Hyper-V Scripting are available.
You will need to include some logic in your build scripts to decide if you should execute the VM code, or just check for the presence of the VM on developers machines.
You may want to check out some of the NAnt and MSBuild task repositories for VM-related tasks to make this easier.

Creating a virtual machine image as a continuous integration artifact?

I'm currently working on a server-side product which is a bit complex to deploy on a new server, which makes it an ideal candidate for testing out in a VM. We are already using Hudson as our CI system, and I would really like to be able to deploy a virtual machine image with the latest and greatest software as a build artifact.
So, how does one go about doing this exactly? What VM software is recommended for this purpose? How much scripting needs to be done to accomplish this? Are there any issues in particular when using Windows 2003 Server as the OS here?
Sorry to deny anyone an accepted answer here, but based on further research (thanks to your answers!), I've found a better solution and wanted to summarize what I've found.
First, both VirtualBox and VMWare Server are great products, and since both are free, each is worth evaluating. We've decided to go with VMWare Server, since it is a more established product and we can get support for it should we need. This is especially important since we are also considering distributing our software to clients as a VM instead of a special server installation, assuming that the overhead from the VMWare Player is not too high. Also, there is a VMWare scripting interface called VIX which one can use to directly install files to the VM without needing to install SSH or SFTP, which is a big advantage.
So our solution is basically as follows... first we create a "vanilla" VM image with OS, nothing else, and check it into the repository. Then, we write a script which acts as our installer, putting the artifacts created by Hudson on the VM. This script should have interfaces to copy files directly, over SFTP, and through VIX. This will allow us to continue distributing software directly on the target machine, or through a VM of our choice. This resulting image is then compressed and distributed as an artifact of the CI server.
Regardless of the VM software (I can recommend VirtualBox, too) I think you are looking at the following scenario:
Build is done
CI launches virtual machine (or it is always running)
CI uses scp/sftp to upload build into VM over the network
CI uses the ssh (if available on target OS running in VM) or other remote command execution facility to trigger installation in the VM environment
VMWare Server is free and a very stable product. It also gives you the ability to create snapshots of the VM slice and rollback to previous version of your virtual machine when needed. It will run fine on Win 2003.
In terms of provisioning new VM slices for your builds, you can simply copy and past the folder that contains the VMWare files, change the SID and IP of the new VM and you have a new machine. Takes 15 minutes depending on the size of your VM slice. No scripting required.
If you use VirtualBox, you'll want to look into running it headless, since it'll be on your server. Normally, VirtualBox runs as a desktop app, but it's possible to start VMs from the commandline and access the virtual machine over RDP.
VBoxManage startvm "Windows 2003 Server" -type vrdp
We are using Jenkins + Vagrant + Chef for this scenario.
So you can do the following process:
Version control your VM environment using vagrant provisioning scripts (Chef or Puppet)
Build your system using Jenkins/Hudson
Run your Vagrant script to fetch the last stable release from CI output
Save the VM state to reuse in future.
Reference:
vagrantup.com
I'd recommend VirtualBox. It is free and has a well-defined programming interface, although I haven't personally used it in automated build situations.
Choosing VMWare is currently NOT a bad choice.
However,
Just like VMWare gives support for VMWare server, SUN gives support for VirtualBOX.
You can also accomplish this task using VMWare Studio, which is also free.
The basic workflow is this:
1. Create an XML file that describes your virtual machine
2. Use studio to create the shell.
3. Use VMWare server to provision the virtual machine.

Resources