Multiple EC2 instances using Vagrant - ruby

I'm trying out Vagrant to provision an environment with multiple EC2 instances in AWS (using vagrant-aws plugins) and using a JSON config file for Vgrant to read from. Below is the JSON file:
macp-3:vagrant-aws sans$ cat scripts/aws.json
{
"env": "dops",
"access_key": "XXXXXXXXXXXXXXXX",
"secret_key": "hfgy5ejfkprg2432432beqo2r",
"region": "eu-west-1",
"availability_zone": "a",
"subnet_id": "subnet-0b766860",
"security_groups": [
"sg-53t48c34",
"sg-11668f7e",
"sg-4a6c8525",
"sg-75168c1e"
],
"ssh_username": "ubuntu",
"keypair": "Xdops_testKey",
"ssh_private_key": "/Users/sans/.ssh/id_rsa",
"ec2s": {
"dops-agg-001": {
"ami_id": "ami-838675f7",
"instance_type": "m3.medium",
"elastic_ip": "ture",
"tags": {
"Name": "dops-agg-001",
"Role": "sql-aggr",
"ServiceType": "database",
"NopeType": "mem_master",
"CostCentre": "QA"
}
},
"dops-nag-001": {
"ami_id": "ami-838675f7",
"instance_type": "m3.medium",
"elastic_ip": "ture",
"tags": {
"Name": "dops-nag-001",
"Role": "monitoring",
"ServiceType": "controller",
"NopeType": "nagios",
"CostCentre": "QA"
}
}
}
}
This is the Vagrantfile that I've come up with:
# -*- mode: ruby -*-
# vi: set ft=ruby :
aws_cfg = (JSON.parse(File.read("scripts/aws.json")))
Vagrant.configure("2") do |config|
config.vm.box = "dummy"
config.vm.box_url = "https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box"
aws_cfg['ec2s'].each do |node|
node_name = node[0]
node_value = node[1]
# Node specific configuration
config.vm.define node_name do |config2|
ec2_tags = node_value['tags']
# Spining up EC2 instances
config2.vm.provider :aws do |ec2, override|
ec2.access_key_id = aws_cfg['access_key']
ec2.secret_access_key = aws_cfg['secret_key']
ec2.keypair_name = aws_cfg['keypair']
ec2.region = aws_cfg['region']
ec2.availability_zone = aws_cfg['region']+aws_cfg['availability_zone']
ec2.subnet_id = aws_cfg['subnet_id']
ec2.security_groups = aws_cfg['security_groups']
#
ec2.ami = node_value['ami_id']
ec2.instance_type = node_value['instance_type']
ec2.elastic_ip = node_value['elastic_ip']
#
ec2.tags = {
'Name' => ec2_tags['Name'],
'Role' => ec2_tags['Role'],
'ServiceType' => ec2_tags['ServiceType'],
'NopeType' => ec2_tags['NopeType'],
'CostCentre' => ec2_tags['CostCentre']
}
#
override.ssh.username = "ubuntu"
override.ssh.private_key_path = "/Users/sans/.ssh/id_rsa"
end
# Final Puppet provisioning
#config2.vm.provision :puppet do |ppt|
# ppt.options = "--verbose --debug"
# ppt.manifests_path = "puppet/manifests"
# ppt.manifest_file = "nodes.pp"
# ppt.module_path = "puppet/modules"
#end
end
end
end
It's working okay but trying to spin up the boxes in parallel, instead of one-by-one:
macp-3:vagrant-aws sans$ vagrant up --provider=aws
Bringing machine 'dops-agg-001' up with 'aws' provider...
Bringing machine 'dops-mon-001' up with 'aws' provider...
[fog][WARNING] Unable to load the 'unf' gem. Your AWS strings may not be properly encoded.
==> dops-mon-001: HandleBoxUrl middleware is deprecated. Use HandleBox instead.
==> dops-mon-001: This is a bug with the provider. Please contact the creator
==> dops-agg-001: HandleBoxUrl middleware is deprecated. Use HandleBox instead.
==> dops-mon-001: of the provider you use to fix this.
....
....
I believe there are some problem with looping in the Vagrantfile? Can anyone tell me what's going wring here? Best!

This question has been answered in a Google Group.
Solution:
--[no-]parallel - Bring multiple machines up in parallel if the provider supports it.

Related

Terraform and AWS spots instances

As described in https://github.com/hashicorp/terraform/issues/17429
After 7 days the spot request is getting cancelled and the instance is still running. So when I run the "terraform apply" its trying to create a new spot. This happens with AWS provider >=1.13.0.
I'm using AWS provider 1.32.0, does anyone know a workaround to this issue? on future installation I will use the valid_until flag which will extend the request lifetime, but what about already installed spots?
Thanks
resource "aws_spot_instance_request" "cheap_worker" {
count = "${var.kube_master_spot_num}"
ami = "${data.aws_ami.nat_ami.id}"
availability_zone ="${element(slice(data.aws_availability_zones.available.names,var.kube_master_on_demand_num,var.availability_zones_num),count.index)}"
spot_price = "3"
instance_type = "${var.kube_master_type}"
subnet_id = "${element(module.aws-vpc.aws_subnet_ids,count.index + var.kube_master_on_demand_num)}" # adjusting to a case with spots & on-demand servers
vpc_security_group_ids = [ "${module.aws-vpc.cluster_sg_id}", "${module.aws-vpc.route53_sg_id}" ]
key_name = "${basename(var.local_ssh_key)}"
associate_public_ip_address = true
root_block_device = [{volume_type="gp2",volume_size="50",delete_on_termination=true}]
spot_type = "one-time"
wait_for_fulfillment = true
tags {
Name = "${var.kube_identify}-${var.kube_type}-master-${count.index}"
}
provisioner "local-exec" {
command = "sleep 120"
}
connection {
type = "ssh"
user = "ubuntu"
private_key = "${file(var.local_ssh_key)}"
}
provisioner "remote-exec" {
inline = [
"sudo apt-get update"
]
}
}

Install Bundler gem with rbenv when provisioning (Chef + Vagrant)

I'm using Vagrant + Chef to provision an Ubuntu16.04 vm for rails development. I would like to install Bundler with Chef (using rbenv) but for some reason it fails with the following message: "undefined method `clear_sources' for Custom resource rbenv_gem from cookbook rbenv".
Any ideas?
Thanks!
The content of my Vagrantfile:
Vagrant.configure("2") do |config|
config.vm.box = "bento/ubuntu-16.04"
config.vm.network :forwarded_port, guest: 3000, host: 3000
config.vm.provision :chef_solo do |chef|
chef.cookbooks_path = ["cookbooks", "site-cookbooks"]
chef.add_recipe "apt"
chef.add_recipe "ruby_build"
chef.add_recipe "rbenv::user"
chef.add_recipe "rbenv::vagrant"
chef.json = {
rbenv: {
user_installs: [{
user: 'vagrant',
rubies: ["2.3.1"],
global: "2.3.1" ,
gems: {
"2.3.1" => [
{ name: "bundler" }
]
}
}]
}
}
end
end
Cheffile:
site "https://supermarket.getchef.com/api/v1"
cookbook 'apt'
cookbook 'build-essential'
cookbook 'ruby_build', '~> 1.0.0'
cookbook 'rbenv', git: 'https://github.com/aminin/chef-rbenv'
You should use the official chef-rbenv cookbook, your issue was probably fixed by #110.
BTW. Any particular reason for not using test-kitchen?

Vagrant/Puppet - Provision Failing (MySQL Not Working)

I have tried different virtual boxes, I have changed tons of configurations (probably recreated, reloaded and reprovisioned 50-100 times) -- it has to do with the MySQL module from Puppet -- which is apparently working for tens of thousands of others, so something I am doing is wrong.
VagrantFile
# -*- mode: ruby -*-
# vi: set ft=ruby :
# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
# Every Vagrant virtual environment requires a box to build off of.
config.vm.box = "puppetlabs/ubuntu-14.04-64-puppet"
# Was necessary
#vagrant plugin install vagrant-vbguest
# Networking
config.vm.network "forwarded_port", guest: 80, host: 8000
# config.vm.network "private_network", ip: "192.168.33.10"
# Synced folders
config.vm.synced_folder ".", "/var/www"
# Update
config.vm.provision "shell", inline: "apt-get -y update"
# Provisioning
config.vm.provision :shell do |shell|
shell.inline = "mkdir -p /etc/puppet/modules;
(puppet module install example42-puppi; true)
(puppet module install example42-apache; true)
(puppet module install example42-php; true)
(puppet module install puppetlabs-stdlib; true)
(puppet module install puppetlabs-mysql; true)
(puppet module install saz-vim; true)
(puppet module install saz-timezone; true)
(puppet module install puppetlabs-git; true)
(puppet module install tPl0ch-composer; true)
"
end
config.vm.provision "puppet" do |puppet|
puppet.facter = {
"fqdn" => "development.mydomain.com",
"aliases" => "*.development.mydomain.com",
"hostname" => "www",
"docroot" => '/var/www/html/',
}
puppet.hiera_config_path = "manifests/hiera.yaml"
puppet.working_directory = "/etc/puppet"
end
end
Puppet File: default.pp
###########################
# MySite Puppet Config #
###########################
# OS : Ubuntu 14 #
# Database : MySQL 5 #
# Web Server : Apache 2 #
# PHP version : 5.4 #
###########################
# Vim
class { 'vim': }
# Set Timezone
class { 'timezone':
timezone => 'America/Chicago',
}
# Puppi
class { 'puppi': }
# Apache setup
class { "apache":
puppi => true,
puppi_helper => "myhelper",
}
apache::vhost { $fqdn :
docroot => $docroot,
server_name => $fqdn,
serveraliases => $aliases,
priority => '',
template => 'apache/virtualhost/vhost.conf.erb',
}
apache::module { 'rewrite': }
apache::module { 'headers': }
# PHP Extensions
class {"php":}
php::module { ['xdebug', 'mysql', 'curl', 'gd', 'mcrypt']:
notify => Service['apache2']
}
# MySQL Server
class { '::mysql::server':
package_ensure => present,
root_password => '[root_password]',
override_options => { 'mysqld' => { 'default_time_zone' => 'America/Chicago' } },
}
class { 'mysql::client':}
mysql::db { '[db_name]':
user => '[user]',
password => '[password]',
host => 'localhost',
grant => ['ALL'],
charset => 'utf8',
}
exec { "database_import":
timeout => 300,
command => "/bin/gzip -dc /vagrant/manifests/provision.sql.gz | /usr/bin/mysql -u root -p[root_password]";
}
mysql::db { '[db_test]':
user => '[user_test]',
password => '[password_test]',
host => 'localhost',
grant => ['ALL'],
charset => 'utf8',
sql => "/var/www/test/db-schema.sql",
}
# Install Composer components
include composer
composer::exec { 'install':
cmd => 'install', # REQUIRED
cwd => '/var/www', # REQUIRED
dev => true, # Install dev dependencies
}
# Git
include git
# MySite Setup
file { $docroot:
ensure => 'directory',
}
$writeable_dirs = ["${docroot}cache/", "${docroot}cache/css/", "${docroot}cache/css/js/"]
file { $writeable_dirs:
ensure => "directory",
mode => '0777',
require => File[$docroot],
}
file { '/var/www/cl/':
ensure => directory
}
file { 'errors_log':
path => "/var/www/cl/errors.log",
ensure => present,
mode => 0777,
}
# Cron Jobs
cron { thirty_minutes:
command => "/usr/bin/php ${docroot}cl-load.php crons/thirty_minutes/",
user => root,
minute => '*/30'
}
cron { hourly:
command => "/usr/bin/php ${docroot}cl-load.php crons/hourly/",
user => root,
hour => '*',
minute => 0,
}
cron { four_hours:
command => "/usr/bin/php ${docroot}cl-load.php crons/four_hours/",
user => root,
hour => '*/4',
minute => 0,
}
cron { daily:
command => "/usr/bin/php ${docroot}cl-load.php crons/daily/",
user => root,
monthday => '*',
hour => '0',
minute => 0,
}
Here is the error message I am receiving (the first):
==> ==> default: Error: Could not start Service[mysqld]: Execution of '/sbin/start mysql' returned 1:
It seems that Puppet is trying to import the database before installing the MySQL server package.
Keep in mind that Puppet don't care about which resources you write first, if you need to manage something in a specific order you have to declare relationships explicitly. So in your case the following statement:
exec { "database_import":
timeout => 300,
command => "/bin/gzip -dc /vagrant/manifests/provision.sql.gz | /usr/bin/mysql -u root -p[root_password]",
require => Class['::mysql::server']
}
Could set the correct order for that resource. Possibly you'll need to declare additional relationships in a similar manner.

Vagrant Vhost with a FQDN (Full Qualified Domain Name)

Take a domain name like:
development.mysite.com
I want to setup a vhost with Vagrant that would allow me to access this location in my browser.
From what I can tell, it won't let me use port 80, so even if I change my hosts file to something like:
127.0.0.1 development.mysite.com
It still won't find it.
Does anyone have an idea?
UPDATE
VagrantFile
# -*- mode: ruby -*-
# vi: set ft=ruby :
# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
# Every Vagrant virtual environment requires a box to build off of.
config.vm.box = "hashicorp/precise32"
# Networking
config.vm.network "forwarded_port", guest: 80, host: 8000
config.vm.network "private_network", ip: "192.168.33.10"
# Synced folders
config.vm.synced_folder ".", "/var/www"
# Provisioning
config.vm.provision :shell, :inline => "apt-get update --fix-missing"
config.vm.provision :shell do |shell|
shell.inline = "mkdir -p /etc/puppet/modules;
(puppet module install example42/puppi; true)
(puppet module install example42/apache; true)
(puppet module install example42/php; true)
(puppet module install puppetlabs/stdlib; true)
(puppet module install puppetlabs/mysql; true)
(puppet module install saz/vim; true)"
end
config.vm.provision "puppet" do |puppet|
puppet.facter = {
"fqdn" => "development.eatologie.com",
"hostname" => "www",
"docroot" => '/var/www/html/'
}
puppet.manifests_path = "puppet"
puppet.manifest_file = "site.pp"
end
end
Puppet site.pp
###########################
# Eatologie Puppet Config #
###########################
# OS : Linux #
# Database : MySQL 5 #
# Web Server : Apache 2 #
# PHP version : 5.4 #
###########################
# Vim
class { 'vim': }
# Puppi
class { 'puppi': }
# Apache setup
class { "apache":
puppi => true,
puppi_helper => "myhelper",
}
apache::vhost { $fqdn :
docroot => $docroot,
server_name => $fqdn,
priority => '',
template => 'apache/virtualhost/vhost.conf.erb',
}
apache::module { 'rewrite': }
# PHP Extensions
class {"php":}
php::module { ['xdebug', 'mysql', 'curl', 'gd']:
notify => Service['apache2']
}
# MySQL Server
class { '::mysql::server':
root_password => 'abc123',
}
mysql::db { 'eatdb':
user => 'admin',
password => 'abc456',
host => 'localhost',
grant => ['all'],
charset => 'utf8',
}
# Eatologie Setup
file { $docroot:
ensure => 'directory',
}
$writeable_dirs = ["${docroot}cache/", "${docroot}cache/css/", "${docroot}cache/css/js/"]
file { $writeable_dirs:
ensure => "directory",
mode => '0777',
require => File[$docroot],
}
As far as I can see, this is the relevant part in your Vagrantfile:
# Networking
config.vm.network "forwarded_port", guest: 80, host: 8000
config.vm.network "private_network", ip: "192.168.33.10"
It means that your Vagrantbox is accessible at ip 192.168.33.10 and port 8000. So, your hosts file should say
192.168.33.10 development.eatologie.com
instead of
127.0.0.1 development.mysite.com
With that change, you should be to access your site at development.eatologie.com:8000
For more info, also take a look at the Vagrant documentation pages for forwarded ports and private networking. For the private networking part take into account that, if you assign a static IP to your Vagrant box (as you did in your setup), you have to be sure the IP is not taken by another device on your local network.

RCov doesn't work

I am currently developing a Ruby gem and want to create metrics.
I am using 'metric_fu', but RCov seems to leave my specs.
Here is my metric_fu configuration:
MetricFu::Configuration.run do |config|
config.metrics = [:churn, :saikuro, :flog, :flay, :reek, :roodi, :rcov]
config.graphs = [:flog, :flay, :reek, :roodi, :rcov]
config.flay = { :dirs_to_flay => ['lib'] }
config.flog = { :dirs_to_flog => ['lib'] }
config.reek = { :dirs_to_reek => ['lib'] }
config.roodi = { :dirs_to_roodi => ['lib'] }
config.saikuro = { :output_directory => 'scratch_directory/saikuro',
:input_directory => ['lib'],
:cyclo => "",
:filter_cyclo => "0",
:warn_cyclo => "5",
:error_cyclo => "7",
:formater => "text"} #this needs to be set to "text"
config.churn = { :start_date => "1 year ago", :minimum_churn_count => 10}
config.rcov = { :test_files => ["spec/**/*_spec.rb"],
:rcov_opts => ["--sort coverage",
"--no-html",
"--text-coverage",
"--no-color",
"--profile",
"--spec-only",
"--exclude /gems/,/Library/,spec"]}
end
Do you have some tips?
Best regards
Well this is going to be hard to diagnose without a stack trace but I would suggest changing your config to this:
MetricFu::Configuration.run do |config|
config.metrics = [:rcov]
config.graphs = [:rcov]
config.rcov = { :test_files => ["spec/**/*_spec.rb"],
:rcov_opts => ["--sort coverage",
"--no-html",
"--text-coverage",
"--no-color",
"--profile",
"--spec-only",
"--exclude /gems/,/Library/,spec"]}
end
So you can isolate the problem. Then run 'rake metrics:all --trace' and if you can't figure it out from there, post the results either here or the metric_fu google group: http://groups.google.com/group/metric_fu
You can also try running rcov straight from the command line (which is essentially what metric_fu does).
Hope that helps.

Resources