Here is the python example of a data set i have in my puppet code similar to the below, :
dict = {'account1': {'uid': ['123456'], 'user': ['appuser1'], 'appname': ['myapp1']},
'account2': {'uid':['567878'], 'user':['appuser'2], 'appname':['myapp2']}}
for i in dict.keys():
print dict[i]['user'], dict[i]['uid']
How do i achieve the same solution in puppet/ruby.TIA.
In Puppet manifests, you can iterate over a hash using the each function:
$ cat foo.pp
$dict = {
'account1' => {
'uid' => ['123456'],
'user' => ['appuser1'],
'appname' => ['myapp1']
},
'account2' => {
'uid' => ['567878'],
'user' => ['appuser2'],
'appname' => ['myapp2']
}
}
$dict.each | $account_key, $account | {
notice("${account['user'][0]}, ${account['uid'][0]}")
}
$ puppet apply foo.pp
Notice: Scope(Class[main]): appuser1, 123456
Notice: Scope(Class[main]): appuser2, 567878
Notice: Compiled catalog for it070137 in environment production in 0.04 seconds
Notice: Applied catalog in 0.03 seconds
If you like, you can use types to check that the key and value in the hash are what you expect:
$dict.each | String $account_key, Hash $account | {
notice("${account['user'][0]}, ${account['uid'][0]}")
}
Related
I'm working with Puppet 4.5 in masterless configuration and am trying to create a Puppet function to read a simple config file that assigns roles and environments. I don't have any integration with hiera/facter that I can change.
The file format is:
host1::java_app_node::qa
host2::nodejs_app_node::prod
The Puppet function that will read this file is in a module called homebase. I want to function to return a hash or array of hashes that split the config values. This will let me use them in templates.
In modules/homebase/manifests/init.pp I define:
$role_file = 'puppet://role.lst'
I then created modules/homebase/functions/get_roles.pp as follows:
function homebase::get_roles() {
$func_name = 'homebase::get_roles()'
if ! File.exists?($::homebase::role_file) {
fail("Could not find #{$::homebase::role_file}")
}
hosts = { }
File.open($::homebase::role_file).each |line| {
parts = line.split(/::/)
hosts[parts[0]] = { 'host' => parts[0], 'role' => parts[1], 'env' => parts[2] }
}
return hosts
}
In other classes, I then want to call:
class myapp {
$servers = homebase::get_roles().each | k, v | {
$v['host'] if $v['role'] =~ /myapp/ && $v['env'] == $environment
}
file { 'myapp.cfg':
ensure => file,
path => '/opt/myapp/myapp.cfg',
source => template("/myapp/myapp.cfg.erb"),
mode => '0644',
owner => myuser,
group => myuser,
}
}
Seems like there would be a better way to do this. Am I completely off base?
There turned out to be a much easier way to this rather than try to create a function to read a non-standard configuration file. Instead, I used a site.pp file to create node {} entries. I also parameterized the myapp class to take inputs based on the node.
So my site.pp looks like:
node 'server1.mydomain', 'server2.mydomain' {
$myvar = [ 'val1', 'val2' ]
class { 'myapp':
values => $myvar
}
}
This could probably be improved. One of the issues is with a non-Puppet configuration file I was able to also able to control execution in my bash wrapper script. Much of the need for that went away when, though, with the node definitions.
I just recently knew about puppet and I'm trying to install laravel while vagrant is provisioning so that everything is already set (can be able to run laravel) when I logged in/ssh to vagrant. But I got stuck, it returned successfully executed but after I do vagrant ssh, laravel command is not available.
php5, php5-cli, etc. composer and other dependencies is already installed before this part of code.
class laravel {
Exec {
path => "/bin:/sbin:/usr/bin:/usr/sbin",
}
exec { "install-laravel" :
command => "/usr/local/bin/composer global require 'laravel/installer'",
require => [Package["php5-cli", "php5-dev"], Exec["install-composer", "set-composer-as-global"]],
cwd => "/home/vagrant/",
environment => ["COMPOSER_HOME=/home/vagrant"],
user => root,
group => root,
}
exec { "add-laravel-command" :
command => "mkdir /usr/local/bin/laravel",
environment => ["LARAVEL_HOME=/home/vagrant"],
onlyif => "test -d /usr/local/bin/composer",
require => Exec["install-laravel"],
user => root,
}
exec { "set-laravel-as-globall" :
command => "mv /home/vagrant/.composer/vendor/bin /usr/local/bin/laravel",
onlyif => "test -d /.composer/vendor/bin",
require => Exec["add-laravel-command"],
user => root,
}
}
Output
==> default: Notice: /Stage[main]/Laravel/Exec[install-laravel]/returns: executed successfully
==> default: Debug: /Stage[main]/Laravel/Exec[install-laravel]: The container Class[Laravel] will propagate my refresh event
==> default: Debug: Exec[add-laravel-command](provider=posix): Executing check 'test -d /usr/local/bin/composer'
==> default: Debug: Executing 'test -d /usr/local/bin/composer'
==> default: Debug: Exec[set-laravel-as-globall](provider=posix): Executing check 'test -d /.composer/vendor/bin'
==> default: Debug: Executing 'test -d /.composer/vendor/bin'
==> default: Debug: Class[Laravel]: The container Stage[main] will propagate my refresh event
Any help would be much appreciated. Thanks
create a file under puppet/modules/laravel/files and name it composer.json with the following content
{
"require": {
"laravel/installer": "^1.3"
}
}
Laravel init.pp file
class laravel {
Exec {
path => "/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin",
user => root,
group => root,
}
file { "/usr/local/bin/laravel-devtools" :
ensure => directory,
owner => root,
group => root,
}
file { "/usr/local/bin/laravel-devtools/composer.json" :
source => "puppet:///modules/laravel/composer.json",
require => File["/usr/local/bin/laravel-devtools"],
owner => root,
group => root,
}
exec { "install-laravel" :
command => "sudo composer require 'laravel/installer'",
onlyif => "test -f /usr/local/bin/composer",
require => [
Package["nginx","php5-cli", "php5-dev", "php5-mysql"],
File["/usr/local/bin/laravel-devtools", "/usr/local/bin/laravel-devtools"],
],
environment => ["COMPOSER_HOME=/home/vagrant"],
cwd => "/usr/local/bin/laravel-devtools",
user => root,
group => root,
}
exec { "set-laravel-as-global" :
command => "sudo ln -s /usr/local/bin/laravel-devtools/vendor/laravel/installer/laravel /usr/local/bin/laravel",
require => [
File["/usr/local/bin/laravel-devtools", "/usr/local/bin/laravel-devtools/composer.json"],
Exec["install-laravel"],
],
}
}
Make sure composer is installed first during provision.
I have tried different virtual boxes, I have changed tons of configurations (probably recreated, reloaded and reprovisioned 50-100 times) -- it has to do with the MySQL module from Puppet -- which is apparently working for tens of thousands of others, so something I am doing is wrong.
VagrantFile
# -*- mode: ruby -*-
# vi: set ft=ruby :
# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
# Every Vagrant virtual environment requires a box to build off of.
config.vm.box = "puppetlabs/ubuntu-14.04-64-puppet"
# Was necessary
#vagrant plugin install vagrant-vbguest
# Networking
config.vm.network "forwarded_port", guest: 80, host: 8000
# config.vm.network "private_network", ip: "192.168.33.10"
# Synced folders
config.vm.synced_folder ".", "/var/www"
# Update
config.vm.provision "shell", inline: "apt-get -y update"
# Provisioning
config.vm.provision :shell do |shell|
shell.inline = "mkdir -p /etc/puppet/modules;
(puppet module install example42-puppi; true)
(puppet module install example42-apache; true)
(puppet module install example42-php; true)
(puppet module install puppetlabs-stdlib; true)
(puppet module install puppetlabs-mysql; true)
(puppet module install saz-vim; true)
(puppet module install saz-timezone; true)
(puppet module install puppetlabs-git; true)
(puppet module install tPl0ch-composer; true)
"
end
config.vm.provision "puppet" do |puppet|
puppet.facter = {
"fqdn" => "development.mydomain.com",
"aliases" => "*.development.mydomain.com",
"hostname" => "www",
"docroot" => '/var/www/html/',
}
puppet.hiera_config_path = "manifests/hiera.yaml"
puppet.working_directory = "/etc/puppet"
end
end
Puppet File: default.pp
###########################
# MySite Puppet Config #
###########################
# OS : Ubuntu 14 #
# Database : MySQL 5 #
# Web Server : Apache 2 #
# PHP version : 5.4 #
###########################
# Vim
class { 'vim': }
# Set Timezone
class { 'timezone':
timezone => 'America/Chicago',
}
# Puppi
class { 'puppi': }
# Apache setup
class { "apache":
puppi => true,
puppi_helper => "myhelper",
}
apache::vhost { $fqdn :
docroot => $docroot,
server_name => $fqdn,
serveraliases => $aliases,
priority => '',
template => 'apache/virtualhost/vhost.conf.erb',
}
apache::module { 'rewrite': }
apache::module { 'headers': }
# PHP Extensions
class {"php":}
php::module { ['xdebug', 'mysql', 'curl', 'gd', 'mcrypt']:
notify => Service['apache2']
}
# MySQL Server
class { '::mysql::server':
package_ensure => present,
root_password => '[root_password]',
override_options => { 'mysqld' => { 'default_time_zone' => 'America/Chicago' } },
}
class { 'mysql::client':}
mysql::db { '[db_name]':
user => '[user]',
password => '[password]',
host => 'localhost',
grant => ['ALL'],
charset => 'utf8',
}
exec { "database_import":
timeout => 300,
command => "/bin/gzip -dc /vagrant/manifests/provision.sql.gz | /usr/bin/mysql -u root -p[root_password]";
}
mysql::db { '[db_test]':
user => '[user_test]',
password => '[password_test]',
host => 'localhost',
grant => ['ALL'],
charset => 'utf8',
sql => "/var/www/test/db-schema.sql",
}
# Install Composer components
include composer
composer::exec { 'install':
cmd => 'install', # REQUIRED
cwd => '/var/www', # REQUIRED
dev => true, # Install dev dependencies
}
# Git
include git
# MySite Setup
file { $docroot:
ensure => 'directory',
}
$writeable_dirs = ["${docroot}cache/", "${docroot}cache/css/", "${docroot}cache/css/js/"]
file { $writeable_dirs:
ensure => "directory",
mode => '0777',
require => File[$docroot],
}
file { '/var/www/cl/':
ensure => directory
}
file { 'errors_log':
path => "/var/www/cl/errors.log",
ensure => present,
mode => 0777,
}
# Cron Jobs
cron { thirty_minutes:
command => "/usr/bin/php ${docroot}cl-load.php crons/thirty_minutes/",
user => root,
minute => '*/30'
}
cron { hourly:
command => "/usr/bin/php ${docroot}cl-load.php crons/hourly/",
user => root,
hour => '*',
minute => 0,
}
cron { four_hours:
command => "/usr/bin/php ${docroot}cl-load.php crons/four_hours/",
user => root,
hour => '*/4',
minute => 0,
}
cron { daily:
command => "/usr/bin/php ${docroot}cl-load.php crons/daily/",
user => root,
monthday => '*',
hour => '0',
minute => 0,
}
Here is the error message I am receiving (the first):
==> ==> default: Error: Could not start Service[mysqld]: Execution of '/sbin/start mysql' returned 1:
It seems that Puppet is trying to import the database before installing the MySQL server package.
Keep in mind that Puppet don't care about which resources you write first, if you need to manage something in a specific order you have to declare relationships explicitly. So in your case the following statement:
exec { "database_import":
timeout => 300,
command => "/bin/gzip -dc /vagrant/manifests/provision.sql.gz | /usr/bin/mysql -u root -p[root_password]",
require => Class['::mysql::server']
}
Could set the correct order for that resource. Possibly you'll need to declare additional relationships in a similar manner.
I recently updated mongodb from 2.4 to 2.6, and the new memory limit in aggregate() is causing my aggregation to fail with the following error:
Moped::Errors::OperationFailure: The operation: #<Moped::Protocol::Command
#length=251
#request_id=6
#response_to=0
#op_code=2004
#flags=[:slave_ok]
#full_collection_name="items.$cmd"
#skip=0
#limit=-1
#selector={:aggregate=>"items", :pipeline=>[{"$group"=>{"_id"=>"$serial_number", "total"=>{"$sum"=>1}}}, {"$match"=>{"total"=>{"$gte"=>2}}}, {"$sort"=>{"total"=>-1}}, {"$limit"=>750000}]}
#fields=nil>
failed with error 16945: "exception: Exceeded memory limit for $group, but didn't allow external sort. Pass allowDiskUse:true to opt in."
So, I'm trying to pass allowDiskUse: true in the query:
dupes = Item.collection.aggregate([{
'$group' => {'_id' => "$serial_number", 'total' => { "$sum" => 1 } } },
{ '$match' => { 'total' => { '$gte' => 2 } } },
{ '$sort' => {'total' => -1}},
{ '$limit' => 750000 }],
{ 'allowDiskUse' => true })
But this isnt working.... no matter how I try I get this error:
Moped::Errors::OperationFailure: The operation: #<Moped::Protocol::Command
#length=274
#request_id=2
#response_to=0
#op_code=2004
#flags=[:slave_ok]
#full_collection_name="items.$cmd"
#skip=0
#limit=-1
#selector={:aggregate=>"items", :pipeline=>[{"$group"=>{"_id"=>"$serial_number", "total"=>{"$sum"=>1}}}, {"$match"=>{"total"=>{"$gte"=>2}}}, {"$sort"=>{"total"=>-1}}, {"$limit"=>750000}, {"allowDiskUse"=>true}]}
#fields=nil>
failed with error 16436: "exception: Unrecognized pipeline stage name: 'allowDiskUse'"
Does anyone know how I can structure this query appropriately to pass allowDiskUse outside of the pipeline arg?
Follow below syntax for Mongoid 5.0.0
Modelname.collection.aggregate(
[your stages, ... ],
:allow_disk_use => true
)
For instance
group = { "$group" => {"_id" => {"column_xyz"=>"$column_xyz" }, "collection_name" => { "$push" => "$$ROOT" }, "count" => { "$sum" => 1 } }};
Hive.collection.aggregate([group], {:allow_disk_use => true})
ref: MongoDB jira Ruby-1041's comments
The problem is that Moped does not currently permit options for Moped::Collection#aggregate, just a pipeline for args,
as can be seen here: https://github.com/mongoid/moped/blob/master/lib/moped/collection.rb#L146 -
the Mongo Ruby driver supports options for Mongo::Collection#aggregate, but Mongoid 3 uses Moped for its driver.
However, thanks to the dynamic nature of Ruby, you can work around this.
The following test includes a monkey-patch for Moped::Collection#aggregate provided that you supply the pipeline
as an array for the first argument, allowing you to tack on options like allowDiskUse.
Hope that this helps.
test/unit/item_test.rb
require 'test_helper'
module Moped
class Collection
def aggregate(pipeline, opts = {})
database.session.command({aggregate: name, pipeline: pipeline}.merge(opts))["result"]
end
end
end
class ItemTest < ActiveSupport::TestCase
def setup
Item.delete_all
end
test "moped aggregate with allowDiskUse" do
puts "\nMongoid::VERSION:#{Mongoid::VERSION}\nMoped::VERSION:#{Moped::VERSION}"
docs = [
{serial_number: 1},
{serial_number: 2},
{serial_number: 2},
{serial_number: 3},
{serial_number: 3},
{serial_number: 3}
]
Item.create(docs)
assert_equal(docs.count, Item.count)
dups = Item.collection.aggregate(
[{'$group' => {'_id' => "$serial_number", 'total' => {"$sum" => 1}}},
{'$match' => {'total' => {'$gte' => 2}}},
{'$sort' => {'total' => -1}},
{'$limit' => 750000}],
{'allowDiskUse' => true})
p dups
end
end
$ rake test
Run options:
# Running tests:
[1/1] ItemTest#test_moped_aggregate_with_allowDiskUse
Mongoid::VERSION:3.1.6
Moped::VERSION:1.5.2
[{"_id"=>3, "total"=>3}, {"_id"=>2, "total"=>2}]
Finished tests in 0.027865s, 35.8873 tests/s, 35.8873 assertions/s.
1 tests, 1 assertions, 0 failures, 0 errors, 0 skips
I have a Ruby file, "one.rb":
require 'yaml'
e = { "names"=>{"first_name" => "shaik", "last_name" => "farooq"} }
puts e.to_yaml
When I run this it gets executed successfully in the console and outputs:
---
names:
first_name: shaik
last_name: farooq
I want to store the executed data in a file with a "yml" extension. How can I do this from the above file (test.rb)?
It's really simple:
require 'yaml'
e = { "names"=>{"first_name" => "shaik", "last_name" => "farooq"} }
File.write('test.yaml', e.to_yaml)
After running that, a file called "test.yaml" will exist in the current directory that contains:
---
names:
first_name: shaik
last_name: farooq
You can reload that data easily also:
new_e = YAML.load_file('test.yaml')
# => {"names"=>{"first_name"=>"shaik", "last_name"=>"farooq"}}
You can write the yaml to file with:
require 'yaml'
e = { "names"=>{"first_name" => "shaik", "last_name" => "farooq"} }
File.open('your_file_name.yml', 'w') { |f| f.write(e.to_yaml) }