How can I create a recipe that will populate its attributes using the fiels from an instance of an object in a generic way?
As an example, consider the following recipe:
component = $auth_docker
docker_image component.name do
registry component.registry
tag component.tag
action :pull
end
When you have 50s of recipes that look like this, maintaining them really gets overwhelming.
In Python, i would probably implement a solution that would look a bit like this:
docker_image = DockerImage(**$auth_docker)
Or, I would create some sort of helper function to build it for me:
def generate_docker_image_lwrp(attributes):
lwrp = DockerImage()
lwrp.registry = attributes.registry
lwrp.tag = attributes.tag
return lwrp
The goal is to reduce maintenance on the recipes. For instance this morning I wanted to add Chef's "retries" attribute on all recipes that pull an image. I had to edit all of them - I don't want that. I should've been able to a) add the attribute to the stack's JSON b) edit the Ruby wrapper class so that instances of it (i.e.: $auth_docker) get the "retries" field, then c) add the retries attribute to the lwrp-generator. Since all recipes would use the same generator, recipes wouldn't need to be edited at all.
Is this possible using Chef, in a way that 'notifies' still work?
Quoting the Documentation
A definition is code that is reused across recipes, similar to a
compile-time macro. A definition is created using arbitrary code
wrapped around built-in chef-client resources—file, execute, template,
and so on—by declaring those resources into the definition as if they
were declared in a recipe. A definition is then used in one (or more)
recipes as if it were a resource.
Though a definition behaves like a resource, some key differences
exist. A definition:
Is not a resource or a lightweight resource Is defined from within the
/definitions directory of a cookbook Is loaded before resources during
the chef-client run; this ensures the definition is available to all
of the resources that may need it May not notify resources in the
resource collection because a definition is loaded before the resource
collection itself is created; however, a resource in a definition may
notify a resource that exists within the same definition Automatically
supports why-run mode, unlike lightweight resources Use a defintion
when repeating patterns exist across resources and/or when a simple,
direct approach is desired. There is no limit to the number of
resources that may be included in a definition: use as many built-in
chef-client resources as necessary.
I.e: you can create a definition for this in a library cookbook used solely for this.
docker_library/defintions/default.rb
define :my_docker_image, :component => nil do
component = params[:component]
docker_image component.name do
registry component.registry
tag component.tag
action :pull
end
end
in your recipes (need to have a depends on the docker_library cookbook in the metadata.rb):
my_component = $auth_docker
my_docker_image my_component.name do
component my_component
end
A more complete exemple of definition is available in the logrotate cookbook
Related
I'm reading https://github.com/Element-Research/dpnn/blob/master/Inception.lua
You can see tons of clone()'s in this source. like
mlp:add(self.transfer:clone())
self.transfer is nothing more than nn.ReLU().
Then,
Why does this code call activation functions using clone()? Does this only concern memory issues?
I thought that the clone shares parameters. Is this right? If it's right, it means all activations of this inception module share parameters. It looks like nonsense. Do I misunderstand Inception-Net?
If you don't clone the module self.transfer then all modules transfer in your net mlp will have the same state variables output and gradInput.
Look for example at this toy code
require 'nn'
module = nn.ReLU()
net = nn.Sequential():add(nn.Linear(2,2)):add(module):add(nn.Linear(2,1)):add(module)
input = torch.Tensor(2,2):random()
net:forward(input)
print(net:get(2).output)
print(net:get(4).output)
Both print statements will return the same value. Modifying one of the module outputs will modify the other one. Since we do not want this behavior we have to clone the module. (However in your case, cloning a simple nn.ReLU() is not that useful.)
The documentation says
If arguments are provided to the clone(...) function it also calls share(...) with those arguments on the cloned module after creating it, hence making a deep copy of this module with some shared parameters.
Therefore if you don't provide any arguments the parameters won't be shared.
I'd like to access attributes that are inside instances of defined types from other classes/instances.
This is very similar to a question asked on SO before - In Puppet, how can I access a variable/attribute inside a defined type?, however from what I understood the answer was specifically related to accessing parameters as opposed to arbitrary variables.
For example, given the following defined type:
define server (
$server_name = 'my_server'
){
$server_history = 'A long story'
}
I can successfully use getparam(...) to fetch server_name but I cannot do the same for server_history.
Also, if server was a class as opposed to to a defined type, accessing this variable is straightforward using something like server::serverhistory
Does anyone have any ideas on how to expose these variables? Or am I approaching this completely the wrong way?
Edit: For some higher level context on what I'm trying to do my server type gets instantiated by 3 other classes. A variable in the server type builds out some directory paths based on parameters provided to it by these classes (which naturally, are specific to those classes). There are some other classes that would like to use the directory path variable to place files there.
You ask
I'd like to access attributes that are inside instances of defined types from other classes/instances.
and you go on to clarify that you're after
arbitrary variables.
In fact, ordinary variables in the body of a defined type are not attributes of that type, nor of any instance thereof. They are not part of the accessible persistent state of instances of such types at all.
More generally, Puppet treats defined types just like native types in almost every observable way, but by the same token, it does not provide any features that serve to distinguish defined types as a special case. You are looking for such a feature, and it does not exist.
Since your design idea will not work, you'll need to think of an alternative. You say
my server type gets instantiated by 3 other classes. A variable in the server type builds out some directory paths based on parameters provided to it by these classes (which naturally, are specific to those classes). There are some other classes that would like to use the directory path variable to place files there.
Since the paths you're after are characteristic of specific classes, it makes sense for them to be accessible directly via those classes. It seems odd to me that you would even want to access them indirectly via resources declared by those classes.
Being relatively new to Chef, I am required to create libraries or definitions from existing recipes.
There recipes use bash resource, ruby block resource (which notifies another ruby block resource with delayed timing), template resource again which notifies a ruby block etc.
What would be the best approach to this? Library or definition?
I have read that if I use definition, I won't be able to notify a resource within the definition, does that mean I can notify a resource in a different definition file?
I also read that in libraries you cant use the resources directly. If this is true, how can I use a resource within my library?
So, this is "primarily opinion based", but I'll answer it anyway. There are 4 distinct choices here:
Definition
LWRP
HWRP
"Library"
A definition is just a wrapper around one or more resources with some parameterization. However, definitions are not added to the resource collection. Meaning you can't "notify" or trigger events on a definition. They are solely for wrapping and naming a series of repeatable steps found in a recipe.
An LWRP (Light-weight resource and provider) is a Chef-specific DSL that actually compiles into an HWRP (Heavy-weight resource and provider) at runtime. Both LWRPs and HWRPs are Chef extensions. In addition to wrapping a series of repeatable tasks, *WRPs will create a top-level resource in Chef (like template or package) that's available for use in your recipe and other cookbook's recipes as well.
The difference between and LWRP and HWRP is really the Ruby. HWRPs use full-blown Ruby classes. If you aren't a Ruby developer, they may be a bit intimidating. Nonetheless, you should give it a try before writing and LWRP. LWRPs use a Chef-specific DSL for creating resources. At the end of the day, they compile to (roughly) the same code as the Heavy-weight counterpart. I'll link some references at the end. You have access to Chef resources inside either implementation, as well as the run_context.
Finally, "libraries" (notice the quotes) are often misunderstood and abused. They are Ruby code, evaluated as Ruby, so they can do pretty much anything. HWRPs are actually a form of a library. Sometimes people use libraries as "helpers". They will create a helper module with methods like best_ip_for or aggregate_some_data and then "mix" (Rubyism) that library into their recipes or resources to DRY things up. Other times, libraries can be use to "hack" Chef itself. The partial-search cookbook is a good example of this. Facebook talked about how they limited the number of attributes sent back to the server last year at ChefConf too. Libraries are really an undefined territory because they are the keys to the kingdom.
So, while I haven't actually answered your question (because it's opinion-based), I hope I've given you enough information about the best direction moving forward. Keep in mind that every infrastructure is a special snowflake and there is no right answer; there's only a best answer. I'd suggest sharing this information with your team and weighing the pros and cons of each approach. You can also try the Chef mailing list, on which people will give you many opinions.
Resources:
LWRPs
HWRPs
Libraries
Modern Chef terminology has rebranded "LWRPs" as "Custom Resources" and the "Provider" that is the "P" of the "LWRP" has melted into the background and users just run actions now (the action_class is the way to access the old provider class).
Definitions still exist, are still discouraged, and still have known bugs which will never be fixed. There is no reason to use them.
Custom Resources are what everyone should use. More people should move recipe code into custom resources for reusability. The basic steps are simple:
Move the code into a resources file
Wrap the code with an action
Add a provides line for the name of the resource
Change the include_recipe call in recipes into a call to the custom resource.
That's all that is necessary for simple cases. Once that is done the resource can now be extended by adding properties, or existing node attributes can be converted into properties (the node attributes can be pushed back into the call to the resource from recipe mode).
For modern resources on Chef-15/16 consider setting unified_mode true to remove the compile/converge phase from the custom resource and simplify writing the resource.
For an example of the conversion of a simple recipe to a custom resource see this answer
I have a puppet environment where I need to add a variable number of very similar objects to a single server. The objects vary only by attributes such as name and path, all of which can be extrapolated from a single name parameter.
This seems like the perfect use for puppet resource definitions since multiple definitions can be added to a single server and their specific attributes can be taken from the definition name declaration.
This setup works well for me and I have had no issues getting it up and running on several servers. I have a requirement now however to hand this over to an ops department with almost no scripting experience, so they aren't really comfortable managing from the file system.
They requested a dashboard, so I setup puppet dashboard for them. After I set the dashboard up however, I found out that it only supports adding classes to servers, not definitions. This seems pretty shortsighted on the part of the folks at PuppetLabs, so I can only think that I am approaching this problem the wrong way and that there must be a solution using classes whereby multiple almost identical entities can be added to a single node.
I realize I could create a class for each entity, but there are hundreds, even thousands of potential variations so that's not really practical.
I have also considered a wrapper class that declares each definition on a per node basis, but this seems like more work to manage than it's worth.
Any thoughts on alternate approaches that would be compatible with the dashboard would be appreciated.
To make ENCs such as dashboard work this way, you have to pass in node data in the form of node variables. Note that Hiera is often preferable to plain node variables.
Generally, the value you want to hand in is an array of your resource titles
$resources = [ "name1", "name2", ... ]
Then classify your node with (at least) one class that instanciates your defined type using this data, e.g.
class my_resources {
my_defined_type { $resources: }
}
I'm wanting to dynamically create and query tables using Datamapper.
While Datamapper allows you to work with legacy tables and schemas, and in this way set the table name used this is only during initialisation, not within the application.
Is there an easy way to tell Datamapper to migrate/upgrade a Model with an assigned table name in application, and to then tell it to query this table?
This should not be a problem.
All Ruby classes can be created, and re-defined at run-time. Even initialization is at run-time. Initialization just happens to be executed first, before other code is executed.
That is why monkey-patches work so easily. It's just additional code at initialization that just re-defines classes to add extra methods, variables etc.
There is no Ruby code that is "special" in the sense that it only runs at compile time. Ruby is an interpreted language.
To dynamically create a class, see Dynamically creating class in Ruby.
Assuming you don't need to dynamically create classes from an array of strings, you can define additional methods with define_method, or call Datamapper methods at runtime to add attributes.
To define new methods in a class:
Post.send :define_method, :new_method_name do
end
To define a new property using the Datamapper property:
class Post
include DataMapper::Resource
property :title, String # the static way
end
Post.send :property, :title, String # add property the dynamic way (at run-time)
Do note that any tables or properties you define at run-time will not be available if you restart your server, unless the code that dynamically generates these are re-executed.
To update your tables at runtime, you simply do the same thing as normal, that is, call:
DataMapper.auto_upgrade!
To upgrade only a single table, you can also do:
Post.auto_upgrade!
2nd warning: If you have multiple processes, the dynamic code will need to be run in each process, or the additional table Models and Properties will not be available.
This is a problem if you have multiple worker processes, as might happen in production (eg. Nginx with multiple Unicorn workers, or multiple Mongrel workers behind a Ha_proxy).
If you have a single process server, then that is not a problem. However, if you have multiple worker processes, you must run the dynamic code to generate these extra classes and properties in EACH process to make it available.
This is actually the same for initialization, because each process goes through initialization (or if forked, inherit any initialization).
The easiest way without changing anything under the hood is to use separate databases instead of tables (assuming that any relationships will also be stored in the separate database) and open a connection to an additional repository in the block.
DataMapper.setup(:external, "adapter://username:password#hostname/dbname")
DataMapper.repository(:external) do...end