I have a Chef recipe for a multi-node web service, each node of which needs to get the hostname and IP of the other nodes, to put it into its own local configuration.
The code is shown below. The problem is that when the node.set[][] assignments are made in the ruby_block as shown, the values are empty when the template that relies upon them is created. If I want to create that template, I have to move all of the ruby_block code outside, and have it "loose" in the recipe. Which makes it harder to do unit-testing with Chefspec and the like.
Can any Chef guru set me straight? Is it just impossible to do node.set[] like this inside of a ruby_block? And if so, why doesn't it say so in the docs?
$cm = { :name => "web", :hostname => "" , :ip_addr => "" }
$ca = { :name => "data", :hostname => "" , :ip_addr => "" }
$cg = { :name => "gateway", :hostname => "" , :ip_addr => "" }
$component_list = [$cm, $ca, $cg]
ruby_block "get host addresses" do
block do
for cmpnt in $component_list
# do REST calls to external service to get cmpnt.hostname, ip_addr
# .......
node.set[cmpnt.name]['name'] = cmpnt.name
node.set[cmpnt.name]['host'] = cmpnt.hostname
node.set[cmpnt.name]['ip'] = cmpnt.ip_addr
end
end
end
template "/etc/app/configuration/config.xml" do
source "config.xml.erb"
variables( :dataHost => node['data']['host'],
:webHost => node['web']['host'],
:gatewayHost => node['gateway']['host'] )
action :create
end
I also added
subscribes :create, "ruby_block[get host addresses]", :immediately
to the template definition to ensure that the ruby_block ran before the template was created. This didn't make a difference.
I realize this is an old post, however for future reference, I just ran across this gist which gives a nice example of node variable assignments in the Compile vs. Converge phases. To adapt the gist to your example, you'll need to add code like the following to your ruby_block:
template_r = run_context.resource_collection.find(:template => "/etc/app/configuration/config.xml")
template_r.content node['data']['host']
template_r.content node['web']['host']
template_r.content node['gateway']['host']
For Chef 11, also see Lazy Attribute Evaluation.
The problem seems to be that attribute values inside your template resource definition get evaluated before actually invoking any resources.
I.e. the file is first executed as simple Ruby, compiling the resources, and only the the resource actions gets invoked. By that time, it is too late already.
I ran into the same problem when trying to encapsulate certain attribute manipulations into a resource. It simply does not work. Should anyone know a solution to this problem, I would appreciate it very much.
EDIT:
b = ruby_block...
...
end
b.run_action(:create)
Could possibly do the trick. It invokes the resource immediately.
The simplest answer to this is to not use chef attributes and not use ruby_block to do the work of talking to the REST API. The code can also be moved to a custom resource for better reuse:
unified_mode true
provides :my_resource
action :run do
cm = { :name => "web", :hostname => "" , :ip_addr => "" }
ca = { :name => "data", :hostname => "" , :ip_addr => "" }
cg = { :name => "gateway", :hostname => "" , :ip_addr => "" }
component_list = [cm, ca, cg]
hash = {}
for cmpnt in component_list
# do REST calls to external service to get cmpnt.hostname, ip_addr
# .......
hash[cmpnt.name] = {}
hash[cmpnt.name]['name'] = cmpnt.name
hash[cmpnt.name]['host'] = cmpnt.hostname
hash[cmpnt.name]['ip'] = cmpnt.ip_addr
end
template "/etc/app/configuration/config.xml" do
source "config.xml.erb"
variables( :dataHost => hash['data']['host'],
:webHost => hash['web']['host'],
:gatewayHost => hash['gateway']['host'] )
action :create
end
end
By using unified_mode and moving into a custom resource, it also makes it easier to use a node attribute without requiring the use of lazy {} or ruby_blocks. It also still allows chef configuration (like setting up resolv.conf or other network requirements before doing the REST calls) prior to calling this code while not having to think about compile/converge two pass issues in recipe context.
There is also no reason to use a resource like ruby_block to do pure ruby processing which does not change the system under management. In this case the ruby_block is hitting a REST service purely to collect data. That does not need to be placed into a Chef resource. It isn't clear from the question if that was being done because the questioner though it was a "best practice" (in this case it is not), or if it was being done to move execution to compile time in order to allow other chef resources that aren't part of the question to fire first (in which case using a custom resource is a much better solution than using a ruby_block).
It's been a while since this question, but in case someone is still looking for it, lazy evaluate is your friend:
template '/tmp/sql_file.sql' do
source "sql_file.sql.erb"
mode 0700
variables lazy {
# Create a new instance of MySQL library
mysql_lib = Acx::MySQL.new(
'127.0.0.1', 'root', node['mysql']['service']['pass']
)
password = node['mysql']['service']['support_admin']['ct_password']
# It returns the encrypted password after evaluate it, to
# be used in template variables
{ admin_password: mysql_lib.encrypted_password(password) }
}
end
https://docs.chef.io/resource_common.html#lazy-evaluation
Related
From https://github.com/rapid7/metasploit-framework/blob/master/modules/auxiliary/scanner/smtp/smtp_version.rb#L26:
class MetasploitModule < Msf::Auxiliary
include Msf::Exploit::Remote::Smtp
include Msf::Auxiliary::Scanner
include Msf::Auxiliary::Report
def initialize
super(
'Name' => 'SMTP Banner Grabber',
'Description' => 'SMTP Banner Grabber',
'References' =>
[
['URL', 'http://www.ietf.org/rfc/rfc2821.txt'],
],
'Author' => 'CG',
'License' => MSF_LICENSE
)
deregister_options('MAILFROM', 'MAILTO')
end
def run_host(ip)
res = connect
banner_sanitized = Rex::Text.to_hex_ascii(banner.to_s)
print_good("#{ip}:#{rport} SMTP #{banner_sanitized}")
report_service(:host => rhost, :port => rport, :name => "smtp", :info => banner)
end
end
I see connect is called above. Is connect a member function? How to know the member function of which super class is called? Thanks.
Most Likely an Inherited or Included Method
You didn't post all the relevant code, so you'd have to go spelunking in the rest of the source to be sure. I'm only going to address the section you specifically addressed.
In the small example you linked to, #connect is probably inherited from Msf::Auxiliary, or one of the included modules. From the name, it's most likely from Msf::Exploit::Remote::Smtp, but I didn't research it for you.
The reason it's most likely a method is that your current class isn't taking a connect argument anywhere. Since Ruby expressions essentially allow you to use method calls like right-hand side assignment values, this is a common idiom.
If you want to know for sure, grep your source for def connect to find where it's defined (assuming it's not done through something more meta like Module#define_method), or /\bconnect\b/ to find all the locations in your source tree where you might want to look more closely for its use or definition.
So i have a bit of a pickle.
I have an encrypted data bag to store LDAP passwords. In my node run list, one of my recipes installs the secret key onto my client machine.
In my problematic cookbook, i have a helper (in /libraries) that pulls data from AD (using LDAP). Problem is, i can't find a way to delay the assignment of my node attribute after initial compile phase.
Take this line of code as example :
node.override['yp_chefserver']['osAdminUser'] = node['yp_chefserver']['osAdminUser'] + get_sam("#{data_bag_item('yp_chefserver', 'ldap', IO.read('/etc/chef/secret/yp_chefserver'))['ldap_password']}")
Im trying to override an attribute by adding an array returned by my helper function "get_sam" which returns an array, but it needs to run AFTER the compile phase since the file "/etc/chef/secret/yp_chefserver" doesnt exist before the convergence of my runlist.
So my question : Is there a way to assign node attributes via data_bag_items during the execution phase?
Some things i've tried :
ruby_block 'attribution' do
only_if { File.exist?('/etc/chef/secret/yp_chefserver')}
block do
node.override['yp_chefserver']['osAdminUser'] = node['yp_chefserver']['osAdminUser'] + get_sam("#{data_bag_item('yp_chefserver', 'ldap', IO.read('/etc/chef/secret/yp_chefserver'))['ldap_password']}")
Chef::Log.warn("content of osAdminUser : #{node['yp_chefserver']['osAdminUser']}")
end
end
This doesn't work because the custom resource ruby_block doesn't have the method "data_bag_item". I've tried using lazy attributes in my "chef_server" custom resource, but same problem.
I also tried having the attribution done directly in my helper module, but since the helper module compiles before the exec phase, the file doesn't exist when it assigns the variable.
Here is the helper function in question should anyone wonder, it pulls the SamAccountName from LDAP to assign admin users to my chef server. :
module YpChefserver
module LDAP
require 'net-ldap'
#ldap
def get_ldap(ldap_password)
if #ldap.nil?
#ldap = Net::LDAP.new :host => "ADSERVER",
:port => 389,
:auth => {
:method => :simple,
:username => "CN=USERNAME,OU=East Service Accounts,OU=System Accounts,DC=ad,DC=ypg,DC=com",
:password => "#{ldap_password}"
}
end
#ldap
end
def get_ldap_users(ldap_password)
filter = Net::LDAP::Filter.eq("cn", "DevOps")
treebase = "dc=ad, dc=ypg, dc=com"
get_ldap(ldap_password).search(:base => treebase, :filter => filter) do |entry|
#puts "DN: #{entry.dn}"
entry.each do |attribute, values|
return values if attribute == :member
end
end
end
def get_sam(ldap_password)
samacc = Array.new
get_ldap_users(ldap_password).entries.each{ |elem|
y = elem.to_s.split(/[,=]/)
filter = Net::LDAP::Filter.eq("cn", y[1])
treebase = "OU=Support Users and Groups,OU=CGI Support,DC=ad,DC=ypg,DC=com"
get_ldap(ldap_password).search(:base => treebase, :filter => filter, :attributes => "SamAccountName") do |entry|
samacc << entry.samaccountname
end
}
return samacc
end
end
end
Turns out you can actually call it inside a ruby block, just by using the actual Chef call instead of the resource name, as follow :
ruby_block 'attributes' do
only_if {File.exist?('/etc/chef/secret/yp_chefserver')}
block do
dtbg = Chef::EncryptedDataBagItem.load('yp_chefserver','ldap',"IO.read('/etc/chef/secret/yp_chefserver')")
end
end
Leaving this here for those who might need it
EDIT :
Here is final function using the code mentionned above to pull accounts from AD, using encrypted data bags to provide the password and to then pass those results to my node attributes, all during the execution phase :
ruby_block 'attributes' do
extend YpChefserver::LDAP
only_if {File.exist?('/etc/chef/secret/yp_chefserver')}
block do
# Chef::Config[:encrypted_data_bag_secret] = '/etc/chef/secret/yp_chefserver'
dtbg = Chef::EncryptedDataBagItem.load('yp_chefserver','ldap')
node.override['yp_chefserver']['ldap_pw'] = dtbg['ldap_password']
userarray = Array.new
userarray.push("#{node['yp_chefserver']['osAdminUser']}")
get_sam("#{node['yp_chefserver']['ldap_pw']}").each { |i| userarray.push(i[0]) }
node.override['yp_chefserver']['authorized_users'] = userarray
node.override['yp_chefserver']['local_admin_pw'] = dtbg['local_admin_pw']
end
end
I read that as of 2016 OpsCode recommends against LWRP and the alternative HWRP. They rather recommend to use custom resources.
While this makes sense it leaves a lot to be desired in terms of leveraging ruby (unless I've misunderstood?)
I would like to be able to modify an array based on some boolean properties. I can then use that array to pass to the template, as in:
property :get_iptables_info, [TrueClass, FalseClass], default: false
property :get_pkglist, [TrueClass, FalseClass], default: false
property :cw_logs, Array, default: [], required: false
action :create do
ruby_block 'cw_iptables' do
block do
new_resource.cw_logs.push({ "#{new_resource.custom_dir}/iptables/iptables.txt" => { "log_group_name" => new_resource.log_group_name+"/iptables"}})
end
action :run
only_if {new_resource.get_iptables_info}
end
template "my_template" do
variables ({:logstreams => cw_logs})
end
end
Then in my template:
<% #logstreams.each_pair do |path, _object| %>
["#{path}"]
log_group_name = _object["log_group_name"]
<% end %>
The problem is that properties are immutable. So I get the error:
RuntimeError
------------
ruby_block[cw_iptables] (/tmp/kitchen/cache/cookbooks/custom_cw/resources/logs.rb line 43) had an error: RuntimeError: can't modify frozen Array
What's the correct way to do this? What's the correct way to write ruby code within the resource so that it's more modular and uses methods/functions?
Mutating properties inside a custom resource is generally a bad idea (there are exceptions but this isn't one of them). Better would be to use a local variable. Related, you don't need to use a ruby_block like that when you're already in a custom resource's action:
action :create do
cw_logs = new_resource.cw_logs
if new_resource.get_iptables_info
cw_logs += [whatever extra stuff you want]
end
template "my_template" do
variables logstreams: cw_logs
end
end
Notably that's using += instead of push which doesn't mutate the original. If you used push you would want cw_logs = new_resource.cw_logs.dup or similar.
I've got a Vagrant setup in which I'm trying to use Chef-solo to generate an conf file which loops though defined variables to pass to the application. Everything is working except the loop and I'm not familiar enough with Ruby/Chef to spot the error.
I'm going to lay out the whole chain of events in case there is something along the way that is the problem, but the first portions of this process seem to work fine.
A config file is written in yaml and includes env variable definitions to be passed:
...
variables:
- DEBUG: 2
...
The config file is read in by the Vagrantfile into a ruby hash and used to create the Chef json nodes:
...
settings = YAML::load(File.read("config.yaml"))
# Provision The Virtual Machine Using Chef
config.vm.provision "chef_solo" do |chef|
chef.json = {
"mysql" => {"server_root_password" => "secret"},
"postgresql" => {"password" => {"postgres" => "secret"}},
"nginx" => {"pid" => "/run/nginx.pid"},
"php-fpm" => {"pid" => "/run/php5-fpm.pid"},
"databases" => settings["databases"] || [],
"sites" => settings["sites"] || [],
"variables" => settings["variables"] || []
}
...
A bunch of chef cookbooks are run (apt, php, nginx, mysql etc) and finally my custom cookbook which is whats giving me grief. The portion of the cookbook responsible for creating a the conf file is shown here:
# Configure All Of The Server Environment Variables
template "#{node['php-fpm']['pool_conf_dir']}/vars.conf" do
source "vars.erb"
owner "root"
group "root"
mode 0644
variables(
:vars => node['variables']
)
notifies :restart, "service[php-fpm]"
end
And the vars.erb is just a one-liner
<%= #vars.each {|key, value| puts "env[" + key + " = " + value } %>
So, when I run all this chef spits out an error about not being able to convert a hash to a string.
can't convert Chef::Node::immutableMash into String
So for some reason this is coming across as an immutableMash and the value of key ends up being the hash [{"DEBUG"=>2}] and value ends up a nil object, but I'm not sure why or how to correct it.
The hash is ending up as the value of key in your example because the YAML file declares DEBUG: 2 as a list member of variables. This translates to variables being an array with a single hash member.
Try changing the template code to this:
<%= #vars[0].each {|key, value| puts "env[" + key + " = " + value } %>
Or try changing the YAML to this and not changing the template code:
variables:
DEBUG: 2
Either change will get your template loop iterating over the hash that you are expecting.
I have to upload multiple files as form request. I am using the Rest Client to post my request. I am able to upload single file but I am not sure how to add multiple files in a single request.
I searched/googled for such option and I am not finding any solution that solves my problem.
Below is my code.
It has variable argument (*yamlfile) which takes one or more files. I have to upload all the files together.
The issue now is , I am getting syntax error when I add the loop to extract the file within the payload.
my assumption is now to form this outside the payload and include it inside the payload block but I am not sure how to do it.
Can someone help me with that.
( I have tried net/http/post/multipart library too and I don't find much documents around it)
def uploadRest(endpoint,archive_file_path,,yaml_file_path,*yamlfile)
$arg_len=yamlfile.length
request = RestClient::Request.new(
:method => :post,
:url => endpoint,
:payload => {
:multipart => true,
:job_upload_archive => File.new(archive_file_path,'rb'),
:job_upload_path => "/tmp",
# Trying to add multiple file, but I get syntax error
yamlfile.each_with_index { |yaml, index|
:job_upload_yaml_file+index => File.new("#{yaml_file_path}/#{pmml}")
}
})
response=request.execute
puts response.code
end
uploadRest(endpoint,archive_file_path,yaml_file_path,*yamlfile)
#files=Array.new
yamlfile.each{ |yaml_file|
#files.push(File.new("#{yaml_file_path}/#{yaml_file}"))
}
request = RestClient::Request.new(
:method => :post,
:url => endpoint,
:payload => { :multipart => true, :job_upload_archive => File.new(archive_file_path,'rb'),
:job_upload_path => "/tmp", :job_upload_yaml_file => #files })
response=request.execute
end
I had a similar problem and was able to get this to work by passing an array of arrays as a requests.
file1 = File.new("#{yaml_file_path}/#{yaml_file1}", 'rb')
file2 = File.new("#{yaml_file_path}/#{yaml_file}", 'rb')
request_body = [["files", file1], ["files", file2]]
RestClient.post url, request_body, request_headers
There were two issues with your question code:
1) Attempt to add a symbol to an integer
2) Attempt to insert contents of yamlfile direct into the hash (because that is what yamlfile.each_with_index returns, as opposed to how it calls your block. The return value from the block is not used)
Both of these code issues read as if you have gained experience in HAML or another templating language, and are using structures/ideas that would work in that?
There are lots of possble solutions in Ruby, but a simple approach to build up the hash in parts, as opposed to generate it in one go with clever hash-returning routines embedded. Try something like this:
payload_hash = {
:multipart => true,
:job_upload_archive => File.new(archive_file_path,'rb'),
:job_upload_path => "/tmp",
}
# This does not use the return value from each_with_index, instead it relies
# on the block to make changes to the hash by adding new key/value pairs
yamlfile.each_with_index { |yaml, index|
# This concatenates two strings, and then converts the combined
# string into the symbol that you want
file_key = ("job_upload_yaml_file"+index.to_s).to_sym
payload_hash[file_key] = File.new("#{yaml_file_path}/#{yaml}")
}
request = RestClient::Request.new(
:method => :post,
:url => endpoint,
:payload => payload_hash
)
For added code cleanliness, you could make the first two parts a separate method, and call it where it currently has payload_hash.
This should get you over current syntax hurdles. However, I have made no attempt to check whether this will allow you to upload multiple files via RESTClient.
Section1:
#params = {
"FacialImage" => UploadIO.new(File.new('C:\temp\ATDD\Test\test\sachin.jpg'), "image/jpeg"),
"Login" => UploadIO.new(File.new('C:\temp\ATDD\Test\test\login.txt'), "application/json")
}