I have successfully deployed my application using AWS OpsWorks, now I am trying to implement a custom Chef Cookbook that will allow me to set the bash environment variables. I have setup the Git repo the cookbook is being updated with OpsWorks. I generated the cookbook using the knife command on my dev box which is really just the directory structure with a recipes/default.rb file containing a few lines of code.
When I try to do something like the following I seem to keep getting errors
node[:deploy].each do |application, deploy|
deploy = node[:deploy][application]
command "ls -la"
end
(Note: ls -la is just for testing i know this will not set the environment variables)
I get the following error: ERROR: Caught exception during execution of custom recipe: xyz-enviroment: NoMethodError - undefined method command' for #<Chef::Recipe:0x7feb59200c00> - /opt/aws/opsworks/releases/20130328224322_109/vendor/bundle/ruby/1.8/gems/chef-0.9.15.5/bin/../lib/chef/mixin/recipe_definition_dsl_core.rb:56:in method_missing
Also if I try something like
execute "setting up the enviroment" do
# TODO: Add code that does something here
end
I get the following error:
execute[setting up the enviroment] (/opt/aws/opsworks/current/site-cookbooks/xyz-enviroment/recipes/default.rb:18:in `from_file') had an error:
No such file or directory - setting up the enviroment
I'm new to Chef so I'm sure there is something simple I'm doing wrong I just haven't been able to figure it out. Thanks in advance for the help.
I already solved my issue before seeing the responses below, they may have worked to solve the issue but I don't have the time to go back and try them now.
My solution was to use a Chef template to create an initializer file to set the variables when rails boots up the application.
# deafult.rb
node[:deploy].each do |application, deploy|
deploy = node[:deploy][application]
execute "restart Rails app #{application}" do
cwd deploy[:current_path]
command node[:opsworks][:rails_stack][:restart_command]
action :nothing
end
template "#{deploy[:deploy_to]}/current/config/initializers/dev_enviroment.rb" do
source "dev_enviroment.erb"
cookbook 'dev-enviroment'
group deploy[:group]
owner deploy[:user]
variables(:dev_env => deploy[:dev_env])
notifies :run, resources(:execute => "restart Rails app #{application}")
only_if do
File.exists?("#{deploy[:deploy_to]}") && File.exists?("#{deploy[:deploy_to]}/current/config/")
end
end
end
dev_enviroment.erb
ENV['VAR1'] = "<%= #dev_env[:VAR1] %>"
ENV['VAR2'] = "<%= #dev_env[:VAR2] %>"
The custom Chef JSON used in the Opsworks stack layer:
{
"deploy": {
"myapp": {
"dev_env": {
"VAR1": "INFO1",
"VAR2": "INFO2",
}
}
}
}
You didn't specify what command to run, so it's actually trying to run setting up the environment, which isn't a valid command.
Try instead specifying the command attribute inside the block:
execute "setting up the enviroment" do
command "/path/to/command --flags"
end
Alternatively, set the resource name to the command itself:
execute "/path/to/command --flags" do
# TODO: Add code that does something here
end
Your second question was correctly answered by clb. As for your first, 'command' is not a valid chef resource, you want something like:
node[:deploy].each do |application, deploy|
deploy = node[:deploy][application]
execute "running a command for #{application}" do
command "ls -la"
end
end
OpsWorks has a new feature, where you can add Environmentvariables on the App
You can access them via
node[:deploy]['appshortname'][:environment_variables][:variable_name]
( see http://docs.aws.amazon.com/opsworks/latest/userguide/attributes-json-deploy.html#attributes-json-deploy-app-environment )
So you can directly set them for your chef context like this:
node[:deploy]['magento']['environment_variables'].each {|key, value| ENV[key]=value }
And you can "dump" that into a shell script for example like this:
file "#{release_path}/environment.sh" do
content ENV.reduce("#!/bin/bash\n") { |a, e| a + "export #{e[0]}=\"#{e[1]}\"\n" }
mode '0775'
end
We do something similar, and solved it by adding our environment variables to the json config file in Opsworks dashboard. In chef, we write the file on deploy with this template: https://github.com/octocall/opsworks-cookbooks/blob/34e60267a4d3b5b9cf84e91829cc80c98c26f8ed/deploy/definitions/opsworks_rails.rb#L26, and then we symlink it using the "symlink_before_migrate" property of the json file, like this:
{
"deploy": {
"APPNAME": {
"symlink_before_migrate": {
"config/.env": ".env"
}
}
}
}
Related
I have the following code to figure out where Java is located on the box. Java comes with our application and what Java version that is included with the application differs.
def app_java_home
if Dir.exist?("#{app_home}/jre-server/linux")
Dir.chdir("#{app_home}/jre-server/linux") do
Dir.glob('jdk*').select { |f| File.directory? f }[0]
end
end
end
Then, in my cookbook I have
aws_s3_file "#{app_download_path}/#{app_s3['archive_file']}" do
bucket app_s3['bucket']
remote_path app_s3['remote_path']
region aws_region
not_if { ::Dir.exists?(app_bin_dir) }
not_if { ::File.exists?("#{app_download_path}/#{app_s3['archive_file']}") }
end
execute 'extract' do
user 'root'
command "unzip #{app_download_path}/#{app_s3['archive_file']} > /dev/null"
not_if { ::Dir.exists?("#{app_home}/ourapp") }
only_if { ::File.exists?("#{app_download_path}/#{app_s3['archive_file']}") }
end
execute 'move' do
user 'root'
command "mv #{app_download_path}/ourapp/ #{app_install_path}"
not_if { ::Dir.exists?(app_home) }
end
cookbook_file "#{app_java_home}/jre/lib/security/local_policy.jar" do
source %W[#{app_release}/local_policy.jar default/local_policy.jar]
owner app_user_name
group app_group_name
mode 0755
end
cookbook_file "#{app_java_home}/jre/lib/security/US_export_policy.jar" do
source %W[#{app_release}/US_export_policy.jar default/US_export_policy.jar]
owner app_user_name
group app_group_name
mode 0755
end
However, the two cookbook_file resources fails because it can't find the directory:
No such file or directory # dir_chdir - /ourapp/jre-server/linux/
After a lot of googling, I've come to the conclusion that it's a .. "missmatch" (?) between compile time and run time of the recipes. Basically, if I understand it correctly, it tries to run the cookbook_file resource(s) first but fails. So never downloads, unpacks and installs the app artefact.
I've tried running app_java_home when the directory exists, and it does seem to work the way I want it..
I tried putting the cookbook_file resources in a ruby_block, but then I instead get:
undefined method `cookbook_file' for Chef::Resource::RubyBlock
The app_java_home .. function (?) used to look like this:
def app_java_home
"#{app_home}/jre-server/linux/#{jdk_version}"
end
Where jdk_version came from the databag. This worked fine, but we have a long standing bug/feature request in our system where it sometimes happens that "they" get the version they put in the databag wrong, causing all sorts of problems.. So they want a way to remove this dependency and instead "figure this out" dynamically.
Ruby and Chef isn't my forte, so I'm not sure what to try next. I have found references to Chef::Resources::CookbookFile (which, if I understand it, could/should be used inside ruby_blocks), but can't find any examples or documentation about it. The link on RubyDocs is broken.
Adding an answer here for a better explanation.
Any (Ruby) code that is not within any of the Chef resources, will run in Compile phase
All resource declarations will run in Convergence phase in the order they are defined
Thankfully, there is a way to make resources run in Compile phase if so required. Though IMHO it should be done sparingly and in exceptional cases.
As per your comment aws_s3_file and execute resources are the ones that unpack the app (and create the directory). In this case, it seems you want them to run in compile phase.
Prior to Chef client 16.0
Use the run_action option with the action that should be performed at the compile time. For example execute resource takes action :run:
# Note action ":nothing" and "run_action"
execute 'extract' do
user 'root'
command "unzip #{app_download_path}/#{app_s3['archive_file']} > /dev/null"
not_if { ::Dir.exists?("#{app_home}/ourapp") }
only_if { ::File.exists?("#{app_download_path}/#{app_s3['archive_file']}") }
action :nothing
end.run_action(:run)
Chef client 16.0 onwards
We can add a common property to the resources. Example with execute resource:
# Note the extra property "compile_time"
execute 'extract' do
user 'root'
command "unzip #{app_download_path}/#{app_s3['archive_file']} > /dev/null"
not_if { ::Dir.exists?("#{app_home}/ourapp") }
only_if { ::File.exists?("#{app_download_path}/#{app_s3['archive_file']}") }
compile_time true
end
And finally to answer the subject of the question:
Chef use cookbook_file in ruby block
This is not possible. Refer to the first point on the top. If we want Ruby code to run during converge (instead of compile), we put it within the ruby_block resource. So it can contain code like (for example):
ruby_block 'get directory' do
block do
def app_java_home
"#{app_home}/jre-server/linux/#{jdk_version}"
end
end
end
With the help of #seshadri_c, I finally managed to solve the problem! It took some doing, because I kept misunderstanding the suggestions etc.
So this is what I came up with (for posterity):
def jdk_version(required = true)
base_dir = "#{app_home}/jre-server/linux"
if Dir.exist?("#{base_dir}")
Dir.chdir("#{app_home}/jre-server/linux") do
Dir.glob("jdk*").each do |f|
if File.directory?(f)
return "#{f}"
end
end
end
end
end
def app_java_home
return "#{app_home}/jre-server/linux/#{jdk_version}"
end
Turns out I need to get just the version, individually, as well, so I rearranged the functions a bit. I'm sure it could be written much cleaner, but here the trick was to use return instead of puts/print! Well, I'm a programmer, but not a Ruby programmer so didn't know that was an option..
Then, in the cookbook, I added the .run_action() where needed. I didn't need them for the cookbook_file, which simplified things a bit:
aws_s3_file "#{app_download_path}/#{app_s3['archive_file']}" do
bucket app_s3['bucket']
remote_path app_s3['remote_path']
region aws_region
not_if { ::Dir.exists?(app_bin_dir) }
not_if { ::File.exists?("#{app_download_path}/#{app_s3['archive_file']}") }
end.run_action(:create)
execute 'extract' do
user 'root'
command "unzip #{app_download_path}/#{app_s3['archive_file']} > /dev/null"
not_if { ::Dir.exists?("#{app_home}/app") }
only_if { ::File.exists?("#{app_download_path}/#{app_s3['archive_file']}") }
end.run_action(:run)
execute 'move' do
user 'root'
command "mv #{app_download_path}/app/ #{app_install_path}"
not_if { ::Dir.exists?(app_home) }
end.run_action(:run)
# JCE Unlimited Strength Jurisdiction Policy Files
cookbook_file "#{app_java_home}/jre/lib/security/local_policy.jar" do
source %W[#{app_release}/local_policy.jar default/local_policy.jar]
owner app_user_name
group app_group_name
mode 0755
end
cookbook_file "#{app_java_home}/jre/lib/security/US_export_policy.jar" do
source %W[#{app_release}/US_export_policy.jar default/US_export_policy.jar]
owner app_user_name
group app_group_name
mode 0755
end
With all that, everything is running exactly when they're supposed to and everything seems to be working.
I try to read out the current recipe name while chef-client run and to store it in a variable or node attribute wihtin an recipe. Until yet i just found a way storing it into a node attribute but it always fails. This is my code:
ruby_block "Fetch Recipe Name From Run List" do
block do
Chef::Resource::RubyBlock.send(:include, Chef::Mixin::ShellOut)
s = shell_out("echo \"#{node['expanded_run_list']}\" | awk -F '::' '{print substr($3, 1, length($3)-1)}'" )
node.default['sftp-selfmade']['extracted_recipe'] = s.stdout
end
end
extracted_recipe = node['sftp-selfmade']['extracted_recipe']
# To debug the output of the node attribute.
execute 'TEST' do
command "echo \"TEST #{extracted_recipe}\""
end
Output:
* execute[TEST] action run
[execute] TEST
- execute echo "TEST "
Output should be:
- execute echo "TEST <Name-Of-Extracted-Recipe>"
I tried lot's of things as also storing the s.stdout output in a variable but this throws an NoMethodError during compiling stage. Also tried to use stronger values like node.override - this works but only by setting node.normal first and the setting it to node.override but this is not a satisfying solution to do this everytime within cookbook code again for deploying to new hosts. Tried also a solution reloading OHAI. But this also did not work. On a completely new host it also doesn't work after a 2nd chef-client run in case of attributes have been set then after first run.
Is there somebody who can help me out?
Found out the following solution:
expandedrecipe = node['expanded_run_list'].select{ |e| e.include? 'sftp-selfmade' }.first.split('::').last
That does the trick.
So, I am trying to run the test but I am getting an error says.
Aruba::LaunchError:Command "seedly-calculator.rb" not found in PATH-variable
-seedly-calculator
-bin
-src
-seedly-calculator.rb
I have tried to change the path in rake file but it doesn't work.
My seedly-calculator.rb file is in the root directory.
require "rspec/core/rake_task"
namespace :spec do
desc "Run the functional suite against the CLI"
RSpec::Core::RakeTask.new(:functional, [] => [:set_path])
task :set_path do
project_bin_dir = File.join(File.dirname(File.expand_path(__FILE__)), '..', 'bin')
ENV['PATH'] = project_bin_dir + ':'+ ENV['PATH']
end
end
it shows error like:
Failure/Error: let(:command) { run "seedly-calculator.rb" }
Aruba::LaunchError:
Command "seedly-calculator.rb" not found in PATH-variable "/Users/bilaltariq/Desktop/seedly-calculator/functional_spec/bin:/Users/bilaltariq/Desktop/seedly-calculator/functional_spec/exe:/Users/bilaltariq/.rbenv/versions/2.6.2/lib/ruby/gems/2.6.0/bin:/Users/bilaltariq/Desktop/seedly-calculator/functional_spec/../bin:/Users/bilaltariq/.rbenv/versions/2.6.2/bin:/usr/local/Cellar/rbenv/1.1.1/libexec:/Users/bilaltariq/.rbenv/shims:/Users/bilaltariq/.asdf/shims:/Users/bilaltariq/.asdf/bin:/usr/local/bin:/Users/bilaltariq/.bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/MacGPG2/bin".
I expect it to hit the file so i can write some test.
am i doing something wrong?
require 'spec_helper'
RSpec.describe 'Command Validation', type: :aruba do
let(:command) { run "seedly-calculator.rb" }
it "wrong/missing arguments" do
command.write("lookup\n")
stop_all_commands
expect(command.output).to end_with("Missing bank_name argument.\n")
end
end
seedly-calculator.rb:
#!/usr/bin/env ruby
# Complete bin/setup so that after it is
# run, ruby seedly-calculator.rb can be used to launch
# it.
# frozen_string_literal: true
require_relative './src/runner'
if !ARGV.length.zero?
input = ARGV
Runner.new.send('process_input', input)
else
puts "Arguments required!."
end
Update
To run a ruby script using run you need to make sure your ruby script is executable and contains a shebang so your system knows to run it with ruby. Here's example from this starter example
#!/usr/bin/env ruby
file = ARGV[0]
if file.nil? || file.empty?
abort "aruba-test-cli [file]: Filename is missing"
elsif !File.exist? file
abort "aruba-test-cli [file]: File does not exist"
end
puts File.read(file).chomp
So in your case you'll need to add this to the first line of your seedly-calculator.rb file
#!/usr/bin/env ruby
Then run this from command line to make it executable.
chmod +x #!/usr/bin/env ruby
I made a simple example forked off the one I reffed above. See this commit
Rspec convention is that it should match the same file structure of your project. It is not a good idea to set PATH manually.
Rake tasks are normally put in a tasks folder so you should have in project root a tasks folder
my_project/tasks/something.rake
Then you should have a spec folder that matches
my_project/spec/tasks/something_spec.rb
Then you should be able to get rid of task :set_path do end block and just run the spec without that.
You should also have a Gemfile to load your gems, run bundle install then invoke your test with
bundle exec rspec spec/tasks/sometask_spec.rb
I am using Amazon opsworks and struggling to get it working through a single script, I have created a script named clamav.rb. The content of script is:
yum_package 'clamav' do
action :install
end
yum_package 'clamav-update' do
action :install
end
file_names = ['/etc/freshclam.conf']
file_names.each do |file_name|
text = File.read(file_name)
replace = text.gsub("Example", "#Example")
# To merely print the contents of the file, use:
puts replace
# To write changes to the file, use:
File.open(file_name, "w") {|file| file.puts replace }
end
execute "Run Freshclam" do
command "/usr/bin/freshclam"
end
When I execute the above script it stuck with an error:
[2016-08-01T13:02:36+00:00] ERROR: Running exception handlers
[2016-08-01T13:02:36+00:00] ERROR: Exception handlers complete
[2016-08-01T13:02:36+00:00] FATAL: Stacktrace dumped to /var/lib/aws/opsworks/cache.stage2/chef-stacktrace.out
[2016-08-01T13:02:36+00:00] ERROR: No such file or directory - /etc/freshclam.conf
[2016-08-01T13:02:36+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
but when I divide the script in two parts it run very well, like creating separate script for yum packages and separate for configuration change.
You're being bitten by Chef's two-pass loading model. At that point in the code, the package hasn't been installed yet. Check out https://coderanger.net/two-pass/ for more details on that, but to fix your actual problem, use the line cookbook which has resources for this kind of search and replace in files, which will handle sequencing correctly for you.
I resolved this problem, below is the solution
code: replaced my older code with this
File.open('/etc/freshclam.conf', "r") do |aFile|
if aFile
text = File.read('/etc/freshclam.conf')
replace = text.gsub("Example", "#Example")
# To merely print the contents of the file, use:
puts replace
# To write changes to the file, use:
File.open('/etc/freshclam.conf', "w") {|file| file.puts replace }
else
puts "Unable to open file!"
end
end
i would like to show you my use case and then discuss possible solutions:
Problem A:
i have 2 recipes, "a" and "b".. "a" installs some program on my file system (say at "/usr/local/bin/stuff.sh" and recipe "b" needs to run this and do something with the output.
so recipe "a" looks something like:
execute "echo 'echo stuff' > /usr/local/bin/stuff.sh"
(the script just echo(es) "stuff" to stdout)
and recipe "b" looks something like:
include_recipe "a"
var=`/usr/local/bin/stuff.sh`
(note the backquotes, var should contain stuff)
and now i need to do something with it, for instance create a user with this username. so at script "b" i add
user "#{node[:var]}"
As it happens, this doesn't work.. apparently chef runs everything that is not a resource and only then runs the resources so as soon as i run the script chef complains that it cannot compile because it first tries to run the "var=..." line at recipe "b" and fails because the "execute ..." at recipe a did not run yet and so the "stuff.sh" script does not exist yet.
Needless to say, this is extremely annoying as it breaks the "Chef runs everything in order from top to bottom" that i was promised when i started using it.
However, i am not very picky so i started looking for alternative solutions to this problem, so:
Problem B: i've run across the idea of "ruby_block". apparently, this is a resource so it will be evaluated along with the other resources. I said ok, then i'd like to create the script, get the output in a "ruby_block" and then pass it to "user". so recipe "b" now looks something like:
include_recipe "a"
ruby_block "a_block" do
block do
node.default[:var] = `/usr/local/bin/stuff.sh`
end
end
user "#{node[:var]}"
However, as it turns out the variable (var) was not passed from "ruby_block" to "user" and it remains empty. No matter what juggling i've tried to do with it i failed (or maybe i just didn't find the correct juggling method)
To the chef/ruby masters around: How do i solve Problem A? How do i solve Problem B?
You have already solved problem A with the Ruby block.
Now you have to solve problem B with a similar approach:
ruby_block "create user" do
block do
user = Chef::Resource::User.new(node[:var], run_context)
user.shell '/bin/bash' # Set parameters using this syntax
user.run_action :create
user.run_action :manage # Run multiple actions (if needed) by declaring them sequentially
end
end
You could also solve problem A by creating the file during the compile phase:
execute "echo 'echo stuff' > /usr/local/bin/stuff.sh" do
action :nothing
end.run_action(:run)
If following this course of action, make sure that:
/usr/local/bin exist during Chef's compile phase;
Either:
stuff.sh is executable; OR
Execute it through a shell (e.g.: var=`sh /usr/local/bin/stuff.sh`
The modern way to do this is to use a custom resource:
in cookbooks/create_script/resources/create_script.rb
provides :create_script
unified_mode true
property :script_name, :name_property: true
action :run do
execute "creating #{script_name}" do
command "echo 'echo stuff' > #{script_name}"
not_if { File.exist?(script_name) }
end
end
Then in recipe code:
create_script "/usr/local/bin/stuff.sh"
For the second case as written I'd avoid the use of a node variable entirely:
script_location = "/usr/local/bin/stuff.sh"
create_script script_location
# note: the user resources takes a username not a file path so the example is a bit
# strange, but that is the way the question was asked.
user script_location
If you need to move it into an attribute and call it from different recipes then there's no need for ruby_blocks or lazy:
some cookbook's attributes/default.rb file (or a policyfile, etc):
default['script_location'] = "/usr/local/bin/stuff.sh"
in recipe code or other custom resources:
create_script node['script_location']
user node['script_location']
There's no need to lazy things or use ruby_block using this approach.
There are actually a few ways to solve the issue that you're having.
The first way is to avoid the scope issues you're having in the passed blocks and do something like ths.
include_recipe "a"
this = self
ruby_block "a_block" do
block do
this.user `/usr/local/bin/stuff.sh`
end
end
Assuming that you plan on only using this once, that would work great. But if you're legitimately needing to store a variable on the node for other uses you can rely on the lazy call inside ruby to do a little work around of the issue.
include_recipe "a"
ruby_block "a_block" do
block do
node.default[:var] = `/usr/local/bin/stuff.sh`.strip
end
end
user do
username lazy { "#{node[:var]}" }
end
You'll quickly notice with Chef that it has an override for all default assumptions for cases just like this.