For server rebuild, I want to skip some section of my cookbook according to the results of a previous run.
For instance, we have a resource to start Weblogic servers.
weblogic_server "server_name" do
action :start
end
These startups take a lot of time during the build. I want to skip this if it was run succesfully in the last build, to avoid having to wait too much for the rebuild. Something like this:
weblogic_server "server_name" do
action :start
not_if { it_was_run_successfully_during_the_previous_run }
end
I know the best way to do it would be have script checking on weblogic servers status, but that's up to another team and I need a temporary solution.
I thought about a logfile in JSON format referencing the different steps of the build.
e.g:
{
"provisioning" : true,
"start_weblogic_servers : true,
"configuring_ohs" : false
}
In this case I would have a template resource for this logfile and then update the values during the run. Then in every run I would check this file first and skip the right section according to the values I find.
Is there a better way to go ?
What I have done in the past is to just create an empty file, if it exists then you skip it (not_if do ::File.exists?('/path/to/some_empty_file') end). You could then have some code when a build is successful or not to either create or delete these files, I realise it is probably not the best approach but it has worked for me as long as I can remember.
If you really want then you could have some script checking the server status (say on a 5 minute interval) and then adjusting that empty file accordingly (by deleting it or keeping it).
Nabeel Amjad solution worked for me. Follow these steps:
Create a file resource with action :nothing
file '/tmp/logfile' do
action: nothing
end
Set your resource to notify the file resource after running
weblogic_server 'server_name' do
action :start
notifies :create, 'file[/tmp/logfile]', :immediately
end
Add a guard not_if that will skip future execution of this resource if the file exists on the server
weblogic_server 'server_name' do
action :start
notifies :create, 'file[/tmp/logfile]', :immediately
not_if { ::File.exist?('/tmp/logfile') }
end
Related
I have the following code to figure out where Java is located on the box. Java comes with our application and what Java version that is included with the application differs.
def app_java_home
if Dir.exist?("#{app_home}/jre-server/linux")
Dir.chdir("#{app_home}/jre-server/linux") do
Dir.glob('jdk*').select { |f| File.directory? f }[0]
end
end
end
Then, in my cookbook I have
aws_s3_file "#{app_download_path}/#{app_s3['archive_file']}" do
bucket app_s3['bucket']
remote_path app_s3['remote_path']
region aws_region
not_if { ::Dir.exists?(app_bin_dir) }
not_if { ::File.exists?("#{app_download_path}/#{app_s3['archive_file']}") }
end
execute 'extract' do
user 'root'
command "unzip #{app_download_path}/#{app_s3['archive_file']} > /dev/null"
not_if { ::Dir.exists?("#{app_home}/ourapp") }
only_if { ::File.exists?("#{app_download_path}/#{app_s3['archive_file']}") }
end
execute 'move' do
user 'root'
command "mv #{app_download_path}/ourapp/ #{app_install_path}"
not_if { ::Dir.exists?(app_home) }
end
cookbook_file "#{app_java_home}/jre/lib/security/local_policy.jar" do
source %W[#{app_release}/local_policy.jar default/local_policy.jar]
owner app_user_name
group app_group_name
mode 0755
end
cookbook_file "#{app_java_home}/jre/lib/security/US_export_policy.jar" do
source %W[#{app_release}/US_export_policy.jar default/US_export_policy.jar]
owner app_user_name
group app_group_name
mode 0755
end
However, the two cookbook_file resources fails because it can't find the directory:
No such file or directory # dir_chdir - /ourapp/jre-server/linux/
After a lot of googling, I've come to the conclusion that it's a .. "missmatch" (?) between compile time and run time of the recipes. Basically, if I understand it correctly, it tries to run the cookbook_file resource(s) first but fails. So never downloads, unpacks and installs the app artefact.
I've tried running app_java_home when the directory exists, and it does seem to work the way I want it..
I tried putting the cookbook_file resources in a ruby_block, but then I instead get:
undefined method `cookbook_file' for Chef::Resource::RubyBlock
The app_java_home .. function (?) used to look like this:
def app_java_home
"#{app_home}/jre-server/linux/#{jdk_version}"
end
Where jdk_version came from the databag. This worked fine, but we have a long standing bug/feature request in our system where it sometimes happens that "they" get the version they put in the databag wrong, causing all sorts of problems.. So they want a way to remove this dependency and instead "figure this out" dynamically.
Ruby and Chef isn't my forte, so I'm not sure what to try next. I have found references to Chef::Resources::CookbookFile (which, if I understand it, could/should be used inside ruby_blocks), but can't find any examples or documentation about it. The link on RubyDocs is broken.
Adding an answer here for a better explanation.
Any (Ruby) code that is not within any of the Chef resources, will run in Compile phase
All resource declarations will run in Convergence phase in the order they are defined
Thankfully, there is a way to make resources run in Compile phase if so required. Though IMHO it should be done sparingly and in exceptional cases.
As per your comment aws_s3_file and execute resources are the ones that unpack the app (and create the directory). In this case, it seems you want them to run in compile phase.
Prior to Chef client 16.0
Use the run_action option with the action that should be performed at the compile time. For example execute resource takes action :run:
# Note action ":nothing" and "run_action"
execute 'extract' do
user 'root'
command "unzip #{app_download_path}/#{app_s3['archive_file']} > /dev/null"
not_if { ::Dir.exists?("#{app_home}/ourapp") }
only_if { ::File.exists?("#{app_download_path}/#{app_s3['archive_file']}") }
action :nothing
end.run_action(:run)
Chef client 16.0 onwards
We can add a common property to the resources. Example with execute resource:
# Note the extra property "compile_time"
execute 'extract' do
user 'root'
command "unzip #{app_download_path}/#{app_s3['archive_file']} > /dev/null"
not_if { ::Dir.exists?("#{app_home}/ourapp") }
only_if { ::File.exists?("#{app_download_path}/#{app_s3['archive_file']}") }
compile_time true
end
And finally to answer the subject of the question:
Chef use cookbook_file in ruby block
This is not possible. Refer to the first point on the top. If we want Ruby code to run during converge (instead of compile), we put it within the ruby_block resource. So it can contain code like (for example):
ruby_block 'get directory' do
block do
def app_java_home
"#{app_home}/jre-server/linux/#{jdk_version}"
end
end
end
With the help of #seshadri_c, I finally managed to solve the problem! It took some doing, because I kept misunderstanding the suggestions etc.
So this is what I came up with (for posterity):
def jdk_version(required = true)
base_dir = "#{app_home}/jre-server/linux"
if Dir.exist?("#{base_dir}")
Dir.chdir("#{app_home}/jre-server/linux") do
Dir.glob("jdk*").each do |f|
if File.directory?(f)
return "#{f}"
end
end
end
end
end
def app_java_home
return "#{app_home}/jre-server/linux/#{jdk_version}"
end
Turns out I need to get just the version, individually, as well, so I rearranged the functions a bit. I'm sure it could be written much cleaner, but here the trick was to use return instead of puts/print! Well, I'm a programmer, but not a Ruby programmer so didn't know that was an option..
Then, in the cookbook, I added the .run_action() where needed. I didn't need them for the cookbook_file, which simplified things a bit:
aws_s3_file "#{app_download_path}/#{app_s3['archive_file']}" do
bucket app_s3['bucket']
remote_path app_s3['remote_path']
region aws_region
not_if { ::Dir.exists?(app_bin_dir) }
not_if { ::File.exists?("#{app_download_path}/#{app_s3['archive_file']}") }
end.run_action(:create)
execute 'extract' do
user 'root'
command "unzip #{app_download_path}/#{app_s3['archive_file']} > /dev/null"
not_if { ::Dir.exists?("#{app_home}/app") }
only_if { ::File.exists?("#{app_download_path}/#{app_s3['archive_file']}") }
end.run_action(:run)
execute 'move' do
user 'root'
command "mv #{app_download_path}/app/ #{app_install_path}"
not_if { ::Dir.exists?(app_home) }
end.run_action(:run)
# JCE Unlimited Strength Jurisdiction Policy Files
cookbook_file "#{app_java_home}/jre/lib/security/local_policy.jar" do
source %W[#{app_release}/local_policy.jar default/local_policy.jar]
owner app_user_name
group app_group_name
mode 0755
end
cookbook_file "#{app_java_home}/jre/lib/security/US_export_policy.jar" do
source %W[#{app_release}/US_export_policy.jar default/US_export_policy.jar]
owner app_user_name
group app_group_name
mode 0755
end
With all that, everything is running exactly when they're supposed to and everything seems to be working.
The challenge prompt is above, and my latest attempt is below. The directories and files are created as expected, and the read-out after executing chef-apply multipleCopies.rb tells me the files are linked, but when I update any one of the files, the others do not follow suit. Any ideas? Here is my code:
for x in 1..3
directory "multipleCopy#{x}" do
mode '0755'
action :create
end
end
file "multipleCopy1/secret.txt" do
mode '0755'
action :create
end
for x in 2..3
link "multipleCopy#{x}/secret.txt" do
to "multipleCopy1/secret.txt"
link_type :hard
subscribes :reload, "multipleCopy1/secret.txt", :immediately
end
end
Note: For less headache, I am testing the recipe locally before uploading to the ubuntu server referenced in the prompt, which is why my file paths are different and why I have not yet included the ownership properties.
So a file hard link doesn't seem to be what the question is going for (though I would say your solution is maybe better since this is really not what Chef is for, more on that later). Instead they seem to want you to have three actually different files, but sync the contents.
So first the easy parts, creating the directories and the empty initial files. It's rare to see those for loops used in Ruby code, though it is syntactically valid:
3.times do |n|
directory "/var/save/multipleCopy#{n+1}" do
owner "ubuntu"
group "root"
mode "755"
end
file "/var/save/multipleCopy#{n+1}/secret.txt" do
owner "root
group "root"
mode "755"
end
end
But that doesn't implement the hard part of sync'ing the files. For that we need to first analyze the mtimes on the files and use the most recent as the file content to set.
latest_file = 3.times.sort_by { |n| ::File.mtime("/var/save/multipleCopy#{n+1}/secret.txt") rescue 0 }
latest_content = ::File.read("/var/save/multipleCopy#{latest_file+1}/secret.txt") rescue nil
and then in the file resource:
file "/var/save/multipleCopy#{n+1}/secret.txt" do
owner "root
group "root"
mode "755"
content latest_content
end
As for this not being a good use of Chef: Chef is about writing code which asserts the desired state of the machine. In the case of files like this, rather than doing this kind of funky stuff to check if a file has been edited, you would just say that Chef owns the file content for all three and if you want to update it, you do it via your cookbook (and then usually use a template or cookbook_file resource).
I have a recipe that looks similar to this:
...
custom_resource1 "example" do
writing stuff to a file
end
log 'File found!' do
message "Found it
level :info
notifies :run, 'custom_resource2[example]', :immediately
only_if { ::File.exists?(file) }
end
...
custom_resource1 is a big resource with other resources inside, and takes some time to complete (iterates over some data_bags and writes to a file).
Sometimes, I see that custom_resource1 fails during a chef run, but still custom_resource2 is triggered before the recipe fails.
Is there any way to ensure that custom_resource1 either failed or completed before moving on?
That isn't possible, Chef uses an entirely blocking execution model (other than the two-pass loading system). The full action method for each resource is run in order with no concurrency. You would have to post more code to isolate the actual problem.
I also thought it was strange, because the log statement was never printed out, even though custom_resource2 were triggered by the notify. The soloution was to remove the log statement and instead add:
custom_resource2 "example" do
do stuff
only_if { ::File.exists?(file) }
end
Guess it has something to do with the different chef phases
I have a following problem. The chef recipe code snippets below does not behave equally although they look the same to me in terms of pure logic.
template "Create a file if not exists" do
path "#{site_docroot}/somefile.php"
source 'somefile.php.erb'
action :create_if_missing
end
VS.
if !File.exists? "#{site_docroot}/somefile.php"
template "Create a file if not exists" do
path "#{site_docroot}/somefile.php"
source 'somefile.php.erb'
action :create
end
end
Both should create a file if it does not yet exist.
But in context of a custom recipe in Amazon OpsWorks "setup" stage, the first sollution works as expected.
But the second sollution delivers a "false positive" exactly every second time I run the recipe.
The "if" statement delivers false but the file does not exist at the end.
So I would like to know if there is any reason for that in chef or/and ruby with nesting "template" resource inside "if" block. Does "template" ressource run some kind of asynchronous?
the short answer is that chef actually runs in 2 phases. You have a compile phase and an execution (or sometimes called convergence) phase.
What this means is depending on the presence of the file the template will get inserted or not in the recipe at compile time.
Further reading:
https://docs.chef.io/chef_client.html
https://serverfault.com/questions/604719/chef-recipe-order-of-execution-redux
So what happens in your case:
first there is not file. The chef recipe (via compile phase) decides there should be a file and creates the file in the converge phase.
on the 2nd run, there is a file and the chef recipe decides (via compile again) that there should not be a file (i.e. missing template). When the node converges in the 2nd phase chef removes the file as it's trying to bring the node to the desired state.
That explains the flipping back and forth that you a see (every other run)
We do over 20 deployments a day using capistrano (actually webistrano) and we have a problem where the disk space on our servers get full of old deployment folders.
Every now and again I run the deploy:cleanup task to clean out all deployments (it keeps the last :keep_releases, currently set to 30). I would like to automate the cleanup.
One solution would be to add the following to the recipe to automatically run the cleanup after every deployment:
after "deploy", "deploy:cleanup"
But, I don't want to do this after every deployment, I'd like to limit it to only when the number of previous deployments gets to a threashold, e.g. 70. Does anyone know how I can do this?
Thoughts:
Does Capistrano provide a variable that holds the number of previous deployments?
If not, does anyone know a way to calculate it. i.e. set :num_releases, <what-can-I-put-here-to-count-previous-deployments>
Is there a way to pimp deploy:cleanup so it uses a minimum threshold, i.e. exit if < :max_releases previous deployments (where :max_releases is different from :keep_releases).
Could the except keyword be used? i.e. something like :except => { :num_releases < 70}.
Does Capistrano provide a variable that holds the number of previous deployments?
Yes, releases.length
Is there a way to pimp deploy:cleanup so it uses a minimum threshold?
Yes, here is a privately namespaced task that will trigger the normal cleanup task ONLY if a certain number of release folders have built up:
namespace :mystuff do
task :mycleanup, :except => { :no_release => true } do
thresh = fetch(:cleanup_threshold, 70).to_i
if releases.length > thresh
logger.info "Threshold of #{thresh} releases reached, runing deploy:cleanup."
deploy.cleanup
end
end
end
To have this run automatically after a deploy, put this at the top of the recipe:
after "deploy", "mystuff:mycleanup"
The good thing about this is that, before and after directives set on deploy:cleanup are executed as normal. For example we require the following:
before 'deploy:cleanup', 'mystuff:prepare_cleanup_permissions'
after 'deploy:cleanup', 'mystuff:restore_cleanup_permissions'
A quick and dirty approach using the current capistrano code:
Change the cleanup task in https://github.com/capistrano/capistrano/blob/master/lib/capistrano/recipes/deploy.rb#L405 into this:
task :cleanup, :except => { :no_release => true } do
thresh = fetch(:cleanup_threshold, 70).to_i
count = fetch(:keep_releases, 5).to_i
if thresh >= releases.length
logger.important "no old releases to clean up"
else
logger.info "threshold of #{thresh} releases reached, keeping #{count} of #{releases.length} deployed releases"
directories = (releases - releases.last(count)).map { |release|
File.join(releases_path, release) }.join(" ")
try_sudo "rm -rf #{directories}"
end
end
and then you will be able to add
set :cleanup_threshold, 70
to your deployment recipe.