OS Specific Puppet Unit Test - continuous-integration

I'm still getting up to speed on Puppet and rspec and all that, but...
We've currently got a CI runner that tests our Puppet module code using a Docker container running on Linux. Well, we're now delving into using Windows-specific features of Puppet, and our tests are failing. So we're wondering if there's any way to have the tests get ignored if the platform running them is Linux?
For example, if we need to run our unit tests against our code that manages the local groups (https://puppet.com/docs/pe/2018.1/managing_windows_configurations.html#manage_local_groups), is there a way so that when we run it locally on our Windows Dev boxes, it works, but when it runs on our (Linux-based) CI runner, it skips that particular test?
Per request, here is an example of the code we're looking to use to manage a local group:
class my_repo::profile::windows::remote_desktop_users (
Array $members = ['MyDomain\MyUserAccount', 'MyDomain\ARandomDomainGroup'],
) {
group{'Set local Remote Desktop Users memberships':
ensure => present,
name => 'Remote Desktop Users',
members => $members,
auth_membership => false
}
}
Note: We're using the Role-Profile pattern
The above code seems to work. It just bombs out when our unit tests run via our CI:
describe 'my_repo::profile::windows::remote_desktop_users' do
on_supported_os.select { |_, f| f[:os]['family'] == 'windows' }.each do |os, os_facts|
context "on #{os}" do
let(:facts) { os_facts }
it { is_expected.to compile }
end
end
end
Thanks

As I remarked in a comment, I was able to reproduce a test failure using the manifest and Spec code (now) presented in the question. If I understand correctly that the failure is observed only when running the unit tests and not when serving up catalogs for real, then it follows that the problem is in the test environment's configuration, which may or may not be on you.
But as for the actual question:
So we're wondering if there's any way to have the tests get ignored if the platform running them is Linux?
Sure you can. Rspec tests are written in Ruby, and you can use substantially all standard Ruby features in them, including control flow statements and mechanisms for executing system commands. Thus, as a temporary workaround, you can put your breaks-when-not-running-on-Windows tests into a conditional statement, like this:
describe 'my_repo::profile::windows::remote_desktop_users' do
on_supported_os.select { |_, f| f[:os]['family'] == 'windows' }.each do |os, os_facts|
if %x{facter os.family}.chomp == 'windows'
context "on #{os}" do
let(:facts) { os_facts }
it { is_expected.to compile }
end
end
end
end
Note in particular the use of a %x expression to execute a system command on the host where the test runs, and in it use of facter to request the specific system fact that tells you whether the test is running on Windows. That resolves the issue for me. Note that this particular implementation will require facter to be installed on your CI machine.

Related

serverspec using wrong container

I have 2 spec files that use different docker images and therefore are suppose to start separate and different docker containers to run the examples.
In the snippets below I'm using the serverspec gem to test my containers
spec/dockerfile/ember_spec.rb
require 'spec_helper'
require 'shared_examples/release'
describe 'ember' do
before(:all) do
#image = Docker::Image.build_from_dir(image_path('ember'))
set :os, family: :alpine
set :backend, :docker
set :docker_image, #image.id
set :docker_container_create_options, { 'Entrypoint' => ['/bin/sh'] }
end
describe command('ember version') do
its(:stdout) { should contain 'ember-cli: 3.3.0' }
its(:stdout) { should contain 'node: 10.10.0' }
end
include_examples 'os release', 'Alpine Linux'
end
spec/dockerfile/gerbv_spec.rb
require 'spec_helper'
require 'shared_examples/release'
describe 'gerbv' do
before(:all) do
#image = Docker::Image.build_from_dir(image_path('gerbv'))
set :os, family: :debian
set :backend, :docker
set :docker_image, #image.id
set :docker_container_create_options, { 'Entrypoint' => ['/bin/sh'] }
end
describe package('gerbv') do
it { should be_installed }
end
include_examples 'os release', 'Ubuntu 18.04'
end
However when running bundle exec rspec it is quite clear that the same container is being used to run each spec file. I have confirmed this by printing out the running containers before each of the examples. This is of course causing the specs to fail for one of the files (whichever runs second).
When the files are run independently using bundle exec rspec path/to/file then all the specs pass.
Is there any way to force a container to be spun down after the examples in one file have run and a new container created for the other set of examples?
I found a way to solve the problem, albeit a pretty hacky one. The key to this problem lay in how the container is finally released. When there are no longer any references pointing to the Docker instance it will be garbage collected and the container killed and deleted. However the object instance is held in a class level variable as a singleton in the base class. It would seem to me the only way to "reset" specinfra is to call the clear method inherited on the Docker class.
In the end the following solved the problem and the correct class is being used to run each spec.
after(:all) {
Specinfra.backend.class.clear
}
It would be great to know there is a better way to access this methods without having to rely on a method not exposed through the serverspec gem.

Ruby Undefined method error - Using Beaker [duplicate]

I'm quite new to all of this. I am trying to test out a puppet module using Beaker. I keep getting this:
NoMethodError: undefined method `describe' for
#Beaker::TestCase:0x007fd6f95e6460
/Users/user1/beaker/Puppet/puppet-files/spec/classes/unit_spec.rb:3
/Users/user1/.rvm/gems/ruby-2.2.7/gems/beaker-3.24.0/bin/beaker:9
/Users/user1/.rvm/gems/ruby-2.2.7/bin/ruby_executable_hooks:15
/Users/user1/.rvm/gems/ruby-2.2.7/bin/ruby_executable_hooks:15.
This is the command that I'm running - "beaker --hosts myhost.yaml --pre-suite spec".
My unit_spec.rb contains this:
require 'puppetlabs_spec_helper/rake_tasks'
describe 'application' do
context 'applied to supported operating system' do
on_supported_os.each do |os, facts|
context "#{os}" do
let(:facts) do
facts
end
context "without any parameters" do
let(:params) {{ }}
it { is_expected.to compile.with_all_deps }
it { is_expected.to contain_class('files') }
end
end
end
end
context 'applied to unsupported operating system' do
describe 'ubuntu-14-x86_64' do
let(:facts) {{
:osfamily => 'Debian',
:operatingsystem => 'Ubuntu'
}}
it { is_expected.to raise_error(Puppet::Error, /Ubuntu not supported/) }
end
end
end
Any help would be much appreciated! Btw, I am using 'puppetlabs_spec_helper/rake_tasks' due to the fact that when I just used 'spec_helper' it gave me an error that it could not "load such file" even though it was there.
Also, I have tried doing
RSpec.Describe
That did not fix the issue either. I get the following error -
NameError: undefined local variable or method `on_supported_os' for #Class:0x007f92a61d5e58
I realise that this might be an RSpec Puppet issue, as this module was previously tested through puppet RSpec, however now I am trying to test using Beaker and not quite sure how to fully achieve that!
You're mixing up unit- and VM-based testing.
rspec-puppet (and rspec-puppet-facts' on_supported_os) are for catalog-based unit testing. They do not require a VM, and can give you quick feedback on syntax, and logic of your module.
beaker, and the recommended beaker-rspec add-on, provide full end-to-end testing capabilities, using actual VMs, and testing a complete stack deploy (as defined in you tests).
The main entry point for existing modules is usually rake. Take a look at the existing rake tasks in the module using rake -T. In a well-written module it should have tasks for both rspec-puppet (usually called spec), and beaker (often called beaker, or acceptance).
If it is your own module, you also might want to look into the new Puppet Development Kit to get the most important tools in a single installer.

Testing multiple hosts with the same test using serverspec

The Advanced Tips section of the Serverspec site shows an example of testing multiple hosts with the same test set. I've built an example of my own (https://gist.github.com/neilhwatson/81249ad393800a76a8ad), but there are problems.
The first problem is that the tests stop at the first failure rather than proceeding through the lot and keeping a tally. The second is that the failure output does not indicate on which host the failure occurred. What can I do to fix these problems and produce a final report for all hosts?
For the first issue, ServerSpec by default will run all your tests. However, since you have a loop that executes a Rake task for each environment, the first environment to have a failure causes the task to fails and so an exception is raised and the rest of your tasks don't run.
I've forked your gist and updated the Rake task to surround it with a begin/rescue.
...
begin
desc "Run serverspec to #{host}"
RSpec::Core::RakeTask.new(host) do |t|
ENV['TARGET_HOST'] = host
t.pattern = "spec/base,cfengine3/*_spec.rb"
end
rescue
end
...
For the second problem, it doesn't look like ServerSpec will output which environment the tests are running in. But since the updated Gist shows that the host gets set in the spec_helper.rb we can use that to add an RSpec configuration that sets up an after(:each) and only output the host on errors. The relevant code changes are in a fork of the gist, but basically you'll just need the below snippet in your spec_helper.rb:
RSpec.configure do |c|
c.after(:each) do |example|
if example.exception
puts "Failed on #{host_run_on}"
end
end
end

run Ruby Unit Tests with Buildr

I am using buildr for some time now, but today I came over a little problem in connection to unit testing of ruby code.
So in my buildfile I have these lines:
define "ruby-project" do
project.version = VERSION_NUMBER
project.group = GROUP
Rake::TestTask.new(:test_rb) do |t|
t.warning = true
t.verbose = true
t.test_files = FileList['test/*.rb']
end
task test => [:test_rb]
end
running buildr test actually runs the tests, what is nice. The test is actually just that:
require 'test/unit'
class TestFileParse < Test::Unit::TestCase
def test_fail
assert(false, 'test to fail')
end
end
As expected it fails, BUT what is strange for me is that buildr quits the build with that message:
sh: 2: Syntax error: Unterminated quoted string
Buildr aborted!
RuntimeError : Command failed with status (1): [/usr/bin/ruby1.9.1 -w -I"lib" -I"/var/lib/...]
Running ruby file-with-failing-test-from-above.rp does not throw a runtime error, instead it prints the test report on screen, what is what is what I want.
Question
How can I make Buildr run the unit tests without quitting with an RuntimeError if a test fails?
Greetings Philipp
Since I am not too familiar with Ruby development, what involves Rake, I was looking for the wrong question. Instead of looking for: »how to run unit test with buildr«, the question should have been »how to run unit tests with rake«, because buildr is a kind of extended Rake (similar to the »maven-ant-relationship«). So everything one can do in Rake, one can do in buildr, too. Therefore on good SO answer to run ruby unit tests in buildr is here.
Additionally it is possible to run RSpec's with buildr, therefore one has two options:
set project.test.using :rspec, what involves the use of JRuby, so one has to set JRUBY_HOME (in my case ~/.rvm/rubies/jruby-1.7.9/), or run buildr within jruby. In my case this slowed test execution down, because each time a jvm needed to be started.
Or one can use rspec's rake task in this manner. I choose this method since my tests run much faster without the jvm overhead.
n.b. as the answer implies, I switched over to rspec.
Happy Testing/Speccing!

Why RSpec runs specs twice when running from within ruby and spec file is reloaded?

I'm trying to use RSpec from within existing ruby runtime and run specs every time when file changes. This is because of JRuby and JVM startup time. To eliminate this on every run I'd like to start ruby once, then only reload changed files and run specs. I was using guard (with diffrent extensions) and watchr but all seem to suffer from an issue described below.
I nailed the issue down to RSpec itself. The problem is, when running RSpec via RSpec::Core::Runner.run several times it works fine until spec file is reloaded using load. Then RSpecs starts running specs twice.
I've created sample project showing this issue live: https://github.com/mostr/rspec_double_run_issue
Below is sample output:
ruby run_spec_in_loop.rb
Running spec from within ruby runtime
.
Finished in 0.00047 seconds
1 example, 0 failures
loading spec file via 'load' as if it was changed and we wanted changes to be picked up
Running spec from within ruby runtime
..
Finished in 0.001 seconds
2 examples, 0 failures
Is there any way to tell RSpec to clear its context between subsequent runs when run from within existing ruby runtime? I've also raised this as an issue #826 for RSpec Core project.
Summarizing the answer here in order to remove this question from the "Unanswered" filter...
Per RJHunter's observation, the explanation has been documented on the GitHub RSpec Core project here:
https://github.com/rspec/rspec-core/issues/826#issuecomment-15089030
For posterity (in case the above link dies), here are the details:
The RSpec runner is already calling load internally, your second load is what's causing the double run issue.
I quickly knocked up a script based off your example which reruns a single spec file, changes the specs to something else, then reruns them, work's correctly without the second load
See: https://gist.github.com/JonRowe/5192007
The aforementioned Gist contains:
require 'rspec'
spec_file = 'spec/sample_spec.rb'
File.open(spec_file, 'w') { |file| file.write 'describe { specify { expect(true).to eq false } }' }
1.upto(5) do |i|
puts "Running spec from within ruby runtime"
::RSpec::Core::Runner.run([spec_file], STDERR, STDOUT)
#rewriting the spec file
File.open(spec_file, 'w') { |file| file.write "describe { specify { expect(#{i}).to eq false } }" }
end

Resources