How to resolve a mixlib-shellout gem version conflict? - ruby

Question

I am trying use to Kitchen/Chef to deploy. I am actually not sure if I've touched Kitchen yet.
My understanding is gem list lists all the gems in the repository, and a project fetches any gems it wants from this repository.
After I ran gem install bundler and gem install berkshelf I have a mixlib-shellout (2.2.6) shown by gem list | grep shellout.
Then I ran bundle install and based on my Gemfile, "mixlib-shellout (1.6.1)" is installed. Then I have "mixlib-shellout (2.2.6, 1.6.1)" shown by gem list | grep shellout.
From Gemfile.lock, I believe 1.6.1 is the version wanted. Then I ran berks install, an exception was thrown, which was:
Unable to activate chef-11.18.12, because mixlib-shellout-2.2.6 conflicts with mixlib-shellout (~> 1.4) - 11.18.12 - (Gem::LoadError)
I don't understand why it tries to fetch 2.2.6 instead of 1.6.1.
My Gemfile
source 'https://rubygems.org'
group :development do
gem 'berkshelf', '~> 4.0'
gem 'berkshelf-api-client', '~> 2.0'
gem 'thor'
gem 'chef', '~> 11.16'
gem 'test-kitchen', '~> 1.2'
gem 'kitchen-vagrant', '>= 0.16'
gem 'serverspec'
gem 'infrataster'
gem 'busser'
gem 'busser-serverspec'
gem 'chefspec', '~>4'
gem 'rubocop'
gem 'foodcritic'
end
gem 'chef-vault', '~> 2.2'
Gemfile.lock
GEM
remote: https://rubygems.org/
specs:
addressable (2.3.8)
ast (2.2.0)
berkshelf (4.1.0)
addressable (~> 2.3.4)
berkshelf-api-client (~> 2.0)
buff-config (~> 1.0)
buff-extensions (~> 1.0)
buff-shell_out (~> 0.1)
celluloid (= 0.16.0)
celluloid-io (~> 0.16.1)
cleanroom (~> 1.0)
faraday (~> 0.9.0)
httpclient (~> 2.6.0)
minitar (~> 0.5.4)
octokit (~> 4.0)
retryable (~> 2.0)
ridley (~> 4.3)
solve (~> 2.0)
thor (~> 0.19)
berkshelf-api-client (2.0.0)
faraday (~> 0.9.1)
httpclient (~> 2.6.0)
buff-config (1.0.1)
buff-extensions (~> 1.0)
varia_model (~> 0.4)
buff-extensions (1.0.0)
buff-ignore (1.1.1)
buff-ruby_engine (0.1.0)
buff-shell_out (0.2.0)
buff-ruby_engine (~> 0.1.0)
busser (0.7.1)
thor (<= 0.19.0)
busser-serverspec (0.5.9)
busser
capybara (2.6.2)
addressable
mime-types (>= 1.16)
nokogiri (>= 1.3.3)
rack (>= 1.0.0)
rack-test (>= 0.5.4)
xpath (~> 2.0)
celluloid (0.16.0)
timers (~> 4.0.0)
celluloid-io (0.16.2)
celluloid (>= 0.16.0)
nio4r (>= 1.1.0)
chef (11.18.12)
chef-zero (~> 2.2, >= 2.2.1)
diff-lcs (~> 1.2, >= 1.2.4)
erubis (~> 2.7)
ffi-yajl (~> 1.2)
highline (~> 1.6, >= 1.6.9)
mime-types (~> 1.16)
mixlib-authentication (~> 1.3)
mixlib-cli (~> 1.4)
mixlib-config (~> 2.0)
mixlib-log (~> 1.3)
mixlib-shellout (~> 1.4)
net-ssh (~> 2.6)
net-ssh-multi (~> 1.1)
ohai (~> 7.4)
plist (~> 3.1.0)
pry (~> 0.9)
rest-client (>= 1.0.4, <= 1.6.7)
chef-vault (2.7.1)
chef-zero (2.2.1)
ffi-yajl (~> 1.1)
hashie (~> 2.0)
mixlib-log (~> 1.3)
rack
chefspec (4.5.0)
chef (>= 11.14)
fauxhai (~> 3.0, >= 3.0.1)
rspec (~> 3.0)
cleanroom (1.0.0)
cliver (0.3.2)
coderay (1.1.0)
diff-lcs (1.2.5)
erubis (2.7.0)
faraday (0.9.2)
multipart-post (>= 1.2, < 3)
faraday_middleware (0.10.0)
faraday (>= 0.7.4, < 0.10)
fauxhai (3.1.0)
net-ssh
ffi (1.9.10)
ffi-yajl (1.4.0)
ffi (~> 1.5)
libyajl2 (~> 1.2)
foodcritic (6.0.0)
erubis
gherkin (~> 2.11)
nokogiri (>= 1.5, < 2.0)
rake
rufus-lru (~> 1.0)
treetop (~> 1.4)
yajl-ruby (~> 1.1)
gherkin (2.12.2)
multi_json (~> 1.3)
hashie (2.1.2)
highline (1.7.8)
hitimes (1.2.3)
httpclient (2.6.0.1)
infrataster (0.3.2)
capybara
faraday
faraday_middleware (>= 0.10.0)
net-ssh
net-ssh-gateway
poltergeist
rspec (>= 2.0, < 4.0)
thor
ipaddress (0.8.2)
json (1.8.3)
kitchen-vagrant (0.19.0)
test-kitchen (~> 1.4)
libyajl2 (1.2.0)
method_source (0.8.2)
mime-types (1.25.1)
mini_portile2 (2.0.0)
minitar (0.5.4)
mixlib-authentication (1.4.0)
mixlib-log
rspec-core (~> 3.2)
rspec-expectations (~> 3.2)
rspec-mocks (~> 3.2)
mixlib-cli (1.5.0)
mixlib-config (2.2.1)
mixlib-log (1.6.0)
mixlib-shellout (1.6.1)
molinillo (0.2.3)
multi_json (1.11.2)
multipart-post (2.0.0)
net-scp (1.2.1)
net-ssh (>= 2.6.5)
net-ssh (2.9.4)
net-ssh-gateway (1.2.0)
net-ssh (>= 2.6.5)
net-ssh-multi (1.2.1)
net-ssh (>= 2.6.5)
net-ssh-gateway (>= 1.2.0)
net-telnet (0.1.1)
nio4r (1.2.1)
nokogiri (1.6.7.2)
mini_portile2 (~> 2.0.0.rc2)
octokit (4.2.0)
sawyer (~> 0.6.0, >= 0.5.3)
ohai (7.4.1)
ffi (~> 1.9)
ffi-yajl (~> 1.1)
ipaddress
mime-types (~> 1.16)
mixlib-cli
mixlib-config (~> 2.0)
mixlib-log
mixlib-shellout (~> 1.2)
systemu (~> 2.6.4)
wmi-lite (~> 1.0)
parser (2.3.0.3)
ast (~> 2.2)
plist (3.1.0)
poltergeist (1.9.0)
capybara (~> 2.1)
cliver (~> 0.3.1)
multi_json (~> 1.0)
websocket-driver (>= 0.2.0)
polyglot (0.3.5)
powerpack (0.1.1)
pry (0.10.3)
coderay (~> 1.1.0)
method_source (~> 0.8.1)
slop (~> 3.4)
rack (1.6.4)
rack-test (0.6.3)
rack (>= 1.0)
rainbow (2.1.0)
rake (10.5.0)
rest-client (1.6.7)
mime-types (>= 1.16)
retryable (2.0.3)
ridley (4.4.1)
addressable
buff-config (~> 1.0)
buff-extensions (~> 1.0)
buff-ignore (~> 1.1)
buff-shell_out (~> 0.1)
celluloid (~> 0.16.0)
celluloid-io (~> 0.16.1)
erubis
faraday (~> 0.9.0)
hashie (>= 2.0.2, < 4.0.0)
httpclient (~> 2.6)
json (>= 1.7.7)
mixlib-authentication (>= 1.3.0)
retryable (~> 2.0)
semverse (~> 1.1)
varia_model (~> 0.4.0)
rspec (3.4.0)
rspec-core (~> 3.4.0)
rspec-expectations (~> 3.4.0)
rspec-mocks (~> 3.4.0)
rspec-core (3.4.2)
rspec-support (~> 3.4.0)
rspec-expectations (3.4.0)
diff-lcs (>= 1.2.0, < 2.0)
rspec-support (~> 3.4.0)
rspec-its (1.2.0)
rspec-core (>= 3.0.0)
rspec-expectations (>= 3.0.0)
rspec-mocks (3.4.1)
diff-lcs (>= 1.2.0, < 2.0)
rspec-support (~> 3.4.0)
rspec-support (3.4.1)
rubocop (0.37.0)
parser (>= 2.3.0.2, < 3.0)
powerpack (~> 0.1)
rainbow (>= 1.99.1, < 3.0)
ruby-progressbar (~> 1.7)
unicode-display_width (~> 0.3)
ruby-progressbar (1.7.5)
rufus-lru (1.0.5)
safe_yaml (1.0.4)
sawyer (0.6.0)
addressable (~> 2.3.5)
faraday (~> 0.8, < 0.10)
semverse (1.2.1)
serverspec (2.29.1)
multi_json
rspec (~> 3.0)
rspec-its
specinfra (~> 2.48)
sfl (2.2)
slop (3.6.0)
solve (2.0.2)
molinillo (~> 0.2.3)
semverse (~> 1.1)
specinfra (2.50.4)
net-scp
net-ssh (>= 2.7, < 3.1)
net-telnet
sfl
systemu (2.6.5)
test-kitchen (1.4.2)
mixlib-shellout (>= 1.2, < 3.0)
net-scp (~> 1.1)
net-ssh (~> 2.7, < 2.10)
safe_yaml (~> 1.0)
thor (~> 0.18)
thor (0.19.0)
timers (4.0.4)
hitimes
treetop (1.6.3)
polyglot (~> 0.3)
unicode-display_width (0.3.1)
varia_model (0.4.1)
buff-extensions (~> 1.0)
hashie (>= 2.0.2, < 4.0.0)
websocket-driver (0.6.3)
websocket-extensions (>= 0.1.0)
websocket-extensions (0.1.2)
wmi-lite (1.0.0)
xpath (2.0.0)
nokogiri (~> 1.3)
yajl-ruby (1.2.1)
PLATFORMS
ruby
DEPENDENCIES
berkshelf (~> 4.0)
berkshelf-api-client (~> 2.0)
busser
busser-serverspec
chef (~> 11.16)
chef-vault (~> 2.2)
chefspec (~> 4)
foodcritic
infrataster
kitchen-vagrant (>= 0.16)
rubocop
serverspec
test-kitchen (~> 1.2)
thor
BUNDLED WITH
1.11.2

Answer 1

You do not want to be bundling chef-11.18.12 with test-kitchen and berkshelf. You don't need to bundle chef with those tools, and you definitely will run into issues trying to bundle chef-11 with them.
The problem with creating a Gemfile with all of your cli tools in them is that those tools never activate the entire set of gems all at once. So test-kitchen does not depend on chef, and berkshelf only recently started pulling in chef-config as a library. When you try to create a bundle you are stating that every gem must be capable of being loaded into the same ruby process with every other gem. Even for the latest set of cli tools this is something which will frequently break. If you add in chef-11.18.12 then you are picking a gem with a frozen set of gem requirements from April of last year (about 9 months ago), and its highly likely that the latest test-kitchen and berkshelf versions will pull in versions of mixlib-shellout that are incompatible with that version of chef. It gets worse because we made breaking changes to mixlib-shellout and chef-11 is forever pinned on an old branch of mixlib-shellout that is not getting updates, while test-kitchen and berkshelf have moved on.
This is why ChefDK exists which is to give a consistent set of command line tools, in something that looks superficially like a bundle, but which is not a bundle. If test-kitchen or berkshelf don't load chef into memory simultaneously, then it is fine to include different versions of those gems which may 'conflict' in their use of mixlib-shellout but will load into different ruby process just fine. Since they never all try to load into the same ruby process the 'conflict' over mixlib-shellout never becomes a real one.
Also, we do try to take a bit of care that they all load together correctly and we don't wind up with multiple versions of mixlib-shellout in the ChefDK, but sometimes this is difficult to avoid.
What you are probably trying to do is bundle everything together because you want to have the latest toolchain, but want to use chef-11 because your production is chef-11. But the issue there is that when you are using berkshelf or test-kitchen there is no requirement to use chef-11 in that ruby process that you are launching. When TK starts, it should start up a virt, and will then install chef on that virt and launch it -- you should be making sure that virt has chef-11 so you can test your cookbooks, not that your TK process is bundled with chef-11. Whatever chef gem TK is bundled with is completely irrelevant to how it operates.
If you're trying to test chef-11 with test-kitchen the right way to do that is by setting the require_chef_omnibus config parameter to "11.18.12" in your kitchen.yml file. That will cause chef-11 to get installed on the virt you are trying to provision.
+1 on the recommendation to simply start by installing ChefDK and not trying to bundle your own toolset.
It is even recommended to use chef-12 knife and ChefDK on your workstation to manage chef-11 virts/hosts. There is one gotcha where chefspec really does run on your workstation and testing chefspec against chef-12 when your production is chef-11 could allow certain kinds of failures to go undetected. You could bundle chefspec against chef-11 and do testing that way -- however, its most likely better to stick with full integration testing with test-kitchen to find those problems, and just use chefspec as quick flame tests (or not use it at all -- unit testing and mocking out your resource collection are conceptually fairly advanced topics). Until you actually hit and issue with chefspec, its best not to worry about the chef version on your workstation and just use the latest chef version for knife.