Over in this question, an answer was given to analyze the Bundler dependency list. That works great, but it doesn't give you the list of packages and versions actually being used, because of ">=" dependencies. Is there a way to get the list of packages and versions actually being used rather than just what the dependencies are?
Looks like the way to do this is similar to what was posted in the other question:
Rails.logger.debug "Type is " + Bundler.environment.specs.class.to_s
Rails.logger.debug "Value is " + Bundler.environment.specs.to_hash.to_s
Produces:
Type is Bundler::SpecSet
Value is {"activemodel"=>[#<Gem::Specification name=activemodel version=3.1.3>],
"actionpack"=>[#<Gem::Specification name=actionpack version=3.1.3>],
"actionmailer"=>[#<Gem::Specification name=actionmailer version=3.1.3>]}
This code will print out all of the gems and versions being used in your current environment. One thing to note about the answer in that other question is that it will return all of the dependencies, even those that aren't in your current rails environment (for example, the ones that are in your "test" gem grouping).
This code was extracted from the Bundler codebase and will do the exact same thing as bundle list from within a Rails console.
Bundler.load.specs.sort_by(&:name).each{|s| puts " * #{s.name} (#{s.version}#{s.git_version})"}; nil
If you just want an array of the dependencies, this will also suffice.
Bundler.load.specs.map{|s| "#{s.name} (#{s.version}#{s.git_version})"}
Related
For the sake of security internal Ruby gems in the Gemfile should always be referenced inside a source block so it never tries to fetch them from rubygems.org. I'd like to automate finding where people fail to do this, so would like to parse the Gemfile, find any gems that match our internal names, and check that rubygems.org isn't in their possible sources list.
source 'https://rubygems.org'
gem 'rails'
gem 'my-private-gem1' # this should be in the source block below
source PRIVATE_GEM_REPO do
gem 'my-private-gem2'
end
I've seen you can parse the Gemfile
Bundler::Definition.build('Gemfile', '', {})
But can't find anything in the returned data structure that shows me the available / allowed sources per gem
If I include the Gemfile.lock I see more source info, but it doesn't seem right because every gem lists all my sources regardless of if they're in a source block
Bundler::Definition.build('Gemfile', 'Gemfile.lock', {}).
locked_gems.
specs.
map {|g| [g.full_name, g.source.remotes.map(&:hostname).join(', ')]}
=> ["rails-6.0.3.4", "my.private.gemserver, rubygems.org"],
["my-private-gem1-1.0.0", "my.private.gemserver, rubygems.org"],
["my-private-gem2-1.0.0", "my.private.gemserver, rubygems.org"]]
Any thoughts on how to parse the Gemfile to find that my-private-gem1 is outside a source block?
Figured it out finally, just took awhile digging through the Bundler methods - and a coworker's help.
Bundler::Definition.
build('Gemfile', '', nil).
dependencies.
map {|dep| [dep.name, dep.source&.remotes&.map(&:hostname)&.join(', ')]}
=>
[["rails", nil],
["my-private-gem1", nil],
["my-private-gem2", "my.private.gemserver"]]
Now I can easily search that resulting data structure for any private gems that aren't locked down to my private gem server.
Preface
While I was writing this answer, the OP found a Bundler-specific answer. However, I offer a more generalizable solution below. This solution also offers user feedback that may make it easier to fix the file.
Finding Candidate Gems by Column Alignment, with Whitelisting
If you can safely assume that your Gemfile is always properly indented, the KISS solution may be to simply identify the gems that aren't indented within a group definition. For example:
# example Gemfile to test against
GEMFILE = <<~'EOF'
source 'https://rubygems.org'
gem 'rails'
gem 'my-private-gem1' # this should be in the source block below
source PRIVATE_GEM_REPO do
gem 'my-private-gem2'
end
EOF
# gems that are acceptable in a non-group context
whitelist = Regexp.new %w[rails sass-rails webpacker].join(?|)
UngroupedGem = Struct.new :line_no, :line_txt, :gem_name
ungrouped_gems = []
GEMFILE.lines.each_with_index do |line_txt, line_no|
next if line_txt =~ whitelist or line_txt !~ /^\s*gem/
gem_name = line_txt.match(/(?<=(['"]))(.*?)(?=\1)/)[0]
ungrouped_gems.append(
UngroupedGem.new line_no.succ, line_txt, gem_name
).compact!
end
# tell the user what actions to take
if ungrouped_gems.any?
puts "Line No.\tGem Name"
ungrouped_gems.each { printf "%d\t\t%s\n", _1.line_no, _1.gem_name }
else
puts "No gems need to be moved."
end
With this example, it will print:
Line No. Gem Name
3 my-private-gem1
5 my-private-gem2
which will give you a solid idea of what lines in the Gemfile need to be moved, and which specific gems are involved.
I am working on a gem related utility and I have observed strange results using Gem.latest_version_for method. Here are some observations under irb:
irb(main):001:0> Gem.latest_version_for('rails').to_s
=> "5.2.2"
irb(main):002:0> Gem.latest_version_for('gosu').to_s
=> "0.7.38"
Note how the first line, gets the correct version of rails, 5.2.2 as I write this and checking with rubygems.org confirms this. The query for the gosu gem returns 0.7.38 which is wildly wrong. The correct answer should be 0.14.4
I am at a loss to explain what is happening here.
I can confirm that my host is https://rubygems.org and
C:\Sites\mysh
8 mysh>ruby --version
ruby 2.3.3p222 (2016-11-21 revision 56859) [i386-mingw32]
C:\Sites\mysh
9 mysh>gem --version
2.5.2
The latest version available for i386-mingw32 platform is 0.7.38. You'll note this comports with what your ruby version is reported as.
https://rubygems.org/gems/gosu/versions
latest_version_for calls latest_spec_for, which calls Gem::SpecFetcher.spec_for_dependency with only the name of the gem as an argument. spec_for_dependency takes another argument, matching_platform, which defaults to true.
It looks like latest_version_for is scoped to your current platform thru that chain, with the matching_platform default. The gem install command might treat i386/x386 as the same/equivalent and allow them.
spec_for_dependency
if matching_platform is false, gems for all platforms are returned
You should be able to mirror the latest_spec_for method and pass in the multi_platform argument to override. Something like
dependency = Gem::Dependency.new name
fetcher = Gem::SpecFetcher.fetcher
spec_tuples, _ = fetcher.spec_for_dependency dependency, true # true added here
With the excellent help of Jay Dorsey, I think I have made some progress here. What I need to say is too large to fit in a comment and is the actual answer to the question about the odd behavior. Well at least I am pretty sure that it is.
As mentioned above: latest_version_for calls latest_spec_for, which calls Gem::SpecFetcher.spec_for_dependency.
The key is that that method then calls Gem::SpecFetcher.search_for_dependency. This is a long rambling method. I want to focus of one line that occurs after the specs have be obtained:
tuples = tuples.sort_by { |x| x[0] }
This sorts the tuples which are an array of [spec, source] arrays. It sorts them in ascending version/platform (as far as I can tell)
Now we return to the Gem class method latest_spec_for(name) and in particular the line:
spec, = spec_tuples.first
This grabs the first sub-array and keeps the spec and discards the source.
Note that it grabs the first element. The one with the lowest version number. This is normally not a problem because for the vast majority of gems, there will be only one spec present. Not so for the gosu gem. Here there are three due to the fact that gosu contains platform specific code. It seems to grab specs for the two Gem platforms ("ruby" and "x86-mingw32") and also the ruby platform (i386-mingw32).
To test my idea, I created the file glmp.rb (get last monkey patch) Here it is:
# The latest_spec_for(name) monkey patch.
module Gem
# Originally in File rubygems.rb at line 816
def self.latest_spec_for(name)
dependency = Gem::Dependency.new name
fetcher = Gem::SpecFetcher.fetcher
spec_tuples, = fetcher.spec_for_dependency dependency
spec_tuples[-1][0]
end
end
Now I know monkey patching is frowned upon, but for now this is just to test the idea. Here are my results:
36 mysh>=Gem.latest_version_for('gosu')
Gem::Version.new("0.7.38")
C:\Sites\ideas\gem_usage
37 mysh>ls
gem_latest.rb gem_usage.rb glmp.rb
C:\Sites\ideas\gem_usage
39 mysh>=require './glmp'
true
C:\Sites\ideas\gem_usage
40 mysh>=Gem.latest_version_for('gosu')
Gem::Version.new("0.14.4")
While I can use this hack to solve my problem for now, I think I will raise an issue with rubygems bringing up this matter.
I have an RSpec suite, run via Bundler, that is testing a number of different command-line applications using Aruba. It works fine ... as long as the command being tested is not itself written in Ruby using Bundler. But I cannot figure out how to prevent the RSpec suite's bundler config from interfering with the execution of commands that themselves use Bundler - at least, not without extreme measures.
I have tried various permutations of unset_bundler_env_vars and with_clean_env, to no avail. Here's an example of a technique I thought would work:
describe 'my ruby app' do
before :each { unset_bundler_env_vars }
it 'should work' do
Bundler.with_clean_env { run_simple ruby_command_name }
end
end
I also tried unset_bundler_env_vars without with_clean_env, and vice-versa, in case they interfered with each other. No dice.
The only way I've gotten it to work is to massage Aruba's copy of the environment manually, like this:
before :all do
aruba.environment.tap do |env|
if env.include? 'BUNDLE_ORIG_PATH' then
env['PATH'] = env['BUNDLE_ORIG_PATH']
%w(BUNDLE_BIN_PATH BUNDLE_GEMFILE BUNDLE_ORIG_PATH GEM_HOME RBENV_DIR
RBENV_HOOK_PATH RUBYLIB RUBYOPT).each do |key|
env.delete key
end
end
end
end
There must be a better way. Neither the test suite nor the command being tested should know or care what language the other is written in. And my test code that uses Aruba and Bundler should not need to know the details of how bundle exec affects the process environment.
So what am I doing wrong? How should I be doing this?
It looks like unset_bundler_env_vars is deprecated and replaced by delete_by_environment_variable which requires a string param (source).
You might try before :each { delete_environment_variable('BUNDLE_GEMFILE') } in your spec. If that does not work, you may need to iterate through the PATH variable list to delete each one.
In the deprecation notice, there is a work-around, though I am not sure how brittle that would be moving forward.
unset_bundler_env_vars
aruba.environment.clear.update(ENV)
Hope this helps.
I'm trying to write a plugin for ohai. It seems like a pretty straightforward task:
Ohai.plugin(:Uname) do
provides 'uname'
depends 'kernel'
collect_data do
uname Mash.new
uname[:message] = `uname -a`
end
end
To me this looks like the online examples provided by Opscode, O'Reilly and others. But here's what happens when I try to test it:
% irb -rohai
irb(main):001:0> Ohai::Config[:plugin_path] << "."
=> ["/home/ll0359/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/ohai-8.3.0/lib/ohai/plugins", "."]
irb(main):002:0> o = Ohai::System.new
=> #<Ohai::System:0x007fed82e43078 #plugin_path="", #data={}, #provides_map=#<Ohai::ProvidesMap:0x007fed82e42fd8 #map={}>, #v6_dependency_solver={}, #d82e42f38 #controller=#<Ohai::System:0x007fed82e43078 ...>, #v6_plugin_classes=[], #v7_plugin_classes=[]>, #runner=#<Ohai::Runner:0x007fed82e42ec0 #prp:0x007fed82e42fd8 #map={}>, #safe_run=true>>
irb(main):003:0> o.all_plugins
And here's where the fun begins. I get this output, over and over and over:
[2015-05-20T03:13:09+00:00] WARN: Plugin Definition Error: <./ohai_uname.rb>: collect_data already defined on platform default
[2015-05-20T03:13:09+00:00] WARN: [DEPRECATION] Plugin at ./test_ohai.rb is a version 6 plugin. Version 6 plugins will not be supported in future releases....
your plugin to version 7 plugin syntax. For more information visit here: docs.chef.io/ohai_custom.html
(the text on my second line was clipped by my screen but you get the idea)
I've tried running this code with and without the 'depends' line. Same result.
I've tried running this code with and without the Mash line, substituing 'uname uname -a' for the assignment line. Same result.
I've tried running with and without passing ":linux" as a parameter to collect_data. The only difference is I get a warning about collect_data(:linux) already being defined instead of :default.
I've tried renaming the plugin to a random 8 character identifier just in case it was tripping over being called :Uname. Same result.
I've tried passing "uname" (capital and lower) as a parameter to o.all_plugins. Same result.
So my questions are:
Why does ohai (8.3, running under Ruby 2.2.1) think this is a version 6 plugin? I can't see anything in it that would make it look like it's not version 7.
How can I get this working?
Thanks
Note to self: Next time you do this, don't try to test from the directory your plugin is in and add "." to your plugin_path. Moving to a different directory and adding the absolute path to the plugin solved the problem.
I plan to leave this up in case someone else has this happen to them.
I'm working on a infrastructure where some servers don't have access to the internet, so I have to push the packages to the local repo before declaring them to be installed on Chef.
However we've been on a situation where Chef failed to install a package since the package wasn't there on some boxes and it has been successful on some other boxes.
What I want to do is to run a Ruby/RSpec test before applying Chef config on the nodes to make sure the packages declared on the recipes do actually exist on the repo.
In order to do that I need to be able to list all the packages exists in the our recipes.
My question is: Is there anyway to list all the declared packages in Chef? I had a quick look at Chef::Platform and ChefSpec but unfortunately couldn't find anything useful to my problem.
Do you have any idea where is the best place to look at?
If you use ChefSpec you can find all the packages by calling chef_run.find_resources(:package) inside some test. See the source code. Like this:
require 'chefspec'
describe 'example::default' do
let(:chef_run) { ChefSpec::Runner.new.converge(described_recipe) }
it 'does something' do
chef_run.find_resources(:package)...
end
end
You could install one or more of the community ohai plugins. For example the following will return information about installed sofware:
debian
Redhat
windows
Once the plugins are enabled they will add additional node attributes that will be searchable from chef-server.