Get changelogs for all gems included in a Rails project - ruby

I have a Rails project that uses bundler. Many of the gems I'm included are quite a few versions behind the latest available. I know that I could just cross my fingers and run bundle update to get everything up to the latest version, taking into account dependencies, but that could potentially break parts of my app that rely on deprecated gem methods, and might not be worth the effort.
Is there a way to get the changelogs for all bundled gems between the version set in Gemfile.lock and latest on rubygems? It seems like something that would be a pretty widespread need, but I haven't found any solution...

That is a really good idea. I haven't heard of any tool that does that currently, so I created a script that does it.
It is very hacky/alpha, but it is still quite effective (more than I thought it would be when I started writing it). So use at your own risk. Hopefully though a feature like this (but much improved) could eventually make it into bundler.
To use, first commit all changes. Then run the script in the base directory (where your Gemfile.lock file is.
It will firstly query all gems in use to find their changelogs. It will then run bundle update. It will then query all gems, find the new ones and diff the changelogs. Finally it will run git checkout Gemfile.lock and bundle install to revert any changes caused by the bundle update command. The results will be saved in a file bundler_gem_diff.txt.
#!/usr/bin/env ruby
def changelog_for_gem(gem)
changelogs = `bundle exec gem contents #{gem}`.lines.grep(/history|changelog/i)
if changelogs.empty?
puts "No changelog found for gem #{gem}"
return nil
end
changelogs.first.chomp
end
class GemDetails
attr_accessor :name, :version, :changelog_file
def initialize(name, version)
#name, #version = name, version
end
def self.from_str(str)
match = str.match(/(\S+) \((.*)\)/)
result = self.new(match[1], Gem::Version.new(match[2]))
result.changelog_file = changelog_for_gem(result.name)
puts "process gem #{result.name}"
result
end
end
def bundler_gems
gem_list = `bundle list`.lines.drop(1).map do |line|
line.gsub(/\s*\* /, '')
end
gem_list.map { |gem_str| GemDetails.from_str(gem_str) }
end
def bundler_update
system("bundle update")
end
def bundler_install
system("bundle install")
end
def updated_gems(initial_gems, new_gems)
result = []
initial_gems.each do |gem|
updated_gem = new_gems.find { |new_gem| gem.name == new_gem.name && gem.version < new_gem.version }
result << [gem, updated_gem] if updated_gem
end
result
end
def changelog_diff
initial_gems = bundler_gems
puts "Updating bundle"
system('bundle update')
new_gems = bundler_gems
File.open('bundler_gem_diff.txt', 'w') do |f|
updated_gems(initial_gems, new_gems).each do |gem_pair|
f.puts "###### #{gem_pair[0].name} #{gem_pair[0].version} -> #{gem_pair[1].version} ######"
if gem_pair[0].changelog_file && gem_pair[1].changelog_file
puts "diff #{gem_pair[0].changelog_file} #{gem_pair[1].changelog_file}"
f.puts `diff #{gem_pair[0].changelog_file} #{gem_pair[1].changelog_file}`
else
f.puts "Missing changelogs"
end
end
end
system('git checkout Gemfile.lock')
system('bundle install')
nil
end
changelog_diff
Here is the output of me running it on my octopress blog installation (rather long, mainly because kramdown uses git log as their changelog):
###### RedCloth 4.2.8 -> 4.2.9 ######
0a1,5
> == 4.2.9 / November 25, 2011
>
> * Fix RbConfig / Config warning in Ruby 1.9.3. [Steve Purcell, Robert Gleeson, and unclaimedbaggage]
> * Use RSTRING_NOT_MODIFIED header for Rubinius [Dirkjan Bussink]
>
###### activesupport 3.2.0 -> 3.2.2 ######
0a1,12
> ## Rails 3.2.1 (January 26, 2012) ##
>
> * Documentation fixes and improvements.
>
> * Update time zone offset information. *Ravil Bayramgalin*
>
> * The deprecated `ActiveSupport::Base64.decode64` calls `::Base64.decode64`
> now. *Jonathan Viney*
>
> * Fixes uninitialized constant `ActiveSupport::TaggedLogging::ERROR`. *kennyj*
>
>
###### chunky_png 1.2.1 -> 1.2.5 ######
Missing changelogs
###### compass 0.11.5 -> 0.11.7 ######
Missing changelogs
###### directory_watcher 1.4.0 -> 1.4.1 ######
0a1,5
> == 1.4.1 / 2011-08-29
>
> Minor Enhancements
> - support for latest 'cool.io'
>
###### fssm 0.2.7 -> 0.2.8.1 ######
Missing changelogs
###### haml 3.1.2 -> 3.1.4 ######
6c6,119
< ## 3.1.0 (Unreleased)
---
> ## 3.1.11
>
> * Allow control directives (such as `#if`) to be nested beneath properties.
> * Allow property names to begin with a hyphen followed by interpolation (e.g. `-#{...}`).
> * Fix a parsing error with interpolation in comma-separated lists.
> * Make `--cache-store` with with `--update`.
> * Properly report `ArgumentError`s that occur within user-defined functions.
> * Don't crash on JRuby if the underlying Java doesn't support every Unicode encoding.
> * Add new `updated_stylesheet` callback, which is run after each stylesheet has
> been successfully compiled. Thanks to [Christian Peters](https://github.com/ChristianPeters).
> * Allow absolute paths to be used in an importer with a different root.
> * Don't destructively modify the options when running `Sass::Plugin.force_update`.
>
> ### Deprecations -- Must Read!
>
> * The `updating_stylesheet` is deprecated and will be removed in a
> future release. Use the new `updated_stylesheet` callback instead.
>
> ## 3.1.10
>
> * Fix another aspect of the 3.1.8 regression relating to `+`.
>
> ## 3.1.9
>
> * Fix a regression in 3.1.8 that broke the `+` combinator in selectors.
>
> * Deprecate the loud-comment flag when used with silent comments (e.g. `//!`).
> Using it with multi-line comments (e.g. `/*!`) still works.
>
> ## 3.1.8
>
> * Deprecate parent selectors followed immediately by identifiers (e.g. `&foo`).
> This should never have worked, since it violates the rule
> of `&` only being usable where an element selector would.
>
> * Add a `--force` option to the `sass` executable which makes `--update`
> always compile all stylesheets, even if the CSS is newer.
>
> * Disallow semicolons at the end of `#import` directives in the indented syntax.
>
> * Don't error out when being used as a library without requiring `fileutil`.
>
> * Don't crash when Compass-style sprite imports are used with `StalenessChecker`
> (thanks to [Matthias Bauer](https://github.com/moeffju)).
>
> * The numeric precision of numbers in Sass can now be set using the
> `--precision` option to the command line. Additionally, the default
> number of digits of precision in Sass output can now be
> changed by setting `Sass::Script::Number.precision` to an integer
> (defaults to 3). Since this value can now be changed, the `PRECISION`
> constant in `Sass::Script::Number` has been deprecated. In the unlikely
> event that you were using it in your code, you should now use
> `Sass::Script::Number.precision_factor` instead.
>
> * Don't crash when running `sass-convert` with selectors with two commas in a row.
>
> * Explicitly require Ruby >= 1.8.7 (thanks [Eric Mason](https://github.com/ericmason)).
>
> * Properly validate the nesting of elements in imported stylesheets.
>
> * Properly compile files in parent directories with `--watch` and `--update`.
>
> * Properly null out options in mixin definitions before caching them. This fixes
> a caching bug that has been plaguing some Rails 3.1 users.
>
> ## 3.1.7
>
> * Don't crash when doing certain operations with `#function`s.
>
> ## 3.1.6
>
> * The option `:trace_selectors` can now be used to emit a full trace
> before each selector. This can be helpful for in-browser debugging of
> stylesheet imports and mixin includes. This option supersedes the
> `:line_comments` option and is superseded by the `:debug_info`
> option.
>
> * Fix a bug where long `#if`/`#else` chains would cause exponential slowdown
> under some circumstances.
>
> ## 3.1.5
>
> * Updated the vendored FSSM version, which will avoid segfaults on OS
> X Lion when using `--watch`.
>
> ## 3.1.4
>
> * Sass no longer unnecessarily caches the sass options hash.
> This allows objects that cannot be marshaled to be placed into the
> options hash.
>
> ## 3.1.3
>
> * Sass now logs message thru a logger object which can be changed to
> provide integration with other frameworks' logging infrastructure.
>
>
> ## 3.1.2
>
> * Fix some issues that were breaking Sass when running within Rubinius.
> * Fix some issues that were affecting Rails 3.1 integration.
> * New function `zip` allows several lists to be combined into one
> list of lists. For example:
> `zip(1px 1px 3px, solid dashed solid, red green blue)` becomes
> `1px solid red, 1px dashed green, 3px solid blue`
> * New function `index` returns the list index of a value
> within a list. For example: `index(1px solid red, solid)`
> returns `2`. When the value is not found `false` is returned.
>
> ## 3.1.1
>
> * Make sure `Sass::Plugin` is loaded at the correct time in Rails 3.
>
> ## 3.1.0
138c251
< However, you can now do more with them using the new {file:Sass/Script/Functions.html#list-functions list functions}:
---
> However, you can now do more with them using the new [list functions](Sass/Script/Functions.html#list-functions):
###### jekyll 0.11.0 -> 0.11.2 ######
0a1,9
> == 0.11.2 / 2011-12-27
> * Bug Fixes
> * Fix gemspec
>
> == 0.11.1 / 2011-12-27
> * Bug Fixes
> * Fix extra blank line in highlight blocks (#409)
> * Update dependencies
>
###### kramdown 0.13.3 -> 0.13.5 ######
0a1,429
> commit ce785c7cc29372cb7c5c47647535b1943348aff6
> Author: Thomas Leitner <t_leitner#gmx.at>
> Date: Sun Feb 19 15:34:37 2012 +0100
>
> Updated release notes, version number and benchmark graphs
>
> doc/img/graph-ruby-1.8.5-231.png
> doc/img/graph-ruby-1.8.6-399.png
> doc/img/graph-ruby-1.8.7-249.png
> doc/img/graph-ruby-1.8.7-302.png
> doc/img/graph-ruby-1.9.2p136-136.png
> doc/img/graph-ruby-1.9.3p0-0.png
> doc/img/graph-ruby-1.9.3p125-125.png
> doc/news/release_0_13_5.page
> doc/sidebar.template
> doc/tests.page
> lib/kramdown/version.rb
>
> commit 90f0583232e814334b2bb8c7fa3228b2f4e30f81
> Author: Thomas Leitner <t_leitner#gmx.at>
> Date: Sun Feb 19 10:14:58 2012 +0100
>
> Bug fix: Empty id attributes are handled better now
>
> * The HTML converter does not output empty id attributes anymore.
> * The kramdown converter now always converts empty id attributes
> to IAL form.
>
> lib/kramdown/converter/kramdown.rb
> lib/kramdown/utils/html.rb
> test/testcases/block/04_header/with_auto_ids.html
> test/testcases/block/04_header/with_auto_ids.text
>
> commit d956b98fdf449af145bc8b52539abdb99a062cda
> Author: Thomas Leitner <t_leitner#gmx.at>
> Date: Sun Feb 19 09:53:03 2012 +0100
>
> Small typo fix in quickref
>
> doc/quickref.page
>
> commit 4df3c5aaad763cc9d8fd4b129a5cdd245ed5a7f7
> Author: Thomas Leitner <t_leitner#gmx.at>
> Date: Sun Feb 19 09:38:29 2012 +0100
>
> Updated homepage contents
>
> doc/documentation.page
> doc/index.page
> doc/installation.page
> doc/news.page
> doc/syntax.page
> doc/tests.page
>
> commit c0bcd765e131b671661af70274ce632111ad89bb
> Author: Thomas Leitner <t_leitner#gmx.at>
> Date: Sun Feb 19 09:37:15 2012 +0100
>
> Updated website design
>
> doc/bg.png
> doc/default.scss.css
> doc/default.template
> doc/design.scss.css
> doc/sidebar.template
>
> commit d530421d0a3794010cd0c022decb097158e90732
> Author: Thomas Leitner <t_leitner#gmx.at>
> Date: Sun Feb 19 09:35:51 2012 +0100
>
> Disabling line numbers in code listings on homepage
>
> Rakefile
>
> commit a47a5c49e35f9ccb220dca7b5b8ac7d585026ef6
> Author: Thomas Leitner <t_leitner#gmx.at>
> Date: Sun Feb 19 09:25:01 2012 +0100
>
> Updated copyright notice updating code
>
> Rakefile
>
> commit 21ce307a6766d131bf3a61de1142d88b0bd0a8be
> Author: Thomas Leitner <t_leitner#gmx.at>
> Date: Sun Feb 19 09:24:22 2012 +0100
>
> Updated copyright notices
>
> bin/kramdown
> lib/kramdown.rb
> lib/kramdown/compatibility.rb
> lib/kramdown/converter.rb
> lib/kramdown/converter/base.rb
> lib/kramdown/converter/html.rb
> lib/kramdown/converter/kramdown.rb
> lib/kramdown/converter/latex.rb
> lib/kramdown/converter/toc.rb
> lib/kramdown/document.rb
> lib/kramdown/element.rb
> lib/kramdown/error.rb
> lib/kramdown/options.rb
> lib/kramdown/parser.rb
> lib/kramdown/parser/base.rb
> lib/kramdown/parser/html.rb
> lib/kramdown/parser/kramdown.rb
> lib/kramdown/parser/kramdown/abbreviation.rb
> lib/kramdown/parser/kramdown/autolink.rb
> lib/kramdown/parser/kramdown/blank_line.rb
> lib/kramdown/parser/kramdown/block_boundary.rb
> lib/kramdown/parser/kramdown/blockquote.rb
> lib/kramdown/parser/kramdown/codeblock.rb
> lib/kramdown/parser/kramdown/codespan.rb
> lib/kramdown/parser/kramdown/emphasis.rb
> lib/kramdown/parser/kramdown/eob.rb
> lib/kramdown/parser/kramdown/escaped_chars.rb
> lib/kramdown/parser/kramdown/extensions.rb
> lib/kramdown/parser/kramdown/footnote.rb
> lib/kramdown/parser/kramdown/header.rb
> lib/kramdown/parser/kramdown/horizontal_rule.rb
> lib/kramdown/parser/kramdown/html.rb
> lib/kramdown/parser/kramdown/html_entity.rb
> lib/kramdown/parser/kramdown/line_break.rb
> lib/kramdown/parser/kramdown/link.rb
> lib/kramdown/parser/kramdown/list.rb
> lib/kramdown/parser/kramdown/math.rb
> lib/kramdown/parser/kramdown/paragraph.rb
> lib/kramdown/parser/kramdown/smart_quotes.rb
> lib/kramdown/parser/kramdown/table.rb
> lib/kramdown/parser/kramdown/typographic_symbol.rb
> lib/kramdown/parser/markdown.rb
> lib/kramdown/utils.rb
> lib/kramdown/utils/entities.rb
> lib/kramdown/utils/html.rb
> lib/kramdown/utils/ordered_hash.rb
> lib/kramdown/version.rb
> test/run_tests.rb
> test/test_files.rb
>
> commit 512b00a6d050f506c43b824861f7bbf459862a19
> Author: Thomas Leitner <t_leitner#gmx.at>
> Date: Sun Feb 19 09:12:38 2012 +0100
>
> Fixed bug RF#29521: HTML math output not always XHTML compatible
>
> The characters < and & are not allowed in a script tag in XHTML.
> So since the HTML converter uses script tags for math elements,
> whenever these characters appear in a value the value is wrapped
> in a CDATA section to make the output XHTML compatible.
>
> lib/kramdown/converter/html.rb
> lib/kramdown/parser/html.rb
> test/testcases/block/15_math/normal.html
>
> commit 8f111947dbcea064a997662d65cd53e6e39c8c40
> Author: Thomas Leitner <t_leitner#gmx.at>
> Date: Sat Feb 18 20:06:42 2012 +0100
>
> Fixed bug RF#29520: inline math statements interpreted as math blocks
>
> If a paragraph started and ended with an inline math statement, it was
> parsed as math block. This is fixed now.
>
> Also, it is now better explained in the kramdown syntax documentation
> how special cases are handled.
>
> doc/syntax.page
> lib/kramdown/converter/kramdown.rb
> lib/kramdown/parser/kramdown/math.rb
> test/testcases/span/math/normal.html
> test/testcases/span/math/normal.text
>
> commit f5b585513508704563cc0bb343869f91520ff853
> Author: Thomas Leitner <t_leitner#gmx.at>
> Date: Sat Feb 18 11:11:45 2012 +0100
>
> Fixed problem with missing methods/constants on older Ruby 1.8.6 versions
>
> lib/kramdown/compatibility.rb
>
> commit 394d5de7cad34ed5e0bdcfc0fcc7a1a28b3f0229
> Author: Thomas Leitner <t_leitner#gmx.at>
> Date: Tue Jan 31 17:32:09 2012 +0100
>
> Moved camelize method from Document#method_to_class_name to Utils.camelize
>
> lib/kramdown/document.rb
> lib/kramdown/utils.rb
>
> commit 7adc3bda20a8290f0af588da7816eec591d4ecaa
> Author: tomykaira <tomykaira#gmail.com>
> Date: Fri Dec 30 18:08:43 2011 +0900
>
> Fix Document#method_missing to accept snake_cased class name
>
> When doc.to_my_converter is called, it should use MyConverter, but used My_converter
>
> lib/kramdown/document.rb
>
> commit a6b181a9baf2eeeabeb55a1a9d5b26791fecad08
> Author: Thomas Leitner <t_leitner#gmx.at>
> Date: Sat Jan 21 12:18:22 2012 +0100
>
> HTML attributes names are now converted to lower case
>
> doc/syntax.page
> lib/kramdown/parser/html.rb
> lib/kramdown/parser/kramdown/html.rb
> test/testcases/block/09_html/html5_attributes.html
> test/testcases/block/09_html/simple.text
> test/testcases/span/05_html/normal.text
>
> commit 170ca36a60d823803f575bf5632d9e9d1e74f7db
> Author: Thomas Leitner <t_leitner#gmx.at>
> Date: Sat Jan 21 12:05:31 2012 +0100
>
> HTML attributes without values are now supported
>
> This means that one can write boolean HTML attributes now without
> assigning an empty value.
>
> doc/syntax.page
> lib/kramdown/converter/kramdown.rb
> lib/kramdown/parser/html.rb
> lib/kramdown/parser/kramdown/html.rb
> test/testcases/block/09_html/html5_attributes.html
> test/testcases/block/09_html/html5_attributes.text
> test/testcases/span/05_html/normal.html
> test/testcases/span/05_html/normal.text
>
> commit 2c1d7c5e66bfe42052dc43374e123a48e7cb2ad6
> Author: Thomas Leitner <t_leitner#gmx.at>
> Date: Fri Dec 16 18:40:15 2011 +0100
>
> Fix to work with newer RDoc versions
>
> Rakefile
>
> commit 2a93cbc0e1bbe705159c645ae617538bad125394
> Author: Thomas Leitner <t_leitner#gmx.at>
> Date: Fri Dec 16 18:26:32 2011 +0100
>
> Updated release notes, version number and benchmark graphs
>
> doc/img/graph-jruby-1.6.0-330.png
> doc/img/graph-ruby-1.8.5-231.png
> doc/img/graph-ruby-1.8.6-399.png
> doc/img/graph-ruby-1.8.7-249.png
> doc/img/graph-ruby-1.8.7-302.png
> doc/img/graph-ruby-1.9.2p136-136.png
> doc/img/graph-ruby-1.9.3p0-0.png
> doc/index.page
> doc/news/release_0_13_4.page
> doc/news/release_0_X_X.page
> doc/tests.page
> lib/kramdown/version.rb
>
> commit dbbabbc5b61b67ed0d22c1aa4a6b17608ed771e6
> Author: Thomas Leitner <t_leitner#gmx.at>
> Date: Tue Dec 13 17:25:53 2011 +0100
>
> Fixed test case to work with coderay 1.0
>
> test/testcases/block/12_extension/options3.html
>
> commit 6828b83435c30676185d675dc7b6054846606df0
> Author: Thomas Leitner <t_leitner#gmx.at>
> Date: Tue Dec 13 17:25:31 2011 +0100
>
> Updated Rakefile and setup.rb
>
> * Use gem package task supplied by rubygems
> * Add coderay as development dependency
> * Fix warning in setup.rb because of Config::CONFIG use
>
> Rakefile
> setup.rb
>
> commit bb7e82ce3f5213830fa6387e6c790a75b919dd38
> Author: Thomas Leitner <t_leitner#gmx.at>
> Date: Tue Dec 13 17:23:51 2011 +0100
>
> Fixed problem with different Array#delete_if internals on Ruby 1.9.3
>
> See http://redmine.ruby-lang.org/issues/5752
>
> lib/kramdown/parser/html.rb
>
> commit cd253a5e9d4dcfe55c12caf9257e4f1fb6113943
> Author: Thomas Leitner <t_leitner#gmx.at>
> Date: Sun Dec 11 15:40:35 2011 +0100
>
> Fixed warnings on Ruby 1.9.3
>
> lib/kramdown/converter/html.rb
> lib/kramdown/parser/html.rb
> lib/kramdown/parser/kramdown.rb
> lib/kramdown/parser/kramdown/header.rb
> lib/kramdown/parser/kramdown/html.rb
> lib/kramdown/parser/kramdown/smart_quotes.rb
> test/test_files.rb
>
> commit e71cc414755e51c3336551e3f011f2f39ea601ee
> Author: Thomas Leitner <t_leitner#gmx.at>
> Date: Sun Dec 11 15:32:52 2011 +0100
>
> Fixed test cases to correctly behave with new footnote rel output
>
> test/testcases/block/12_extension/options.html
> test/testcases/block/12_extension/options2.html
> test/testcases/block/14_table/table_with_footnote.html
>
> commit b3b2ebc233160f4f649f0d0aed070d7540781c4f
> Merge: a58ba5b ef70112
> Author: gettalong <t_leitner#gmx.at>
> Date: Sun Dec 11 06:26:18 2011 -0800
>
> Merge pull request #4 from joefiorini/master
>
> HTML5 Compatibility for Footnotes
>
> Replacing the rev='footnote' with rel='reference'.
>
> commit a58ba5b364081e896e2a49cc06db5aec643f8bb4
> Author: Thomas Leitner <t_leitner#gmx.at>
> Date: Sun Dec 11 12:38:37 2011 +0100
>
> Fixed bug RF#29426: Content of <style> tags is treated as raw text now
>
> lib/kramdown/parser/html.rb
> test/testcases/block/09_html/parse_as_raw.html
> test/testcases/block/09_html/parse_as_raw.htmlinput
> test/testcases/block/09_html/parse_as_raw.text
>
> commit 14800a606565ff7d1759fea9815be1f9eff9b633
> Author: Thomas Leitner <t_leitner#gmx.at>
> Date: Sun Dec 11 12:27:52 2011 +0100
>
> Fixed bug RF#29350: Parsing of HTML tags with mismatched case now works
>
> lib/kramdown/parser/html.rb
> lib/kramdown/parser/kramdown/html.rb
> test/test_files.rb
> test/testcases/block/09_html/parse_as_raw.htmlinput
> test/testcases/block/09_html/parse_as_raw.text
> test/testcases/block/09_html/parse_as_span.htmlinput
> test/testcases/block/09_html/parse_as_span.text
> test/testcases/block/09_html/parse_block_html.text
> test/testcases/span/05_html/normal.html
> test/testcases/span/05_html/normal.text
>
> commit b585efb3086c3462a573b57d060dfd203070bcc1
> Merge: 64a4e7e 3a72b72
> Author: Thomas Leitner <t_leitner#gmx.at>
> Date: Sat Sep 24 14:41:34 2011 +0200
>
> Merge branch 'master' of github.com:gettalong/kramdown
>
> commit 3a72b72564d4f788fa1423dd8eb1089e355b0a4c
> Merge: e6c1b12 df5e705
> Author: gettalong <t_leitner#gmx.at>
> Date: Sat Sep 24 05:28:10 2011 -0700
>
> Merge pull request #5 from jacius/span
>
> span tags must have a separate closing tag to be valid HTML.
>
> commit e6c1b12fa45c7b3284c4b41247ade9ff4b4379a4
> Merge: 9188906 1270445
> Author: gettalong <t_leitner#gmx.at>
> Date: Sat Sep 24 05:24:32 2011 -0700
>
> Merge pull request #6 from postmodern/patch-1
>
> Fixed a typo. --output not --ouput
>
> commit 12704459192bdcbf9b02d1d90483c8d835f3e9f1
> Author: Postmodern <postmodern.mod3#gmail.com>
> Date: Thu Aug 4 18:57:59 2011 -0700
>
> Fixed a typo. --output not --ouput
>
> bin/kramdown
>
> commit df5e705ef41ca2c47b5b0d54509cdfb80c2d5f4d
> Author: John Croisant <jacius#gmail.com>
> Date: Sun Jul 10 23:52:35 2011 -0500
>
> span tags must have a separate closing tag to be valid HTML.
>
> "<span />" is invalid markup in HTML 4 and 5, but valid in XHTML.
> "<span></span>" is valid in all three, so it should be used instead.
>
> lib/kramdown/converter/html.rb
> test/testcases/span/05_html/normal.html
> test/testcases/span/05_html/normal.text
>
> commit ef7011229e7d93c31c22468e5cfd797051ac280c
> Author: Joe Fiorini <joe#faithfulgeek.org>
> Date: Wed May 11 02:08:38 2011 -0400
>
> Replace rev attribute for HTML5 compatibility
>
> lib/kramdown/converter/html.rb
> test/testcases/span/04_footnote/footnote_nr.html
> test/testcases/span/04_footnote/markers.html
>
> commit 64a4e7e61da0d720ffd702fd6c45cefafc568d24
> Author: Thomas Leitner <t_leitner#gmx.at>
> Date: Tue May 10 20:47:16 2011 +0200
>
> Added a TOC converter
>
> The TOC converter can be used to get an element tree containing the
> table of contents. Since it uses a custom element type the resulting
> tree cannot be fed directly to any converter.
>
> doc/news/release_0_X_X.page
> lib/kramdown/converter.rb
> lib/kramdown/converter/toc.rb
>
> commit 5ed13d24d9375771e3a0bd0d4fbe779d0043563b
> Author: Thomas Leitner <t_leitner#gmx.at>
> Date: Tue May 10 07:42:23 2011 +0200
>
> Extracted Element class into own file and made some small doc fixes
>
> lib/kramdown/document.rb
> lib/kramdown/element.rb
>
###### liquid 2.2.2 -> 2.3.0 ######
1c1,16
< 2.2.1 / 2010-08-23
---
> # Liquid Version History
>
> ## 2.3.0
>
> * Several speed/memory improvements
> * Numerous bug fixes
> * Added support for MRI 1.9, Rubinius, and JRuby
> * Added support for integer drop parameters
> * Added epoch support to `date` filter
> * New `raw` tag that suppresses parsing
> * Added `else` option to `for` tag
> * New `increment` tag
> * New `split` filter
>
>
> ## 2.2.1 / 2010-08-23
5c20,21
< 2.2.0 / 2010-08-22
---
>
> ## 2.2.0 / 2010-08-22
10c26,27
< 1.9.0 / 2008-03-04
---
>
> ## 1.9.0 / 2008-03-04
15,17d31
< Before 1.9.0
<
< * Added If with or / and expressions
19c33
< * Implemented .to_liquid for all objects which can be passed to liquid like Strings Arrays Hashes Numerics and Booleans. To export new objects to liquid just implement .to_liquid on them and return objects which themselves have .to_liquid methods.
---
> ## Before 1.9.0
20a35,36
> * Added If with or / and expressions
> * Implemented .to_liquid for all objects which can be passed to liquid like Strings Arrays Hashes Numerics and Booleans. To export new objects to liquid just implement .to_liquid on them and return objects which themselves have .to_liquid methods.
22d37
<
24d38
<
26d39
<
28d40
<
30d41
<
32d42
<
34d43
<
36d44
<
38,42c46,47
<
< {{ 'Typo' | link_to: 'http://typo.leetsoft.com', 'Typo - a modern weblog engine' }}
<
<
< * Added Liquid::Drop. A base class which you can use for exporting proxy objects to liquid which can acquire more data when used in liquid. [Tobias Luetke]
---
> {{ 'Typo' | link_to: 'http://typo.leetsoft.com', 'Typo - a modern weblog engine' }}
> * Added Liquid::Drop. A base class which you can use for exporting proxy objects to liquid which can acquire more data when used in liquid. [Tobias Luetke]
48c53
< end
---
> end
51,52d55
<
<
###### multi_json 1.0.4 -> 1.1.0 ######
Missing changelogs
###### rack 1.3.2 -> 1.4.1 ######
Missing changelogs
###### rake 0.9.2 -> 0.9.2.2 ######
Missing changelogs
###### rb-fsevent 0.4.3.1 -> 0.9.0 ######
Missing changelogs
###### sass 3.1.5 -> 3.1.15 ######
Missing changelogs
###### stringex 1.3.0 -> 1.3.2 ######
Missing changelogs
###### tilt 1.3.2 -> 1.3.3 ######
Missing changelogs

This doesn't exactly answer your question, but it may be helpful to know how Gem version numbers work. According to the specification, the first of the three numbers in a gem version should tell you whether the gem is guaranteed to be backwards compatible.
For example, if you start your project with some gem version 3.1.0, it should be compatible with any 3.x.x version of the gem. According to the spec, "A category 3 change (incompatible) will increment the major version number and reset the minor and build numbers"
See here for details: http://docs.rubygems.org/read/chapter/7
You might also find this post about freezing gem versions to test out new ones helpful: https://stackoverflow.com/a/2147227/834512

Is the new bundle outdated command suiting your needs?
http://patshaughnessy.net/2011/11/5/besides-being-faster-what-else-is-new-in-bundler-1-1
The most useful and important new command introduced in Bundler 1.1 is bundle outdated, which will identify the outdated gems in your bundle – these are the gems for which there is a newer version available

There is also a nice webservice which is free for public open source projects (github service hook available): http://Gemnasium.com
It checks which gems are up to date and serves an overview plus email notifications about what has changed between versions (github tag compare diff).

Related

Where to store custom module unit test in a project used by Ansible Tower?

I finished coding three Ansible custom module that are stored in a Git Repository that is used by a Ansible Tower Project.
The directory tree looks like that :
> files/
> - a.txt
> - b.txt
> library/
> - common_code.py
> module_utils/
> - custom_module1.py
> - custom_module2.py
> templates/
> - c.txt.j2
> - d.txt.j2
> playbook.yml
I couldn't find anywhere where I should store the custom module unit tests, or even the module_utils unit test ? Right next to the py files ? What is the best pratice ?
Thank you.
I went on a more traditional way by adding a tests directory and multiple __init__.py files :
> files/
> - a.txt
> - b.txt
> library/
> - __init__.py
> - common_code.py
> module_utils/
> - __init__.py
> - custom_module1.py
> - custom_module2.py
> templates/
> - c.txt.j2
> - d.txt.j2
> tests/
> - __init__.py
> - test_custom_module1.py
> - test_custom_module2.py
> - test_common_code.py
> playbook.yml
and imported the other files to test in the units tests like this :
from module_utils.custom_module1 import *
from module_utils.custom_module2 import *
from library.common_code import *
I had to modify the import in the custom modules :
try:
from ansible.module_utils.common_code import util
except ImportError: # We except an ImportError in unit tests so we import it differently :D
from module_utils.common_code import util
And added some specific Exception to handle the exit_json and fail_json as stated here like this :
self.module = MagicMock()
self.module.exit_json.side_effect = AnsibleExitJson(Exception)
self.module.fail_json.side_effect = AnsibleFailJson(Exception)
and later in a unit test function
self.module.fail_json.assert_called()
self.module.exit_json.assert_not_called()

Can't Find Gradle Workers Module?

I'm trying to set up a fresh gradle environment for our build server, but I seem to be missing some module or jar that's meant to come with an install. I've done a clean install three times, so what would cause the module "gradle-workers" to not be found?
Environment: Windows Server 2012 R2
> 12:57:27.432 org.gradle.internal.service.ServiceCreationException:
> Could not create service of type ClassLoaderRegistry using
> GlobalScopeServices.createClassLoaderRegistry(). 12:57:27.433 at
> org.gradle.internal.service.DefaultServiceRegistry$FactoryMethodService.invokeMethod(DefaultServiceRegistry.java:807)
> 12:57:27.433 at
> org.gradle.internal.service.DefaultServiceRegistry$FactoryService.create(DefaultServiceRegistry.java:761)
> 12:57:27.433 at
> org.gradle.internal.service.DefaultServiceRegistry$ManagedObjectProvider.getInstance(DefaultServiceRegistry.java:598)
> 12:57:27.433 at
> org.gradle.internal.service.DefaultServiceRegistry$SingletonService.get(DefaultServiceRegistry.java:643)
> 12:57:27.434 at
> org.gradle.internal.service.DefaultServiceRegistry.applyConfigureMethod(DefaultServiceRegistry.java:253)
> 12:57:27.434 at
> org.gradle.internal.service.DefaultServiceRegistry.findProviderMethods(DefaultServiceRegistry.java:214)
> 12:57:27.434 at
> org.gradle.internal.service.DefaultServiceRegistry.addProvider(DefaultServiceRegistry.java:352)
> 12:57:27.435 at
> org.gradle.internal.service.ServiceRegistryBuilder.build(ServiceRegistryBuilder.java:52)
> 12:57:27.435 at
> org.gradle.launcher.cli.BuildActionsFactory.createGlobalClientServices(BuildActionsFactory.java:148)
> 12:57:27.435 at
> org.gradle.launcher.cli.BuildActionsFactory.runBuildWithDaemon(BuildActionsFactory.java:108)
> 12:57:27.435 at
> org.gradle.launcher.cli.BuildActionsFactory.createAction(BuildActionsFactory.java:83)
> 12:57:27.435 at
> org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.createAction(CommandLineActionFactory.java:249)
> 12:57:27.437 at
> org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:239)
> 12:57:27.437 at
> org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:217)
> 12:57:27.437 at
> org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:33)
> 12:57:27.437 at
> org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:24)
> 12:57:27.438 at
> org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:33)
> 12:57:27.438 at
> org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:22)
> 12:57:27.438 at
> org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:210)
> 12:57:27.438 at
> org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:174)
> 12:57:27.439 at org.gradle.launcher.Main.doAction(Main.java:33)
> 12:57:27.439 at
> org.gradle.launcher.bootstrap.EntryPoint.run(EntryPoint.java:45)
> 12:57:27.439 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method) 12:57:27.439 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 12:57:27.440 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 12:57:27.440 at java.lang.reflect.Method.invoke(Method.java:498)
> 12:57:27.453 at
> org.gradle.launcher.bootstrap.ProcessBootstrap.runNoExit(ProcessBootstrap.java:60)
> 12:57:27.453 at
> org.gradle.launcher.bootstrap.ProcessBootstrap.run(ProcessBootstrap.java:37)
> 12:57:27.454 at
> org.gradle.launcher.GradleMain.main(GradleMain.java:23) 12:57:27.454
> Caused by: org.gradle.api.internal.classpath.UnknownModuleException:
> Cannot locate JAR for module 'gradle-workers' in distribution
> directory 'D:\apps\gradle-3.4.1'.

Open directory and create new file

I have this statement that creates nested directories:
require 'fileutils'
FileUtils::mkdir_p ''+project_name+'/new_folder'
I want to add for example a new file called README.md inside new_folder. Is there a way to accomplish that with Ruby?
You just need to do
path_name = "#{project_name}/new_folder"
FileUtils::mkdir_p path_name
FileUtils::touch("#{path_name}/README.md")
This will create an empty file named README.md inside your project_name/new_folder directory.
You can do it like this
2.1.2 :015 > project_name = "foo"
=> "foo"
2.1.2 :016 > fld = FileUtils::mkdir_p "#{project_name}/new_folder"
=> ["foo/new_folder"]
2.1.2 :017 > FileUtils.touch "#{fld[0]}/README.md" if fld
=> ["foo/new_folder/README.md"]
[retgoat#iMac-Roman ~/foo/new_folder]$ ls -la
total 0
drwxr-xr-x 3 retgoat staff 102 17 май 17:44 .
drwxr-xr-x# 119 retgoat staff 4046 17 май 17:45 ..
-rw-r--r-- 1 retgoat staff 0 17 май 17:44 README.md
Please note I didn't test your code, but if it works for you, following example will create a file README.md inside new_folder
For building filespecs, it's better to use the File.join method. This is higher level, handles any extra or missing directory separators, and uses the right file separator for the OS on which it's running.
For example:
2.3.0 :006 > project_name = 'my_project'
=> "my_project"
2.3.0 :009 > filespec = File.join(project_name, 'new_folder', 'README.md')
=> "my_project/new_folder/README.md"
When the slashes are provided before and after 'new_folder', it still works:
2.3.0 :010 > filespec = File.join(project_name, '/new_folder/', 'README.md')
=> "my_project/new_folder/README.md"

Time Zone difference between Time.now and 1.day.ago

My app time_zone is set to UTC(default, I didn't change it in config/application.rb).
This is what I see in rails console
1.9.3p194 :004 > Time.now
=> 2014-03-20 14:45:23 -0500
1.9.3p194 :005 > 1.day.ago
=> Wed, 19 Mar 2014 19:45:48 UTC +00:00
Why do I get the time in central when I do Time.now? It should return time in UTC like 1.day.ago
Time.now uses the locale of the machine it is running on. For consistency, you can do Time.now.utc to force UTC:
1.9.3-p484 :001 > Time.now
2014-03-20 16:14:23 -0400
1.9.3-p484 :002 > Time.now.utc
2014-03-20 20:14:26 UTC
use Time.zone.now - it will take into account your application's time zone:
project with default TZ:
2.0.0-p353 :001 > Time.now
=> 2014-04-01 23:12:06 +0300
2.0.0-p353 :002 > Time.zone.now
=> Tue, 01 Apr 2014 20:12:11 UTC +00:00
2.0.0-p353 :003 >
project with custom TZ:
2.1.0 :001 > Time.now
=> 2014-04-01 23:14:23 +0300
2.1.0 :002 > Time.zone.now
=> Tue, 01 Apr 2014 22:14:27 CEST +02:00
2.1.0 :003 >
Here you'll find more useful info about working with timezones (especially take a look at DOs and DONTs section):
http://www.elabs.se/blog/36-working-with-time-zones-in-ruby-on-rails

Pipe Broken exception every time when I run Mahout samples at EC2 server

I've installed mahout at bitnami AMI ami-02fb006b, (as well as several other ami's, otherwise I won't be asking the question)
according to instructions provided here
and here:
I'm always getting stuck when trying to run ./examples/bin/build-reuters.sh
Here's the output of the command:
> Please select a number to choose the corresponding clustering
> algorithm
> 1. kmeans clustering
> 2. lda clustering Enter your choice : 1 ok. You chose 1 and we'll use
> kmeans Clustering Downloading Reuters-21578 % Total % Received %
> Xferd Average Speed Time Time Time Current
> Dload Upload Total Spent
> Left Speed 100 7959k 100 7959k 0 0 294k 0 0:00:26
> 0:00:26 --:--:-- 305k Extracting... Running on hadoop, using
> HADOOP_HOME=/usr/local/hadoop-0.20.2
> HADOOP_CONF_DIR=/usr/local/hadoop-0.20.2/conf MAHOUT-JOB:
> /usr/local/mahout-0.4/examples/target/mahout-examples-0.6-SNAPSHOT-job.jar
> 11/08/16 20:10:25 WARN driver.MahoutDriver: No
> org.apache.lucene.benchmark.utils.ExtractReuters.props found on
> classpath, will use command-line arguments only
> Deleting all files in mahout-work/reuters-out-tmp
> 11/08/16 20:10:30 INFO driver.MahoutDriver: Program took 4906 ms
> MAHOUT_LOCAL is set, running locally
> CLASSPATH:
> :/usr/local/mahout-0.4/src/conf:/usr/local/hadoop-0.20.2/conf:/usr/lib/jvm/java-6-openjdk//lib/tools.jar:/usr/local/mahout-0.4/mahout-*.jar:/usr/local/mahout-0.4/examples/target/mahout-examples-0.6-SNAPSHOT-job.jar:/usr/local/mahout-0.4/mahout-examples-*-job.jar:/usr/local/mahout-0.4/lib/*.jar:/usr/local/mahout-0.4/examples/target/dependency/antlr-2.7.7.jar:/usr/local/mahout-0.4/examples/target/dependency/antlr-3.2.jar:/usr/local/mahout-0.4/examples/target/dependency/antlr-runtime-3.2.jar:/usr/local/mahout-0.4/examples/target/dependency/avro-1.4.0-cassandra-1.jar:/usr/local/mahout-0.4/examples/target/dependency/bson-2.5.jar:/usr/local/mahout-0.4/examples/target/dependency/cassandra-all-0.8.1.jar:/usr/local/mahout-0.4/examples/target/dependency/cassandra-thrift-0.8.1.jar:/usr/local/mahout-0.4/examples/target/dependency/cglib-nodep-2.2.jar:/usr/local/mahout-0.4/examples/target/dependency/commons-beanutils-1.7.0.jar:/usr/local/mahout-0.4/examples/target/dependency/commons-beanutils-core-1.8.0.jar:/usr/local/mahout-0.4/examples/target/dependency/commons-cli-1.2.jar:/usr/local/mahout-0.4/examples/target/dependency/commons-cli-2.0-mahout.jar:/usr/local/mahout-0.4/examples/target/dependency/commons-codec-1.4.jar:/usr/local/mahout-0.4/examples/target/dependency/commons-collections-3.2.1.jar:/usr/local/mahout-0.4/examples/target/dependency/commons-compress-1.1.jar:/usr/local/mahout-0.4/examples/target/dependency/commons-configuration-1.6.jar:/usr/local/mahout-0.4/examples/target/dependency/commons-dbcp-1.4.jar:/usr/local/mahout-0.4/examples/target/dependency/commons-digester-1.7.jar:/usr/local/mahout-0.4/examples/target/dependency/commons-httpclient-3.0.1.jar:/usr/local/mahout-0.4/examples/target/dependency/commons-lang-2.6.jar:/usr/local/mahout-0.4/examples/target/dependency/commons-logging-1.1.1.jar:/usr/local/mahout-0.4/examples/target/dependency/commons-math-2.2.jar:/usr/local/mahout-0.4/examples/target/dependency/commons-pool-1.5.6.jar:/usr/local/mahout-0.4/examples/target/dependency/concurrentlinkedhashmap-lru-1.1.jar:/usr/local/mahout-0.4/examples/target/dependency/easymock-3.0.jar:/usr/local/mahout-0.4/examples/target/dependency/google-collections-1.0-rc2.jar:/usr/local/mahout-0.4/examples/target/dependency/guava-r09.jar:/usr/local/mahout-0.4/examples/target/dependency/hadoop-core-0.20.203.0.jar:/usr/local/mahout-0.4/examples/target/dependency/hector-core-0.8.0-2.jar:/usr/local/mahout-0.4/examples/target/dependency/high-scale-lib-1.1.2.jar:/usr/local/mahout-0.4/examples/target/dependency/httpclient-4.0.1.jar:/usr/local/mahout-0.4/examples/target/dependency/httpcore-4.0.1.jar:/usr/local/mahout-0.4/examples/target/dependency/jackson-core-asl-1.8.2.jar:/usr/local/mahout-0.4/examples/target/dependency/jackson-mapper-asl-1.8.2.jar:/usr/local/mahout-0.4/examples/target/dependency/jakarta-regexp-1.4.jar:/usr/local/mahout-0.4/examples/target/dependency/jamm-0.2.2.jar:/usr/local/mahout-0.4/examples/target/dependency/jcommon-1.0.12.jar:/usr/local/mahout-0.4/examples/target/dependency/jetty-6.1.22.jar:/usr/local/mahout-0.4/examples/target/dependency/jetty-util-6.1.22.jar:/usr/local/mahout-0.4/examples/target/dependency/jfreechart-1.0.13.jar:/usr/local/mahout-0.4/examples/target/dependency/jline-0.9.94.jar:/usr/local/mahout-0.4/examples/target/dependency/json-simple-1.1.jar:/usr/local/mahout-0.4/examples/target/dependency/jul-to-slf4j-1.6.1.jar:/usr/local/mahout-0.4/examples/target/dependency/junit-4.8.2.jar:/usr/local/mahout-0.4/examples/target/dependency/libthrift-0.6.1.jar:/usr/local/mahout-0.4/examples/target/dependency/log4j-1.2.16.jar:/usr/local/mahout-0.4/examples/target/dependency/lucene-analyzers-3.1.0.jar:/usr/local/mahout-0.4/examples/target/dependency/lucene-benchmark-3.1.0.jar:/usr/local/mahout-0.4/examples/target/dependency/lucene-core-3.1.0.jar:/usr/local/mahout-0.4/examples/target/dependency/lucene-highlighter-3.1.0.jar:/usr/local/mahout-0.4/examples/target/dependency/lucene-memory-3.1.0.jar:/usr/local/mahout-0.4/examples/target/dependency/lucene-queries-3.1.0.jar:/usr/local/mahout-0.4/examples/target/dependency/lucene-xercesImpl-3.1.0.jar:/usr/local/mahout-0.4/examples/target/dependency/mahout-collections-1.0.jar:/usr/local/mahout-0.4/examples/target/dependency/mahout-core-0.6-SNAPSHOT.jar:/usr/local/mahout-0.4/examples/target/dependency/mahout-core-0.6-SNAPSHOT-tests.jar:/usr/local/mahout-0.4/examples/target/dependency/mahout-integration-0.6-SNAPSHOT.jar:/usr/local/mahout-0.4/examples/target/dependency/mahout-math-0.6-SNAPSHOT.jar:/usr/local/mahout-0.4/examples/target/dependency/mahout-math-0.6-SNAPSHOT-tests.jar:/usr/local/mahout-0.4/examples/target/dependency/mongo-java-driver-2.5.jar:/usr/local/mahout-0.4/examples/target/dependency/objenesis-1.2.jar:/usr/local/mahout-0.4/examples/target/dependency/servlet-api-2.5-20081211.jar:/usr/local/mahout-0.4/examples/target/dependency/servlet-api-2.5.jar:/usr/local/mahout-0.4/examples/target/dependency/slf4j-api-1.6.1.jar:/usr/local/mahout-0.4/examples/target/dependency/slf4j-jcl-1.6.1.jar:/usr/local/mahout-0.4/examples/target/dependency/slf4j-log4j12-1.6.1.jar:/usr/local/mahout-0.4/examples/target/dependency/snakeyaml-1.6.jar:/usr/local/mahout-0.4/examples/target/dependency/solr-commons-csv-3.1.0.jar:/usr/local/mahout-0.4/examples/target/dependency/speed4j-0.9.jar:/usr/local/mahout-0.4/examples/target/dependency/stringtemplate-3.2.jar:/usr/local/mahout-0.4/examples/target/dependency/uncommons-maths-1.2.2.jar:/usr/local/mahout-0.4/examples/target/dependency/uuid-3.2.0.jar:/usr/local/mahout-0.4/examples/target/dependency/watchmaker-framework-0.6.2.jar:/usr/local/mahout-0.4/examples/target/dependency/watchmaker-swing-0.6.2.jar:/usr/local/mahout-0.4/examples/target/dependency/xml-apis-1.0.b2.jar:/usr/local/mahout-0.4/examples/target/dependency/xpp3_min-1.1.4c.jar:/usr/local/mahout-0.4/examples/target/dependency/xstream-1.3.1.jar
> SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found
> binding in
> [jar:file:/usr/local/mahout-0.4/examples/target/mahout-examples-0.6-SNAPSHOT-job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/local/mahout-0.4/examples/target/dependency/slf4j-jcl-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/local/mahout-0.4/examples/target/dependency/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation. WARNING: org.apache.hadoop.metrics.jvm.EventCounter is
> deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in
> all the log4j.properties files. 11/08/16 20:10:32 INFO
> common.AbstractJob: Command line arguments: {--charset=UTF-8,
> --chunkSize=5, --endPhase=2147483647,
> --fileFilterClass=org.apache.mahout.text.PrefixAdditionFilter,
> --input=mahout-work/reuters-out, --keyPrefix=,
> --output=mahout-work/reuters-out-seqdir, --startPhase=0,
> --tempDir=temp} Exception in thread "main" java.io.IOException: Call
> to localhost/127.0.0.1:9000 failed on local exception:
> java.io.IOException: Broken pipe
> at
> org.apache.hadoop.ipc.Client.wrapException(Client.java:1065)
> at org.apache.hadoop.ipc.Client.call(Client.java:1033)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224)
> at $Proxy1.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:364)
> at
> org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:208)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:175)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1310)
> at
> org.apache.hadoop.fs.FileSystem.access$100(FileSystem.java:65)
> at
> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1328)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:109)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:210)
> at
> org.apache.mahout.text.SequenceFilesFromDirectory.run(SequenceFilesFromDirectory.java:59)
> at
> org.apache.mahout.text.SequenceFilesFromDirectory.run(SequenceFilesFromDirectory.java:110)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at
> org.apache.mahout.text.SequenceFilesFromDirectory.main(SequenceFilesFromDirectory.java:85)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> at
> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
> at
> org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:188)
> Caused by: java.io.IOException: Broken pipe
> at sun.nio.ch.FileDispatcher.write0(Native Method)
> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:122)
> at sun.nio.ch.IOUtil.write(IOUtil.java:93)
> at
> sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:352)
> at
> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
> at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> at
> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> at java.io.DataOutputStream.flush(DataOutputStream.java:123)
> at
> org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:746)
> at org.apache.hadoop.ipc.Client.call(Client.java:1011)
> ... 25 more rmr: cannot remove mahout-work/reuters-out-seqdir:
> No such file or directory. put: File mahout-work/reuters-out-seqdir
> does not exist.
this is a consistent error and I am getting it in every single installation I attempt.
What do I do to fix this?
This looks like an error from Hadoop and/or EC2. Hadoop workers failed writing data to each other for some reason. Why, I don't know, but I might guess that ports aren't open, even locally.
I have always used Amazon EMR directly instead.
Maybe you can debug by trying another M/R job to test. It is not related to Mahout directly as far as I can tell.

Resources