Is there a way to make Psych a little dumber? If a string matches certain regular expression it tries to convert it to date and fails miserably. So far I switched to Syck for serializing but it's no longer maintained and I'm looking for better solution.
Here's an example of what's going on:
s = "2222-33-44" # => "2222-33-44"}
require 'yaml' # => true
YAML::dump s # ArgumentError: invalid date
YAML::ENGINE.yamler = 'syck' # => "syck"
YAML::dump s # => "--- \"2222-33-44\"\n"
I have found that it is a known bug. It has been fixed but looks like the version in RubyGems is not containing the fix.
Also as the git repository doesn't have a proper .gemspec is not easy way to install from the repo.
You can fork the repo and add a proper .gemspec and use a Gemfile that points to your fork repo.
Also you can ask kindly to Aaron to release the fix.
Related
A recent change in Ruby's YAML Library (Psych 4) causes "unsafe" YAMLs to fail if they contain aliases, or try to instantiate unspecified classes. This is discussed in multiple places, like this StackOverflow question.
I am trying to figure out how to tell the derivative yaml/store library to allow loading unsafe YAMLs, or to provide it with my list of allowed classes.
The documentation is scarce as far as I could find, and after reading it, this is the only logical attempt I could come up with:
require 'date'
require 'yaml/store'
# 1. These options work perfectly with YAML.load_file, but not with YAML::Store
# 2. These options are not needed in Psych < 4.0
yaml_opts = { aliases: true, permitted_classes: [Time, Date, Symbol] }
store = YAML::Store.new 'log.yml', yaml_opts
data = store.transaction { store[:entries] }
p data
using this YAML file:
# log.yml
:entries:
- :timestamp: 2018-07-09 00:00:00.000000000 +03:00
:action: Comment
:comment: Started logging
This fails with Psych 4, and succeeds with Psych 3.
# Gemfile
source "https://rubygems.org"
gem 'psych', '>= 4.0' # fail
# gem 'psych', '< 4.0' # pass
As a related anecdote, the example demonstrated in the docs, also fails as-is when trying to load it with store.transaction { store["people"] }
Although this is not the proper way of doing things, until there is a better answer, I found that adding the below code fixes the problem.
module YAML
class << self
alias_method :load, :unsafe_load
end
end
This simply restores the underlying YAML::load method to its 3.x behavior of unsafe_load instead of safe_load.
In cases where my YAMLs come from a trusted source (100% of my use cases), I do not see any benefit in the new Psych 4 behavior, and feel it is ok (although awkward) to revert it.
The relevant source code reference is the 3.3.2 → 4.0.0 diff
For the sake of security internal Ruby gems in the Gemfile should always be referenced inside a source block so it never tries to fetch them from rubygems.org. I'd like to automate finding where people fail to do this, so would like to parse the Gemfile, find any gems that match our internal names, and check that rubygems.org isn't in their possible sources list.
source 'https://rubygems.org'
gem 'rails'
gem 'my-private-gem1' # this should be in the source block below
source PRIVATE_GEM_REPO do
gem 'my-private-gem2'
end
I've seen you can parse the Gemfile
Bundler::Definition.build('Gemfile', '', {})
But can't find anything in the returned data structure that shows me the available / allowed sources per gem
If I include the Gemfile.lock I see more source info, but it doesn't seem right because every gem lists all my sources regardless of if they're in a source block
Bundler::Definition.build('Gemfile', 'Gemfile.lock', {}).
locked_gems.
specs.
map {|g| [g.full_name, g.source.remotes.map(&:hostname).join(', ')]}
=> ["rails-6.0.3.4", "my.private.gemserver, rubygems.org"],
["my-private-gem1-1.0.0", "my.private.gemserver, rubygems.org"],
["my-private-gem2-1.0.0", "my.private.gemserver, rubygems.org"]]
Any thoughts on how to parse the Gemfile to find that my-private-gem1 is outside a source block?
Figured it out finally, just took awhile digging through the Bundler methods - and a coworker's help.
Bundler::Definition.
build('Gemfile', '', nil).
dependencies.
map {|dep| [dep.name, dep.source&.remotes&.map(&:hostname)&.join(', ')]}
=>
[["rails", nil],
["my-private-gem1", nil],
["my-private-gem2", "my.private.gemserver"]]
Now I can easily search that resulting data structure for any private gems that aren't locked down to my private gem server.
Preface
While I was writing this answer, the OP found a Bundler-specific answer. However, I offer a more generalizable solution below. This solution also offers user feedback that may make it easier to fix the file.
Finding Candidate Gems by Column Alignment, with Whitelisting
If you can safely assume that your Gemfile is always properly indented, the KISS solution may be to simply identify the gems that aren't indented within a group definition. For example:
# example Gemfile to test against
GEMFILE = <<~'EOF'
source 'https://rubygems.org'
gem 'rails'
gem 'my-private-gem1' # this should be in the source block below
source PRIVATE_GEM_REPO do
gem 'my-private-gem2'
end
EOF
# gems that are acceptable in a non-group context
whitelist = Regexp.new %w[rails sass-rails webpacker].join(?|)
UngroupedGem = Struct.new :line_no, :line_txt, :gem_name
ungrouped_gems = []
GEMFILE.lines.each_with_index do |line_txt, line_no|
next if line_txt =~ whitelist or line_txt !~ /^\s*gem/
gem_name = line_txt.match(/(?<=(['"]))(.*?)(?=\1)/)[0]
ungrouped_gems.append(
UngroupedGem.new line_no.succ, line_txt, gem_name
).compact!
end
# tell the user what actions to take
if ungrouped_gems.any?
puts "Line No.\tGem Name"
ungrouped_gems.each { printf "%d\t\t%s\n", _1.line_no, _1.gem_name }
else
puts "No gems need to be moved."
end
With this example, it will print:
Line No. Gem Name
3 my-private-gem1
5 my-private-gem2
which will give you a solid idea of what lines in the Gemfile need to be moved, and which specific gems are involved.
I am following the Padrino tutorial from here:
https://www.padrinorb.com/guides/blog-tutorial
I am copy and pasting the commands but I quickly ran into an error I don't understand:
$ padrino g controller posts get:index get:show
create app/controllers/posts.rb
create app/views/posts
apply tests/shoulda
/Users/waprin/.rvm/gems/ruby-2.1.0/gems/padrino-gen-0.12.4/lib/padrino-gen/generators/controller.rb:66:in `prepend': can't modify frozen String (RuntimeError)
from /Users/waprin/.rvm/gems/ruby-2.1.0/gems/padrino-gen-0.12.4/lib/padrino-gen/generators/controller.rb:66:in `create_controller'
This might be a bit late, but in case anyone else runs across this error (and because I just worked through the same tutorial) I'll post anyway...
It looks like there's an issue when generating controllers if a test component is specified. In this case you're using shoulda, but the same happens when using rspec and maybe others. It's been reported as a bug: https://github.com/padrino/padrino-framework/issues/1850 and has been fixed, but isn't yet part of a stable release.
One option to fix this would be to change your Gemfile to work with the latest from their github repo. To do this delete your GemFile.lock file, and comment out the line under 'Padrino Stable Gem' in your GemFile:
gem 'padrino', '0.12.4'
then uncomment the line under 'Or Padrino Edge':
gem 'padrino', :github => 'padrino/padrino-framework'
then re-run bundle install.
Of course, you'll no longer be running the stable release, and that may come with other trade-offs.
As a side-note, I believe that the guide on that page is fairly out of date. I also needed to replace:
get :index do
#posts = Post.all(:order => 'created_at desc')
render 'posts/index'
end
with:
get :index, :provides => [:html, :rss, :atom] do
#posts = Post.order('created_at desc')
render 'posts/index'
end
in the Post controller as the active record interface has changed since the time that the guide was written.
I was able to sole this problem by simply going to padrino gem path.
For me it was:
/Users/ahmadhassan/.rvm/gems/ruby-2.2.0/gems/padrino-gen-0.12.4/lib/padrino-gen/generators
open controller.rb and change line number 61:
path = #controller
to
path = #controller.dup
Is there anyway to get Ruby's require statement to download a file from somewhere like github rather than just the local file system?
Update: Sorry I should have made the question clearer. I want to download a file that contains Ruby module and import it into my script rather than just downloading an image or some other arbitrary file within my script.
In other words something like this
require 'http:\\github.com\myrepo\snippet.rb'
puts 'hi'
By default, this is not possible. Also, it's not a good idea for security reasons.
You have a couple of alternatives. If the file you want to include is a Gem and Git repository, then you can use Bundler to download and package the dependency in your project. Then you'll be able to require the file directly in your source code.
This is the best and safest way to include an external dependency.
If you trust the source and you really know what you are doing, you can download the file using Net::HTTP (or any other HTTP library) and eval the body directly in your Ruby code.
You can package everything in a custom require_remote function.
You could download and eval it
require "open-uri"
alias :require_old :require
def require(path)
return false if $".include?(path)
unless path=~ /\Ahttp:\/\/
return require_old(path)
end
eval(open(path).read)
$"<< path
true
end
Be aware, this code has no error checking for network outages nonexisting files, ... . I also believe it is in general not a good idea to require libraries this way, there are security and reliability problems in this approach. But maybe you have a valid usecase for this.
you can include a remote gem from within Gemfiles then it will download when you run bundle install
After reading this question and answers I wanted something a little more bullet proof and verbose that used a paradigm of creating a local file from a repo and then requiring it, only if it didn't already exist locally already. The request for the repo version is explicit via the method repo_require. Used on files you control, this approach improves security IMO.
# try local load
def local_require(filename, relative_path)
relative_flname = File.join(relative_path, filename)
require_relative(relative_flname)
end
# try loading locally first, try repo version on load error
# caution: only use with files you control access to!
def repo_require(raw_repo_prefix, filename, relative_path = '')
local_require(filename, relative_path)
rescue LoadError => e
puts e.message
require 'open-uri'
tempdir = Dir.mktmpdir("repo_require-")
temp_flname = File.join(tempdir, File.basename(filename))
return false if $LOADED_FEATURES.include?(temp_flname)
remote_flname = File.join(raw_repo_prefix, filename)
puts "file not found locally, checking repo: #{remote_flname}"
begin
File.open(temp_flname, 'w') do |f|
f.write(URI.parse(remote_flname).read)
end
rescue OpenURI::HTTPError => e
raise "Error: Can't load #{filename} from repo: #{e.message} - #{remote_flname}"
end
require(temp_flname)
FileUtils.remove_entry(tempdir)
end
Then you could call repo_require like this:
repo_require('https://raw.githubusercontent.com/username/reponame/branch',
'filename', 'relative_path')
The relative_path would the the relative path you would use for the file if the repo was locally installed. For example, you may have something like require_relative '../lib/utils.rb'. In this example filename='lib/utils.rb' and relative_path='..'. This information allows the repo url to be constructed correctly as it does not use the relative path portion.
Ive seen lots of stuff
Foo::VERSION::STRING
Foo::VERSION
Foo::Version
any suggestions what the best/most-common default is ?
First of all, Foo::Version seems wrong to me. That implicates that version is a Class or a Module, but should be a constant, as first two of your examples.
Apart from that, there is no golden rule here. Here are a few examples:
Rails:
Rails::VERSION::STRING
(Note: a module VERSION with all capitals.)
SimpleForm:
SimpleForm::VERSION
Nokogiri:
Nokogiri::VERSION
RSpec:
RSpec::Version::STRING
(Note, a module Version, compare with Rails above.)
Cucumber:
Cucumber::VERSION
SimpleForm:
SimpleForm::VERSION
Most of the above have their version defined in the lib/[name]/version.rb file, except for Rails, which has a version.rb in the root of the project, and Cucumber, which has buried it in the lib/cucumber/platform.rb file.
Also note that all of the above use Strings as a version. (Rails even concatenates MAJOR, MINOR, TINY, and PRE constants into a single VERSION constant.) This allows for easy comparison of versions:
"1.2.3" > "0.4.2" # => true
"1.2.3" > "1.2.2" # => true
"1.2.3" > "1.2" # => true
However, such a comparison fails when you run into double digits for the patch level:
"1.2.3" > "1.2.12" # => true
To me, defining Foo::VERSION is your best bet, define it in lib/foo/version.rb, and make it a String value.