In an ERB template used by Puppet, I'm trying to sort a YAML output of a hash to make sure it's always the same output.
Here is what I have so far (in mytest/templates/hash.erb):
<%=
class MyHash < Hash
def to_yaml( opts = {} )
YAML::quick_emit( self, opts ) do |out|
out.map( taguri, to_yaml_style ) do |map|
keys.sort.each do |k|
v = self[k]
map.add( k, v )
end
end
end
end
end
myScope = scope.to_hash.reject{|k,v| k.to_s =~ /(uptime|timestamp|free)/}
MyHash[myScope].to_yaml
-%>
which yields:
$ puppet apply -e 'include mytest' --modulepath .
Failed to parse template mytest/hash.erb:
Filepath: /usr/lib/ruby/1.8/yaml.rb
Line: 391
Detail: wrong argument type String (expected Data)
at /home/rpinson/bas/puppet/mytest/manifests/init.pp:3 on node foo.example.com
This is the content of mytest/manifests/init.pp:
class mytest {
notify { 'toto':
message => template('mytest/hash.erb'),
}
}
I can't seem to understand where this type Data comes from, and how to cast the parameters properly in order to get this to work…
It turns out, the source of problem is ZAML, YAML replacement which Puppet uses for performance purposes.
It can be proved by testing this erb-template without Puppet.
I'm pretty sure it will work.
I'm investigating to allow to work them together.
UPDATE:
ZAML.dump(MyHash[myScope]) # instead of MyHash[myScope].to_yaml
This may heal problem, at least it does my one that really similar.
Related
E.G.
def do_the_thing(file_to_load, hash_path)
file = File.read(file)
data = JSON.parse(file, { symbolize_names: true })
data[sections.to_sym]
end
do_the_thing(file_I_want, '[:foo][:bar][0]')
Tried a few methods but failed so far.
Thanks for any help in advance :)
Assuming you missed the parameters names...
Lets assume our file is:
// test.json
{
"foo": {
"bar": ["foobar"]
}
}
Recomended solution
Does your param really need to be a string??
If your code can be more flexible, and pass arguments as they are on ruby, you can use the Hash dig method:
require 'json'
def do_the_thing(file, *hash_path)
file = File.read(file)
data = JSON.parse(file, symbolize_names: true)
data.dig(*hash_path)
end
do_the_thing('test.json', :foo, :bar, 0)
You should get
"foobar"
It should work fine !!
Read the rest of the answer if that doesn't satisfy your question
Alternative solution (using the same argument)
If you REALLY need to use that argument as string, you can;
Treat your params to adapt to the first solution, it won't be a small or fancy code, but it will work:
require 'json'
BRACKET_REGEX = /(\[[^\[]*\])/.freeze
# Treats the literal string to it's correspondent value
def treat_type(param)
# Remove the remaining brackets from the string
# You could do this step directly on the regex if you want to
param = param[1..-2]
case param[0]
# Checks if it is a string
when '\''
param[1..-2]
# Checks if it is a symbol
when ':'
param[1..-1].to_sym
else
begin
Integer(param)
rescue ArgumentError
param
end
end
end
# Converts your param to the accepted pattern of 'dig' method
def string_to_args(param)
# Scan method will break the match results of the regex into an array
param.scan(BRACKET_REGEX).flatten.map { |match| treat_type(match) }
end
def do_the_thing(file, hash_path)
hash_path = string_to_args(hash_path)
file = File.read(file)
data = JSON.parse(file, symbolize_names: true)
data.dig(*hash_path)
end
so:
do_the_thing('test.json', '[:foo][:bar][0]')
returns
"foobar"
This solution though is open to bugs when the "hash_path" is not on an acceptable pattern, and treating it's bugs might make the code even longer
Shortest solution (Not safe)
You can use Kernel eval method which I EXTREMELY discourage to use for security reasons, read the documentation and understand its danger before using it
require 'json'
def do_the_thing(file, hash_path)
file = File.read(file)
data = JSON.parse(file, symbolize_names: true)
eval("data#{hash_path}")
end
do_the_thing('test.json', '[:foo][:bar][0]')
If the procedure you were trying to work with was just extracting the JSON data to an object, you might find yourself using either of the following scenarios:
def do_the_thing(file_to_load)
file = File.read(file)
data = JSON.parse(file, { symbolize_names: true })
data[sections.to_sym]
end
do_the_thing(file_I_want)[:foo][:bar][0]
or use the dig function of Hash :
def do_the_thing(file_to_load, sections)
file = File.read(file)
data = JSON.parse(file, { symbolize_names: true })
data.dig(*sections)
end
do_the_thing(file_I_want, [:foo, :bar, 0])
I have a ruby (sinatra) app that I am working on, and my input is a url and if verbose or not (true or false), so basically like this:
The url would look like this: http://localhost:4567/git.company.com&v=false for example.
And the code to fetch those is this:
get '/:url' do |tool_url|
url = params[:url].to_s
is_verbose = params[:v].to_s
I have different classes separated in different files and I'm including them into my main script like this:
Dir["#{File.dirname(__FILE__)}/lib/*.rb"].each { |f| require(f) }
(And a sample file would be something like this), gitlab.rb:
class Gitlab
$gitlab_token = 'TOKEN_GOES_HERE'
def initialize(url, v)
##regex =~ /git.company.com/
##gitlab_url = url
##is_verbose = v
end
def check_gitlab(gitlab_url, is_verbose)
_gitlab_overall = '/health_check.json?token=#{gitlab_token}'
_gitlab_cache = '/health_check/cache.json?token=#{gitlab_token}'
_gitlab_database = '/health_check/database.json?token=#{gitlab_token}'
_gitlab_migrations = '/health_check/migrations.json?token=#{gitlab_token}'
unless is_verbose = true
CheckString.check_string_from_page('https://' + gitlab_url + gitlab_overall, 'success')
else
end
end
end
Now, I want to be able to dynamically know which "class" to use to do a certain job based on the URL that's entered by the user, so my idea was to iterate through those classes looking for a particular variable to match with the input.
I need guidance in this because I've been stuck on this for quite some time now; I've tried so many things that I can think of, but none worked.
Disclaimer: Please bear with me here, because I'm very new to Ruby and I'm not that great in OOP languages (haven't really practiced them that much).
EDIT: I'm open to any suggestion, like if there's a different logic that's better than this, please do let me know.
Make a hash { Regexp ⇒ Class }:
HASH = {
/git.company.com/ => Gitlab,
/github.com/ => Github
}
and then do:
handler = HASH.detect { |k, _| k =~ url }.last.new
The above will give you an instance of the class you wanted.
Sidenotes:
is_verbose = params[:v].to_s always results in is_verbose set to truthy value, check for params[:v].to_s == "true"
is_verbose = true is an assignment, you wanted to use just unless is_verbose.
To make it runtime-resolving, force the plugins to a) include Plugin and b) declare resolve method. Plugin module should define a callback hook:
module Plugin
def self.included(base)
Registry::HASH[-> { base.resolve }] = base
end
end
resolve method should return a regexp, the lambda is here to make it resolved on parsing stage:
class PluginImpl
include Plugin
def resolve
/git.company.com/
end
end
And then match when needed:
handler = HASH.detect { |k, _| k.() =~ url }.last.new
Other way round would be to use ObjectSpace to detect classes, including the module, or declare the TracePoint on base in included callback to provide a direct map, but all this is overcomplicating.
I am writing a helper API in Ruby to automatically create and manipulate node definitions. My code is working; it can read and write the node defs successfully, however, it is a bit clunky.
Ruby is not my main language, so I'm sure there is a cleaner, and more rubyesque solution. I would appreciate some advice or suggestions.
Each host has its own file in manifests/nodes containing just the node definition. e.g.
node 'testnode' {
class {'firstclass': }
class {'secondclass': enabled => false }
}
The classes all are either enabled (default) or disabled elements. In the Ruby code, I store these as an instance variable hash #elements.
The read method looks like this:
def read()
data = File.readlines(#filepath)
for line in data do
if line.include? 'class'
element = line[/.*\{'([^\']*)':/, 1]
if #elements.include? element.to_sym
if not line.include? 'enabled => false'
#elements[element.to_sym] = true
else
#elements[element.to_sym] = false
end
end
end
end
end
And the write method looks like this:
def write()
data = "node #{#hostname} {\n"
for element in #elements do
if element[1]
line = " class {'#{element[0]}': }\n"
else
line = " class {'#{element[0]}': enabled => false}\n"
end
data += line
end
data += "}\n"
file = File.open(#filepath, 'w')
file.write(data)
file.close()
end
One thing to add is that these systems will be isolated from the internet. So I'd prefer to avoid large number of dependency libraries as I'll need to install / maintain them manually.
If your goal is to define your node's programmatically, there is a much more straightforward way then reading and writing manifests. One of the built-in features of puppet is "External Node Classifiers"(ENC). The basic idea is that something external to puppet will define what a node should look like.
In the simplest form, the ENC can be a ruby/python/whatever script that writes out yaml with the list of classes and enabled parameters. Reading and writing yaml from ruby is as simple as it gets.
Ruby has some pretty good methods to iterate over data structures. See below for an example of how to rubify your code a little bit. I am by no means an expert on the subject, and have not tested the code. :)
def read
data = File.readlines(#filepath)
data.each_line do |line|
element = line[/.*\{'([^\']*)':/, 1].to_sym
if #elements.include?(element)
#elements[element] = line.include?('enabled => false') ? false : true
end
end
end
def write
File.open(#filepath, 'w') do |file|
file.puts "node #{#hostname} {"
#elements.each do |element|
if element[1]
file.puts " class {'#{element[0]}': }"
else
file.puts " class {'#{element[0]}': enabled => false }"
end
end
file.puts '}'
end
end
Hope this points you in the right direction.
In the following Ruby example, is there a mode to have YAML NOT silently ignore the duplicate key 'one'?
irb(main):001:0> require 'yaml'
=> true
irb(main):002:0> str = '{ one: 1, one: 2 }'
=> "{ one: 1, one: 2 }"
irb(main):003:0> YAML.load(str)
=> {"one"=>2}
Thanks!
Using Psych, you can traverse the AST tree to find duplicate keys. I'm using the following helper method in my test suite to validate that there are no duplicate keys in my i18n translations:
def duplicate_keys(file_or_content)
yaml = file_or_content.is_a?(File) ? file_or_content.read : file_or_content
duplicate_keys = []
validator = ->(node, parent_path) do
if node.is_a?(Psych::Nodes::Mapping)
children = node.children.each_slice(2) # In a Mapping, every other child is the key node, the other is the value node.
duplicates = children.map { |key_node, _value_node| key_node }.group_by(&:value).select { |_value, nodes| nodes.size > 1 }
duplicates.each do |key, nodes|
duplicate_key = {
file: (file_or_content.path if file_or_content.is_a?(File)),
key: parent_path + [key],
occurrences: nodes.map { |occurrence| "line: #{occurrence.start_line + 1}" },
}.compact
duplicate_keys << duplicate_key
end
children.each { |key_node, value_node| validator.call(value_node, parent_path + [key_node.try(:value)].compact) }
else
node.children.to_a.each { |child| validator.call(child, parent_path) }
end
end
ast = Psych.parse_stream(yaml)
validator.call(ast, [])
duplicate_keys
end
There is a solution involving a linter, but I'm not sure it will be relevant to
you since it's not a 100% Ruby solution. I'll post it anyway since I don't know
any way to do this in Ruby:
You can use the yamllint command-line tool:
sudo pip install yamllint
Specifically, it has a rule key-duplicates that detects duplicated keys:
$ cat test.yml
{ one: 1, one: 2 }
$ yamllint test.yml
test.yml
1:11 error duplication of key "one" in mapping (key-duplicates)
One of the things I do to help maintain the YAML files I use, is write code to initially generate it from a known structure in Ruby. That gets me started.
Then, I'll write a little snippet that loads it and outputs what it parsed using either PrettyPrint or Awesome Print so I can compare that to the file.
I also sort the fields as necessary to make it easy to look for duplicates.
No. You'd have to decide how to rename the keys since hash keys have to be unique - I'd go for some workaround like manually looking for keys that are the same and renaming them before you do a YAML::load.
I've tried and tried, but I can't make this less ugly/more ruby-like. It seems like there just must be a better way. Help me learn.
class Df
attr_accessor :thresh
attr_reader :dfo
def initialize
#dfo = []
#df = '/opt/TWWfsw/bin/gdf'
case RUBY_PLATFORM
when /hpux/i
#fstyp = 'vxfs'
when /solaris/i
# fix: need /tmp too
#fstyp = 'ufs'
when /linux/i
#df = '/bin/df'
#fstyp = 'ext3'
end
#dfo = parsedf
end
def parsedf
ldf = []
[" "," -i"] .each do |arg|
fields = %w{device size used avail capp mount}
fields = %w{device inodes inodesused inodesavail iusep mount} if arg == ' -i'
ldf.push %x{#{#df} -P -t #{#fstyp}#{arg}}.split(/\n/)[1..-1].collect{|line| Hash[*fields.zip(line.split).flatten]}
end
out = []
# surely there must be an easier way
ldf[0].each do |x|
ldf[1].select { |y|
if y['device'] == x['device']
out.push x.merge(y)
end
}
end
out
end
end
In my machine, your ldf array after the df calls yields the following:
irb(main):011:0> ldf
=> [[{"device"=>"/dev/sda5", "size"=>"49399372", "mount"=>"/", "avail"=>"22728988", "used"=>"24161036", "capp"=>"52%"}], [{"device"=>"/dev/sda5", "inodes"=>"3137536", "mount"=>"/", "iusep"=>"13%", "inodesavail"=>"2752040", "inodesused"=>"385496"}]]
The most flexible approach to merging such a structure is probably something along these lines:
irb(main):013:0> ldf.flatten.inject {|a,b| a.merge(b)}
=> {"device"=>"/dev/sda5", "inodes"=>"3137536", "size"=>"49399372", "mount"=>"/", "avail"=>"22728988", "inodesavail"=>"2752040", "iusep"=>"13%", "used"=>"24161036", "capp"=>"52%", "inodesused"=>"385496"}
Some ruby programmers frown on this use of inject, but I like it, so your mileage may vary.
As for helping making your code more ruby like, I suggest you talk to some experienced rubyist you might know over your code to help you rewriting it in a way that follows good style and best practices. Probably that would the preferable than to just have someone rewrite it for you here.
Best of Luck!
Didn't test the code, but here goes:
ARGUMENTS = {
" " => %w{size used avail capp mount},
" -i" => %w{inodes inodesused inodesavail iusep mount}
}
def parsedf
# Store resulting info in a hash:
device_info = Hash.new do |h, dev|
h[dev] = {} # Each value will be a empty hash by default
end
ARGUMENTS.each do |arg, fields|
%x{#{#df} -P -t #{#fstyp}#{arg}}.split(/\n/)[1..-1].each do |line|
device, *data = line.split
device_info[device].merge! Hash[fields.zip(data)]
end
end
device_info
end
Notes: returns something a bit different than what you had:
{ "/dev/sda5" => {"inodes" => "...", ...},
"other device" => {...}
}
Also, I'm assuming Ruby 1.8.7 or better for Hash[key_value_pairs], otherwise you can resort to the Hash[*key_value_pairs.flatten] form you had
Depending on your needs, you should consider switch the fields from string to symbols; they are the best type of keys.