I have defined a custom function currently based on the very simple example here: https://docs.puppet.com/guides/custom_functions.html
module Puppet::Parser::Functions
newfunction(:transform_service_hash) do |args|
filename = args[0]
hash_to_be_transformed = args[1]
File.open(filename, 'a') {|fd| fd.puts hash_to_be_transformed }
end
end
This kinda works. I can call it like this:
$my_hash = { key => "value1" , key2 => "value2" }
notify{ "new hash!! $my_hash" :}
transform_service_hash('/var/tmp/blah',$my_hash)
and the file displays:
mgt21 ~ # cat /var/tmp/blah
keyvalue1key2value2
But, if I try to access elements of the hash, nothing changes:
module Puppet::Parser::Functions
newfunction(:transform_service_hash) do |args|
filename = args[0]
hash_to_be_transformed = args[1]
element1 = hash_to_be_transformed["key"]
File.open(filename, 'a') {|fd| fd.puts element1 }
end
end
The above block outputs the exact same data to /var/tmp/blah.
And, interestingly, if I remove the filename pass and define it statically in the module:
$my_hash = { key => "value1" , key2 => "value2" }
notify{ "new hash!! $my_hash. element1 is: $my_hash.key" :}
transform_service_hash($my_hash)
and
module Puppet::Parser::Functions
newfunction(:transform_service_hash) do |args|
hash_to_be_transformed = args[0]
element1 = hash_to_be_transformed["key"]
File.open('/var/tmp/blah2', 'a') {|fd| fd.puts element1 }
end
end
I get the following error: "Error 400 on SERVER: can't convert Hash into String" with a line reference pointing to "transform_service_hash($my_hash)"
I am new to both puppet and ruby...so I'm unsure I am not passing the element properly, if I am not receiving it properly, or if it something that puppet cannot handle. Please note that I am using version 3.8 of puppet and 1.8.7 of ruby.
Thanks for any help. I've been banging my head against this, and google hasn't been forthcoming yet.
---Edit to clarify my goals (I also edited my code a bit for specificity): I am attempting to pass a hash into a custom ruby function within puppet. The "test" hash has two elements: one string and one array. It is defined as such:
$my_hash = { key => "value1" , key2 => ['array_value1', 'array_value2'] }
$my_display_element=$my_hash["key2"][0]
notify{ "new hash!! $my_hash. the first value of the array stored in element2 is: $my_display_element" :}
transform_service_hash('/var/tmp/blah',$my_hash)
The function appears like so:
module Puppet::Parser::Functions
newfunction(:transform_service_hash) do |args|
filename = args[0]
hash_to_be_transformed = args[1]
element1 = args[1]["key"]
element2 = args[1]["key2"][0]
#element1 = hash_to_be_transformed["key"]
#element2 = hash_to_be_transformed["key2"][0]
File.open(filename, 'a') {|fd| fd.puts "hash_to_be_transformed: #{hash_to_be_transformed}\n" }
File.open(filename, 'a') {|fd| fd.puts "element1: #{element1}\n" }
File.open(filename, 'a') {|fd| fd.puts "element2: #{element2}\n" }
end
end
For now, I just want to be able to see that I am able to access elements within the passed hash like a hash. So I'd love for the output file to look like:
hash_to_be_transformed: keyvalue1key2array_value1array_value2
element1: value1
element2: array_value1
However, in the output file, I see:
mgt21 ~ # cat /var/tmp/blah
keyvalue1key2array_value1array_value2
Clearly, something is off here as my text is not being added and the full hash is just printed out just once and seemingly in string form.
I believe that this may be related to the error that I get when I don't pass in a file name (see above). I think that my hash is getting interpreted (or passed) as a string and, as such, I am unable to access the elements. Unfortunately, I still have been unable to verify this or figure out why it might be happening.
---Edit2 based on Matt's answer below.
I decided to simplify my code to isolate this "can't convert Hash into String error". I also made his suggested changes to remove the ambiguity from my key declarations.
$my_hash = { 'key' => "value1" , 'key2' => ['array_value1', 'array_value2'] }
$my_display_element=$my_hash["key2"][0]
notify{ "new hash!! $my_hash. the first value of the array stored in element2 is: $my_display_element" :}
transform_service_hash($my_hash)
and
module Puppet::Parser::Functions
newfunction(:transform_service_hash) do |args|
hash_to_be_transformed = args[0]
element1 = args[0]['key']
element2 = args[0]['key2'][0]
File.open('/var/tmp/blah', 'a') {|fd| fd.puts "hash_to_be_transformed: #{hash_to_be_transformed}\n" }
File.open('/var/tmp/blah', 'a') {|fd| fd.puts "element1: #{element1}\n" }
File.open('/var/tmp/blah', 'a') {|fd| fd.puts "element2: #{element2}\n" }
end
end
But, I still end up with the same "Hash to String error". It is worth noting that I also tried simplifying my hash to:
$my_hash = { 'key' => "value1" , 'key2' => "value2" }
and I still get the "Hash to String error".
I quickly took your custom parser function and converted it into pure ruby like the following:
hash = { 'key' => 'value1', 'key2' => %w(array_value1 array_value2) }
def newfunction(filename, a_hash)
element1 = a_hash['key']
element2 = a_hash['key2'][0]
File.open(filename, 'a') do |fd|
fd.puts "hash_to_be_transformed: #{a_hash}"
fd.puts "element1: #{element1}"
fd.puts "element2: #{element2}"
end
end
newfunction('foo.txt', hash)
This results in the output text file like the following:
hash_to_be_transformed: {"key"=>"value1", "key2"=>["array_value1", "array_value2"]}
element1: value1
element2: array_value1
This seems to confirm my initial suspicion about what is going wrong here. Your hash in Puppet of:
$my_hash = { key => "value1" , key2 => ['array_value1', 'array_value2'] }
has keys of implicit/ambiguous types. In the ruby code I used to test, I explicitly established them as strings. This also correlates strongly with these lines in your code failing:
element1 = args[1]["key"]
element2 = args[1]["key2"][0]
and your error message of:
Error 400 on SERVER: can't convert Hash into String
because you are specifying in your ruby code that you expect the keys to be string. Changing your hash in Puppet to:
$my_hash = { 'key' => "value1" , 'key2' => "value2" }
should fix this.
On an unrelated note, I recommend the use of linters to help you learn these languages. Puppet-Lint, Rubocop, and Reek will all help point out suboptimal and messy parts of your code to help you learn the new languages.
On a related note, you may want to put something like this at the top of your custom parser function:
raise(Puppet::ParseError, 'newfunction expects two arguments') if args.length != 2
After much gnashing of teeth (and some very helpful pointers from #MattSchuchard), I realized that none of the changes to my function were going into effect. One needs to restart the puppetmaster service after each change to a custom function: docs.puppet.com/guides/custom_functions.html (appropriately under "Gotchas").
Once I started restarting this service after each change to the function, my hash was able to be parsed properly:
from the .pp file:
$filename = "/var/tmp/test"
$my_hash = { 'key' => "value1" , 'key2' => ["M\'lady\n*doffs cap*", 'array_value2'] }
transform_service_hash($filename, $my_hash)
from the ruby file:
module Puppet::Parser::Functions
newfunction(:transform_service_hash) do |args|
filename = args[0]
hash_to_be_transformed = args[1]
array_val = hash_to_be_transformed['key2'][0]
File.open(filename, 'a') {|fd| fd.puts "#{array_val}\n" }
end
end
and output:
mgt21 tmp # cat test
M'lady
*doffs cap*
Related
I am trying to parse multiple XML files then output them into CSV files to list out the proper rows and columns.
I was able to do so by processing one file at a time by defining the filename, and specifically output them into a defined output file name:
File.open('H:/output/xmloutput.csv','w')
I would like to write into multiple files and make their name the same as the XML filenames without hard coding it. I tried doing it multiple ways but have had no luck so far.
Sample XML:
<?xml version="1.0" encoding="UTF-8"?>
<record:root>
<record:Dataload_Request>
<record:name>Bob Chuck</record:name>
<record:Address_Data>
<record:Street_Address>123 Main St</record:Street_Address>
<record:Postal_Code>12345</record:Postal_Code>
</record:Address_Data>
<record:Age>45</record:Age>
</record:Dataload_Request>
</record:root>
Here is what I've tried:
require 'nokogiri'
require 'set'
files = ''
input_folder = "H:/input"
output_folder = "H:/output"
if input_folder[input_folder.length-1,1] == '/'
input_folder = input_folder[0,input_folder.length-1]
end
if output_folder[output_folder.length-1,1] != '/'
output_folder = output_folder + '/'
end
files = Dir[input_folder + '/*.xml'].sort_by{ |f| File.mtime(f)}
file = File.read(input_folder + '/' + files)
doc = Nokogiri::XML(file)
record = {} # hashes
keys = Set.new
records = [] # array
csv = ""
doc.traverse do |node|
value = node.text.gsub(/\n +/, '')
if node.name != "text" # skip these nodes: if class isnt text then skip
if value.length > 0 # skip empty nodes
key = node.name.gsub(/wd:/,'').to_sym
if key == :Dataload_Request && !record.empty?
records << record
record = {}
elsif key[/^root$|^document$/]
# neglect these keys
else
key = node.name.gsub(/wd:/,'').to_sym
# in case our value is html instead of text
record[key] = Nokogiri::HTML.parse(value).text
# add to our key set only if not already in the set
keys << key
end
end
end
end
# build our csv
File.open('H:/output/.*csv', 'w') do |file|
file.puts %Q{"#{keys.to_a.join('","')}"}
records.each do |record|
keys.each do |key|
file.write %Q{"#{record[key]}",}
end
file.write "\n"
end
print ''
print 'output files ready!'
print ''
end
I have been getting 'read memory': no implicit conversion of Array into String (TypeError) and other errors.
Here's a quick peer-review of your code, something like you'd get in a corporate environment...
Instead of writing:
input_folder = "H:/input"
input_folder[input_folder.length-1,1] == '/' # => false
Consider doing it using the -1 offset from the end of the string to access the character:
input_folder[-1] # => "t"
That simplifies your logic making it more readable because it's lacking unnecessary visual noise:
input_folder[-1] == '/' # => false
See [] and []= in the String documentation.
This looks like a bug to me:
files = Dir[input_folder + '/*.xml'].sort_by{ |f| File.mtime(f)}
file = File.read(input_folder + '/' + files)
files is an array of filenames. input_folder + '/' + files is appending an array to a string:
foo = ['1', '2'] # => ["1", "2"]
'/parent/' + foo # =>
# ~> -:9:in `+': no implicit conversion of Array into String (TypeError)
# ~> from -:9:in `<main>'
How you want to deal with that is left as an exercise for the programmer.
doc.traverse do |node|
is icky because it sidesteps the power of Nokogiri being able to search for a particular tag using accessors. Very rarely do we need to iterate over a document tag by tag, usually only when we're peeking at its structure and layout. traverse is slower so use it as a very last resort.
length is nice but isn't needed when checking whether a string has content:
value = 'foo'
value.length > 0 # => true
value > '' # => true
value = ''
value.length > 0 # => false
value > '' # => false
Programmers coming from Java like to use the accessors but I like being lazy, probably because of my C and Perl backgrounds.
Be careful with sub and gsub as they don't do what you're thinking they do. Both expect a regular expression, but will take a string which they do a escape on before beginning their scan.
You're passing in a regular expression, which is OK in this case, but it could cause unexpected problems if you don't remember all the rules for pattern matching and that gsub scans until the end of the string:
foo = 'wd:barwd:' # => "wd:barwd:"
key = foo.gsub(/wd:/,'') # => "bar"
In general I recommend people think a couple times before using regular expressions. I've seen some gaping holes opened up in logic written by fairly advanced programmers because they didn't know what the engine was going to do. They're wonderfully powerful, but need to be used surgically, not as a universal solution.
The same thing happens with a string, because gsub doesn't know when to quit:
key = foo.gsub('wd:','') # => "bar"
So, if you're looking to change just the first instance use sub:
key = foo.sub('wd:','') # => "barwd:"
I'd do it a little differently though.
foo = 'wd:bar'
I can check to see what the first three characters are:
foo[0,3] # => "wd:"
Or I can replace them with something else using string indexing:
foo[0,3] = ''
foo # => "bar"
There's more but I think that's enough for now.
You should use Ruby's CSV class. Also, you don't need to do any string matching or regex stuff. Use Nokogiri to target elements. If you know the node names in the XML will be consistent it should be pretty simple. I'm not exactly sure if this is the output you want, but this should get you in the right direction:
require 'nokogiri'
require 'csv'
def xml_to_csv(filename)
xml_str = File.read(filename)
xml_str.gsub!('record:','') # remove the record: namespace
doc = Nokogiri::XML xml_str
csv_filename = filename.gsub('.xml', '.csv')
CSV.open(csv_filename, 'wb' ) do |row|
row << ['name', 'street_address', 'postal_code', 'age']
row << [
doc.xpath('//name').text,
doc.xpath('//Street_Address').text,
doc.xpath('//Postal_Code').text,
doc.xpath('//Age').text,
]
end
end
# iterate over all xml files
Dir.glob('*.xml').each { |filename| xml_to_csv(filename) }
What is the correct way to view the output of the puts statements below? My apologies for such a simple question.... Im a little rusty on ruby. github repo
require 'active_support'
require 'active_support/core_ext'
require 'indicators'
my_data = Indicators::Data.new(Securities::Stock.new(:symbol => 'AAPL', :start_date => '2012-08-25', :end_date => '2012-08-30').output)
puts my_data.to_s #expected to see Open,High,Low,Close for AAPL
temp=my_data.calc(:type => :sma, :params => 3)
puts temp.to_s #expected to see an RSI value for each data point from the data above
Maybe check out the awesome_print gem.
It provides the .ai method which can be called on anything.
An example:
my_obj = { a: "b" }
my_obj_as_string = my_obj.ai
puts my_obj_as_string
# ... this will print
# {
# :a => "b"
# }
# except the result is colored.
You can shorten all this into a single step with ap(my_obj).
There's also a way to return objects as HTML. It's the my_obj.ai(html: true) option.
Just use .inspect method instead of .to_s if you want to see internal properties of objects.
Currently I want to create an md5 hash from an argument. Then I want to write the hash into a file (the path is another argument).
That is the custom function:
module Puppet::Parser::Functions
newfunction(:write_line_to_file) do |args|
require 'md5'
filename = args[0]
str = MD5.new(lookupvar(args[1])).to_s
File.open(filename, 'a') {|fd| fd.puts str }
end
end
And the call in the puppet manifest:
write_line_to_file('/tmp/some_hash', "Hello world!")
The result I get is a file and the content is not the hash but the original string. (In the example Hello World!)
I know that this custom function has no practical use. I just want to understand how the md5 hash works.
---UPD---
new Function (it works properly):
require 'digest'
module Puppet::Parser::Functions
newfunction(:lxwrite_line_to_file) do |args|
filename = args[0]
str = Digest::MD5.hexdigest args[1]
File.open(filename, 'w') {|fd| fd.puts str }
end
end
Which ruby you are using?
In Ruby 2.0+ there is a Digest module (documentation here) - why you don't use it instead?.
You can use any hash, available in Digest, like this:
Digest::MD5.digest '123'
=> " ,\xB9b\xACY\a[\x96K\a\x15-#Kp"
or use hexdigest if you prefer hex representation
Digest::MD5.hexdigest '123'
=> "202cb962ac59075b964b07152d234b70"
There are also other hash-functions available there:
Digest::SHA2.hexdigest '123'
=> "a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3"
Given the following CSV file, how would you remove all rows that contain the word 'true' in the column 'foo'?
Date,foo,bar
2014/10/31,true,derp
2014/10/31,false,derp
I have a working solution, however it requires making a secondary CSV object csv_no_foo
#csv = CSV.read(#csvfile, headers: true) #http://bit.ly/1mSlqfA
#headers = CSV.open(#csvfile,'r', :headers => true).read.headers
# Make a new CSV
#csv_no_foo = CSV.new(#headers)
#csv.each do |row|
# puts row[5]
if row[#headersHash['foo']] == 'false'
#csv_no_foo.add_row(row)
else
puts "not pushing row #{row}"
end
end
Ideally, I would just remove the offending row from the CSV like so:
...
if row[#headersHash['foo']] == 'false'
#csv.delete(true) #Doesn't work
...
Looking at the ruby documentation, it looks like the row class has a delete_if function. I'm confused on the syntax that that function requires. Is there a way to remove the row without making a new csv object?
http://ruby-doc.org/stdlib-1.9.2/libdoc/csv/rdoc/CSV/Row.html#method-i-each
You should be able to use CSV::Table#delete_if, but you need to use CSV::table instead of CSV::read, because the former will give you a CSV::Table object, whereas the latter results in an Array of Arrays. Be aware that this setting will also convert the headers to symbols.
table = CSV.table(#csvfile)
table.delete_if do |row|
row[:foo] == 'true'
end
File.open(#csvfile, 'w') do |f|
f.write(table.to_csv)
end
You might want to filter rows in a ruby manner:
require 'csv'
csv = CSV.parse(File.read(#csvfile), {
:col_sep => ",",
:headers => true
}
).collect { |item| item[:foo] != 'true' }
Hope it help.
I'm creating a hash that will eventually be dumped on disk in YAML, but I need to capture multiple values stored in a file on disk and insert them into a hash. I can successfully create a variable with comma separated values, but I need to insert those values into a my "classes" key:
variable_values = "class1,class2,class3"
Ultimately, I need to get them into my test hash so it simulates something like this:
test_hash = {'Classes' => ['class1', 'class2', 'class3']}
Finally, I can output them to yaml so it looks like this:
---
classes:
- class1
- class2
- class3
What's the best way to iterate through the values and insert them into the hash? Thanks for any help you can offer!
You'd probably want something like:
test_hash = {'Classes' => variable_values.split(',')}
If you're wanting to serialize Ruby Classes (I'm not able to tell for sure), you'll probably want the following code (courtesy of opensoul.org, and as used in the Small Eigen Collider)
class Module
yaml_as "tag:ruby.yaml.org,2002:module"
def Module.yaml_new( klass, tag, val )
if String === val
val.split(/::/).inject(Object) {|m, n| m.const_get(n)}
else
raise YAML::TypeError, "Invalid Module: " + val.inspect
end
end
def to_yaml( opts = {} )
YAML::quick_emit( nil, opts ) { |out|
out.scalar( "tag:ruby.yaml.org,2002:module", self.name, :plain )
}
end
end
class Class
yaml_as "tag:ruby.yaml.org,2002:class"
def Class.yaml_new( klass, tag, val )
if String === val
val.split(/::/).inject(Object) {|m, n| m.const_get(n)}
else
raise YAML::TypeError, "Invalid Class: " + val.inspect
end
end
def to_yaml( opts = {} )
YAML::quick_emit( nil, opts ) { |out|
out.scalar( "tag:ruby.yaml.org,2002:class", self.name, :plain )
}
end
end
The code currently throws an exception if you try to serialize/deserialize anonymous classes (something I could fix but don't need to), and apart from that it works well for me.