Update hash values with hash key - ruby

I'm facing a problem that I couldn't find a working solution yet.
I have my YAML config file for the environment, let's call it development.yml.
This file is used to create the hash that should be updated:
data = YAML.load_file(File.join(Rails.root, 'config', 'environments', 'development.yml'))
What I'm trying to accomplish is something along these lines. Let's suppose we have an element of the sort
data['server']['test']['user']
data['server']['test']['password']
What I want to have is:
data['server']['test']['user'] = #{Server.Test.User}
data['server']['test']['password'] = #{Server.Test.Password}
The idea is to create a placeholder for each value that is the key mapping for that value dynamically, going until the last level of the hash and replacing the value with the mapping to this value, concatenating the keys.
Sorry, it doesn't solve my problem. The location data['server']['test']['user'] will be built dynamically, via a loop that will go through a nested Hash. The only way I found to do it was to append to the string the key for the current iteration of the Hash. At the end, I have a string like "data['server']['test']['name']", which I was thinking on converting to a variable data['server']['test']['name'] and then assigning to this variable the value #{Server.Test.Name}. Reading my question I'm not sure if this is clear, I hope this helps to clarify it.
Input sample:
api: 'active'
server:
test:
user: 'test'
password: 'passwordfortest'
prod:
user: 'nottest'
password: 'morecomplicatedthantest'
In this case, the final result should be to update this yml file in this way:
api: #{Api}
server:
test:
user: #{Server.Test.User}
password: #{Server.Test.Password}
prod:
user: #{Server.Prod.User}
password: #{Server.Prod.Password}
It sounds silly, but I couldn't figure out a way to do it.

I am posting another answer now since I realize what the question is all about.
Use Iteraptor gem:
require 'iteraptor'
require 'yaml'
# or load from file
yaml = <<-YAML.squish
api: 'active'
server:
test:
user: 'test'
password: 'passwordfortest'
prod:
user: 'nottest'
password: 'morecomplicatedthantest'
YAML
mapped =
yaml.iteraptor.map(full_parent: true) do |parent, (k, _)|
v = parent.map(&:capitalize).join('.')
[k, "\#{#{v}}"]
end
puts YAML.dump(mapped)
#⇒ ---
# api: "#{Api}"
# server:
# test:
# user: "#{Server.Test.User}"
# password: "#{Server.Test.Password}"
# prod:
# user: "#{Server.Prod.User}"
# password: "#{Server.Prod.Password}"
puts YAML.dump(mapped).delete('"')
#⇒ ---
# api: #{Api}
# server:
# test:
# user: #{Server.Test.User}
# password: #{Server.Test.Password}
# prod:
# user: #{Server.Prod.User}
# password: #{Server.Prod.Password}

Use String#%:
input = %|
data['server']['host']['name'] = %{server_host}
data['server']['host']['user'] = %{server_host_user}
data['server']['host']['password'] = %{server_host_password}
|
puts (
input % {server_host: "Foo",
server_host_user: "Bar",
server_host_password: "Baz"})
#⇒ data['server']['host']['name'] = Foo
# data['server']['host']['user'] = Bar
# data['server']['host']['password'] = Baz

You can not add key-value pair to a string.
data['server']['host'] # => which results in a string
Option 1:
You can either save Server.Host as host name in the hash
data['server']['host']['name'] = "#{Server.Host}"
data['server']['host']['user'] = "#{Server.Host.User}"
data['server']['host']['password'] = "#{Server.Host.Password}"
Option 2:
You can construct the hash in a single step with Host as key.
data['server']['host'] = { "#{Server.Host}" => {
'user' => "#{Server.Host.User}",
'password' => "#{Server.Host.Password}"
}
}

Related

Ruby update all same object data in hash array for JSON.parse

So a = first is
=> <Ng::EntityConfiguration id: 15903, entity_id: 1, entity_type: "Ng::Company", key: "wpa2.psk", value: "[{"ssid":"Main-Hall-Staff","password":"abc123","dhcp":"Enabled"},{"ssid":"Main-Hall-Guest","password":"def456","dhcp":"Disabled"}]", created_at: "2016-11-08 11:03:51", updated_at: "2016-11-08 11:03:51", name: "WIFI/Main Hall">
I have a.value which is will return:
"[
{\"ssid\":\"Main-Hall-Staff\",\"password\":\"abc123\"},
{\"ssid\":\"Main-Hall-Guest\",\"password\":\"def456\"}
]"
My question is, how to update both password value and save it?
new_pass1 = 'xyz123'
new_pass2 = 'xyz321'
I have tried code (below) but this will only update first password if i only have one hash_array.
Here is my full code
def encrypt_pass
# get the actual password
parse = JSON.parse(self.value)
get_pass = parse.last['password']
# encrypt the password
crypt = ActiveSupport::MessageEncryptor.new(ENV["SECRET_KEY_BASE"])
encrypted = crypt.encrypt_and_sign(get_pass)
# save the new encrypted password
parse.first['password'] = encrypted
encrypt_pass = parse.to_json
self.value = encrypt_pass
end
Just to be clear, you're trying to update both the Main-Hall-Staff password and the Main-Hall-Guest password (all passwords) from your record to be the encrypted version of themselves? I'm assuming this method is called in a before_save callback of some sort? If you show more code related to the model I can give you more details.
def encrypt_pass
# Changed the name to devises, always use easy to understand naming
# Also rescuing from a JSON parse error, this isnt always recommended
# as it hides other errors that might be unrelated to parsing
devices = JSON.parse(self.value) rescue []
crypt = ActiveSupport::MessageEncryptor.new(ENV["SECRET_KEY_BASE"])
devices.each do |device|
# get the actual password
password = device['password']
# encrypt the password
encrypted_pass = crypt.encrypt_and_sign(password)
# Save the encrypted password
device['password'] = encrypted_pass
end
self.value = devices.to_json
end
Hopefully you have some logic surrounding when this method is called as you dont want to encrypt an already encrypted password.

Extracting data from yml files with ruby

I read some quick tutorials on Yaml or yml file format. I made a yaml document to represent my data. I saw some ruby tutorials which tell you how to extract yaml with ruby. Unfortunately, they just print the whole data or just keys and values. It does not meet my needs. Please help.
yaml file -
dev:
game1:
server1:
url: 'dev-game1-a-srv01.gamer.com'
log-path: '/srv/logs'
server2:
url: 'dev-game1-a-srv02.gamer.com'
log-path: '/srv/logs'
game2:
server1:
url: 'dev-game2-a-srv01.gamer.com'
log-path: '/srv/logs'
server2:
url: 'dev-game2-b-srv02.gamer.com'
log-path: '/srv/logs'
server3:
url: 'dev-game2-b-srv01.gamer.com'
log-path: '/srv/logs'
prod:
etc....
How do I select dev, game2, server 3, url using ruby code ?
Using the code below, I get an exception -
require 'yaml'
def server_info
path = 'C:\Code\demo-srv.yml'
yml = YAML::load(File.open(path))
game2 = yml['dev']['game2']
game2.each{|server|
if server['server3']
puts server['server3']['url']
end
}
end
server_info
error -
server.rb:8:in `[]': can't convert String into Integer (TypeError)
from server.rb:8:in `server_info'
from server.rb:7:in `each'
from server.rb:7:in `server_info'
from server.rb:14
Did you define the yaml-data or are you only the consumer of an existing yaml-file?
If you defined it, I would replace the array of servers with a Hash (see the missing - before the server names):
dev:
game1:
server1:
url: 'dev-game1-a-srv01.gamer.com'
log-path: '/srv/logs'
server2:
url: 'dev-game1-a-srv02.gamer.com'
log-path: '/srv/logs'
game2:
server1:
url: 'dev-game2-a-srv01.gamer.com'
log-path: '/srv/logs'
server2:
url: 'dev-game2-b-srv02.gamer.com'
log-path: '/srv/logs'
server3:
url: 'dev-game2-b-srv01.gamer.com'
log-path: '/srv/logs'
Then you can try yml['dev']['game2']['server3']['url'].
Attention: There are no checks for missing/wrong data. if the entry for game2 would miss, this code will raise an exception.
So, maybe you shoudl do something like
if yml['dev'] and yml['dev'].kind_of?(Hash)
if yml['dev']['game2'] and ....
...
else
puts "No dev-branch defined"
end
Else you can try something like:
def server_info
yml = YAML::load(DATA)
yml['dev']['game2'].each{|server|
if server['server3']
p server['server3']['url']
end
}
end
Attention (for both solutions):
There are no checks for missing/wrong data. The existence of server['server3'] is checked here. For real code, you should also check the existence of the dev and game2 data.
Answer continuation after edit:
The error convert String into Integer is often thrown if you have an array but expect a hash and you try to access an array element with a string.
You can try the following code. There are two changes:
line 8 contains the output of server - you will see it is an array, no hash.
line 9+10: The array is checked and used by its two elements (via #first and #last)
require 'yaml'
def server_info
path = 'C:\Code\demo-srv.yml'
#~ yml = YAML::load(File.open(path))
yml = YAML::load(DATA)
game2 = yml['dev']['game2']
game2.each{|server|
p server #-> you get an array
if server.first == 'server3'
puts server.last['url']
end
}
end
server_info
The file -
dev:
game1:
server1:
url: 'dev-game1-a-srv01.gamer.com'
log-path: '/srv/logs'
server2:
url: 'dev-game1-a-srv02.gamer.com'
log-path: '/srv/logs'
game2:
server1:
url: 'dev-game2-a-srv01.gamer.com'
log-path: '/srv/logs'
server2:
url: 'dev-game2-b-srv02.gamer.com'
log-path: '/srv/logs'
server3:
url: 'dev-game2-b-srv01.gamer.com'
log-path: '/srv/logs'
I don't know why but yml['dev']['game2']=>
[{"server1"=>{"url"=>"dev-game2-a-srv01.gamer.com", "log-path"=>"/srv/logs"}},
{"server2"=>{"url"=>"dev-game2-b-srv02.gamer.com", "log-path"=>"/srv/logs"}},
{"server3"=>{"url"=>"dev-game2-b-srv01.gamer.com", "log-path"=>"/srv/logs"}}]
So you have to use find on this Array to have the key.
require 'yaml'
# require 'pry'
def server3_url
yml = YAML::load(File.read('yaml.yml'))
# binding.pry
begin
yml['dev']['game2'].find{|x| x['server3']}['server3']['url']
rescue
end
end
puts server3_url
server3_url will return nil if it doesn't find a key
Changing your code the following should fix the issue.
# ...
game2.each{ |server, data|
if server == 'server3'
puts data['url']
end
}
# ...
You are encountering the type error because the yielded value server is an array, not a hash. This is happening because you are calling each on the game2 variable, which is a hash, and only yielding to a single variable.
Examples
hash = { one: 1, two: 2, three: 3 }
Yielding to a single variable
When Hash#each is called with only one variable, the current key and value are assigned to that variable as an array in the order [key, value]
hash.each do |number|
puts number.inspect
end
# Prints
# [:one, 1]
# [:two, 2]
# [:three, 3]
Yielding to multiple variables
When Hash#each is called with two variables, the current key will be assigned to the first variable, and the current value will be assigned to the second.
hash.each do |key, value|
puts "Key: #{key}; Value: #{value}"
end
# Prints:
# Key: one; Value: 1
# Key: two; Value: 2
# Key: three; Value: 3

Chef template loop: can't convert Chef::Node::immutableMash into String

I've got a Vagrant setup in which I'm trying to use Chef-solo to generate an conf file which loops though defined variables to pass to the application. Everything is working except the loop and I'm not familiar enough with Ruby/Chef to spot the error.
I'm going to lay out the whole chain of events in case there is something along the way that is the problem, but the first portions of this process seem to work fine.
A config file is written in yaml and includes env variable definitions to be passed:
...
variables:
- DEBUG: 2
...
The config file is read in by the Vagrantfile into a ruby hash and used to create the Chef json nodes:
...
settings = YAML::load(File.read("config.yaml"))
# Provision The Virtual Machine Using Chef
config.vm.provision "chef_solo" do |chef|
chef.json = {
"mysql" => {"server_root_password" => "secret"},
"postgresql" => {"password" => {"postgres" => "secret"}},
"nginx" => {"pid" => "/run/nginx.pid"},
"php-fpm" => {"pid" => "/run/php5-fpm.pid"},
"databases" => settings["databases"] || [],
"sites" => settings["sites"] || [],
"variables" => settings["variables"] || []
}
...
A bunch of chef cookbooks are run (apt, php, nginx, mysql etc) and finally my custom cookbook which is whats giving me grief. The portion of the cookbook responsible for creating a the conf file is shown here:
# Configure All Of The Server Environment Variables
template "#{node['php-fpm']['pool_conf_dir']}/vars.conf" do
source "vars.erb"
owner "root"
group "root"
mode 0644
variables(
:vars => node['variables']
)
notifies :restart, "service[php-fpm]"
end
And the vars.erb is just a one-liner
<%= #vars.each {|key, value| puts "env[" + key + " = " + value } %>
So, when I run all this chef spits out an error about not being able to convert a hash to a string.
can't convert Chef::Node::immutableMash into String
So for some reason this is coming across as an immutableMash and the value of key ends up being the hash [{"DEBUG"=>2}] and value ends up a nil object, but I'm not sure why or how to correct it.
The hash is ending up as the value of key in your example because the YAML file declares DEBUG: 2 as a list member of variables. This translates to variables being an array with a single hash member.
Try changing the template code to this:
<%= #vars[0].each {|key, value| puts "env[" + key + " = " + value } %>
Or try changing the YAML to this and not changing the template code:
variables:
DEBUG: 2
Either change will get your template loop iterating over the hash that you are expecting.

Can Ruby's YAML module be used to embed comments?

The to_yaml method produces nice YAML output, but I would like to include comment lines before some of the elements. Is there a way to do so?
For example, I would like to produce:
# hostname or IP address of client
client: host4.example.com
# hostname or IP address of server
server: 192.168.222.222
From something similar to:
{
:client => 'host4.example.com',
:server => '192.168.222.222',
}.to_yaml
... but am not sure if the YAML module even has a way to accomplish.
UPDATE: I ended up not using the solution which used regexes to insert the comments, since it required the separation of the data from the comments. The easiest and most understandable solution for me is:
require 'yaml'
source = <<SOURCE
# hostname or IP address of client
client: host4.example.com
# hostname or IP address of server
server: 192.168.222.222
SOURCE
conf = YAML::load(source)
puts source
The benefit to me is that nothing is repeated (for example, 'client:' is only specified once), the data and comments are together, the source can be outputted as YAML, and the data structure (available in conf) is available for use.
You can do a string replace on all the insertions:
require 'yaml'
source = {
:client => 'host4.example.com',
:server => '192.168.222.222',
}.to_yaml
substitution_list = {
/:client:/ => "# hostname or IP address of client\n:client:",
/:server:/ => "# hostname or IP address of server\n:server:"
}
substitution_list.each do |pattern, replacement|
source.gsub!(pattern, replacement)
end
puts source
output:
---
# hostname or IP address of client
:client: host4.example.com
# hostname or IP address of server
:server: 192.168.222.222
Something like this:
my_hash = {a: 444}
y=YAML::Stream.new()
y.add(my_hash)
y.emit("# this is a comment")
Of course, you will need to walk the input hash yourself and either add() or emit() as needed.
You could look at the source of the to_yaml method for a quick start.
This isn't perfect (no mid-Array support, for example), but it works for my needs.
def commentify_yaml(db)
ret = []
db.to_yaml(line_width: -1).each_line do |l|
if l.match(/^\s*:c\d+:/)
l = l.sub(/:c(\d+)?:/, '#').
sub(/(^\s*# )["']/, '\1').
sub(/["']\s*$/, '').
gsub(/''(\S+?)''/, "'\\1'").
gsub(/(\S)''/, "\\1'")
end
ret << l.chomp
end
ret * "\n"
end
Example usage.
commentify_yaml(
{
c1: 'Comment line 1',
c2: 'Comment line 2',
'hash_1' => {
c1: 'Foo',
c2: 'Bar',
'key_1' => "Hello!",
},
'baz' => qux,
c3: 'Keep up-numbering the comments in the same hash',
'array_1' => [
1,
2,
3
]
}
)
==>
# Comment line 1
# Comment line 2
hash_1:
# Foo
# Bar
key_1: "Hello!"
baz: "Value of qux"
# Keep up-numbering the comments in the same hash
array_1:
- 1
- 2
- 3
(Note: Syck does not indent arrays they way it arguably should.)

Read and write YAML files without destroying anchors and aliases?

I need to open a YAML file with aliases used inside it:
defaults: &defaults
foo: bar
zip: button
node:
<<: *defaults
foo: other
This obviously expands out to an equivalent YAML document of:
defaults:
foo: bar
zip: button
node:
foo: other
zip: button
Which YAML::load reads it as.
I need to set new keys in this YAML document and then write it back out to disk, preserving the original structure as much as possible.
I have looked at YAML::Store, but this completely destroys the aliases and anchors.
Is there anything available that could something along the lines of:
thing = Thing.load("config.yml")
thing[:node][:foo] = "yet another"
Saving the document back as:
defaults: &defaults
foo: bar
zip: button
node:
<<: *defaults
foo: yet another
?
I opted to use YAML for this due to the fact it handles this aliasing well, but writing YAML that contains aliases appears to be a bit of a bleak-looking playing field in reality.
The use of << to indicate an aliased mapping should be merged in to the current mapping isn’t part of the core Yaml spec, but it is part of the tag repository.
The current Yaml library provided by Ruby – Psych – provides the dump and load methods which allow easy serialization and deserialization of Ruby objects and use the various implicit type conversion in the tag repository including << to merge hashes. It also provides tools to do more low level Yaml processing if you need it. Unfortunately it doesn’t easily allow selectively disabling or enabling specific parts of the tag repository – it’s an all or nothing affair. In particular the handling of << is pretty baked in to the handling of hashes.
One way to achieve what you want is to provide your own subclass of Psych’s ToRuby class and override this method, so that it just treats mapping keys of << as literals. This involves overriding a private method in Psych, so you need to be a little careful:
require 'psych'
class ToRubyNoMerge < Psych::Visitors::ToRuby
def revive_hash hash, o
#st[o.anchor] = hash if o.anchor
o.children.each_slice(2) { |k,v|
key = accept(k)
hash[key] = accept(v)
}
hash
end
end
You would then use it like this:
tree = Psych.parse your_data
data = ToRubyNoMerge.new.accept tree
With the Yaml from your example, data would then look something like
{"defaults"=>{"foo"=>"bar", "zip"=>"button"},
"node"=>{"<<"=>{"foo"=>"bar", "zip"=>"button"}, "foo"=>"other"}}
Note the << as a literal key. Also the hash under the data["defaults"] key is the same hash as the one under the data["node"]["<<"] key, i.e. they have the same object_id. You can now manipulate the data as you want, and when you write it out as Yaml the anchors and aliases will still be in place, although the anchor names will have changed:
data['node']['foo'] = "yet another"
puts Yaml.dump data
produces (Psych uses the object_id of the hash to ensure unique anchor names (the current version of Psych now uses sequential numbers rather than object_id)):
---
defaults: &2151922820
foo: bar
zip: button
node:
<<: *2151922820
foo: yet another
If you want to have control over the anchor names, you can provide your own Psych::Visitors::Emitter. Here’s a simple example based on your example and assuming there’s only the one anchor:
class MyEmitter < Psych::Visitors::Emitter
def visit_Psych_Nodes_Mapping o
o.anchor = 'defaults' if o.anchor
super
end
def visit_Psych_Nodes_Alias o
o.anchor = 'defaults' if o.anchor
super
end
end
When used with the modified data hash from above:
#create an AST based on the Ruby data structure
builder = Psych::Visitors::YAMLTree.new
builder << data
ast = builder.tree
# write out the tree using the custom emitter
MyEmitter.new($stdout).accept ast
the output is:
---
defaults: &defaults
foo: bar
zip: button
node:
<<: *defaults
foo: yet another
(Update: another question asked how to do this with more than one anchor, where I came up with a possibly better way to keep anchor names when serializing.)
YAML has aliases and they can round-trip, but you disable it by hash merging. << as a mapping key seems a non-standard extension to YAML (both in 1.8's syck and 1.9's psych).
require 'rubygems'
require 'yaml'
yaml = <<EOS
defaults: &defaults
foo: bar
zip: button
node: *defaults
EOS
data = YAML.load yaml
print data.to_yaml
prints
---
defaults: &id001
zip: button
foo: bar
node: *id001
but the << in your data merges the aliased hash into a new one which is no longer an alias.
Have you try Psych ? Another question with psych here.
I'm generating my CircleCI config file with a Ruby script and ERB templates. My script parses and regenerates the YAML, so I wanted to preserve all the anchors. The anchors in my config all have the same name as the key that defines them, e.g.
docker_images:
docker_auth: &docker_auth
username: '$DOCKERHUB_USERNAME'
password: '$DOCKERHUB_TOKEN'
cimg_base_image: &cimg_base_image
image: cimg/base:2022.09
auth: *docker_auth
jobs:
tests:
docker:
- *cimg_ruby_image
So I was able to solve this with regular expressions on the generated YAML string. It wrote a #restore_yaml_anchors method that converts &1 and *1 back into &docker_auth and *docker_auth.
# Ruby 3.1.2
require 'rubygems'
require 'yaml'
yaml = <<EOS
docker_images:
docker_auth: &docker_auth
username: '$DOCKERHUB_USERNAME'
password: '$DOCKERHUB_TOKEN'
cimg_base_image: &cimg_base_image
image: cimg/base:2022.09
auth: *docker_auth
jobs:
tests:
docker:
- *cimg_base_image
EOS
data = YAML.load yaml, aliases: true # needed for Ruby 3.x
def restore_yaml_anchors(yaml)
yaml.scan(/([A-Z0-9a-z_]+|<<): &(\d+)/).each do |anchor_name, anchor_id|
yaml.gsub!(/([:-]) (\*|&)#{anchor_id}/, "\\1 \\2#{anchor_name}")
end
yaml
end
puts [
"Original #to_yaml:",
data.to_yaml,
"-----------------------", '',
"With restored anchors:",
restore_yaml_anchors(data.to_yaml)
].join("\n")
Output:
Original #to_yaml:
---
docker_images:
docker_auth: &1
username: "$DOCKERHUB_USERNAME"
password: "$DOCKERHUB_TOKEN"
cimg_base_image: &2
image: cimg/base:2022.09
auth: *1
jobs:
tests:
docker:
- *2
-----------------------
With restored anchors:
---
docker_images:
docker_auth: &docker_auth
username: "$DOCKERHUB_USERNAME"
password: "$DOCKERHUB_TOKEN"
cimg_base_image: &cimg_base_image
image: cimg/base:2022.09
auth: *docker_auth
jobs:
tests:
docker:
- *cimg_base_image
It's working well for my CI config, but you may need to update it to handle some other cases in your own YAML.

Resources