I have YAML file with inheritance and I want to add or edit a key programatically.
I load the YAML into hash using YAML.load method but when I save the hash back using YAML.dump I lose all the inheritance info.
Is there a way to edit the YAML in Ruby without losing the inheritance info?
YAML example:
main:
prod: &prod
key1: true
key2: 50
key3: "abc"
prod_v_3_5: &prod_v_3_5
<<: *prod
key2: 100
prod_v_3_6: &prod_v_3_6
<<: *prod_v_3_5
key2: 150
Code example:
config = Api.get(id)
yaml = YAML.load(config)
yaml["main"][section].store(key, value)
config = YAML.dump(yaml)
Api.set(id, config)
As far as I know (I also use that option to import and override) the YAML source is read and processed and then the hash elements are exposed and not linked internally. So the mechanism is copy paste override and not linking and overload.
I guess you have to modify the YAML source by opening the file and modify its content if you won't destroy your source.
To add new section:
config = Api.get(id)
config = "#{config}\n \n #{section}: &#{section}\n <<: *#{parent_section}"
To add new value:
config = Api.get(id)
matches = /^(.+)(\n #{section}:\s*&#{section}\s*\n )(<<:[^\n]+)?(.*)$/m.match(config)
config = "#{matches[1]}#{matches[2]}#{matches[3]}\n #{key}: #{value}#{matches[4]}\n"
Related
I'm trying to understand some yaml syntax related to using hydra in a machine learning approach. So given the following, extracted from the original github repo:
datamodule:
_target_: cdvae.pl_data.datamodule.CrystDataModule
datasets:
train:
_target_: cdvae.pl_data.dataset.CrystDataset
name: Formation energy train
path: ${data.root_path}/train.csv
prop: ${data.prop}
niggli: ${data.niggli}
primitive: ${data.primitive}
graph_method: ${data.graph_method}
lattice_scale_method: ${data.lattice_scale_method}
preprocess_workers: ${data.preprocess_workers}
I don't understand what the syntax ${} stands for in this context. Is that some kind of text formatting? Or is it calling some specific module? Could anybody provide some examples?
EDIT:
Ok, as pointed by #flyx, it seems that the syntax ${} refers to hydra which is indeed part of the project I'm analyzing.
The string foo: ${bar} in yaml corresponds roughly to the dict {"foo": "${bar}"} in python. This is to say, the dollar-bracket notation ${} is not given special meaning by the yaml grammar.
OmegaConf, which is the backend used by Hydra, does give special meaning to the dollar-bracket syntax; it is used for "variable interpolation." See the OmegaConf docs on variable interpolation.
To summarize briefly, the idea of variable interpolation is that the string "${bar}" is a "pointer" to the value with key bar that would appear elsewhere in your OmegaConf config object.
Here's a short demo of variable interpolations in action:
from omegaconf import OmegaConf
yaml_data = """
foo: ${bar}
bar: baz
"""
config = OmegaConf.create(yaml_data)
assert config.bar == "baz"
assert config.foo == "baz" # the ${bar} interpolation points to the "baz" value
I want to read a list of objects from a yaml file via terraform code and map it to a local variable. Also i need search an object with a key and get the values from a yaml file. Can anyone suggest suitable solution?
my yaml file looks like below. Here use will be the primary key
list_details:
some_list:
- use: a
path: somepath
description : "some description"
- use: b
path: somepath2
description : "some description 2"
I have loaded the yaml file in my variable section in Terraform like this
locals {
list = yamldecode(file("${path.module}/mylist.yaml"))
}
Now the problem is how I can get one object with its values by passing the "use" value to the list?
"
Assuming that use values are unique, you can re-organize your list into map:
locals {
list_as_map = {for val in local.list["list_details"]["some_list"]:
val["use"] => val["path"]}
}
which gives list_as_map as:
"a" = "somepath"
"b" = "somepath2"
then you access the path based on a value of use:
path_for_a = local.list_as_map["a"]
Update
If you want to keep description, its better to do:
list_as_map = {for val in local.list["list_details"]["some_list"]:
val["use"] => {
path = val["path"]
description = val["description"]
}
}
then you access the path or description as:
local.list_as_map["a"].path
local.list_as_map["a"].description
I have a yaml file that includes the following:
:common
:substitue
:foo: fee
I read this data like:
data = YAML.load(erb_data[File.basename(__FILE__, '.*')].result(binding))
common = data[:common]
def substitute_if_needed(original_value)
mapping = common.dig(:substitue, original_value)
if mapping.nil? ? original_value : mapping
end
Unfortunately, this doesn't do the substitution that I want. I want to call substitute_if_needed('foo') and get 'fee' back. I also want to call substitute_if_needed('bar') and get 'bar' back.
How can I do this?
There are several problems in your code:
YAML example looks broken. The proper one should looks like:
common:
substitute:
foo: fee
You're trying to fetch common key in common = data[:common] using a symbol as a key, but it should be a string (data["common"]). Also, I'd say it's a bad idea to spilt fetching logic into two pieces - first extract "common" outside of substitute_when_needed and then dig into it inside.
if statement is broken. It should be either proper if or proper ternary operator.
Fixing all this gives us something like (I've just replaced a file with StringIO for convenience - to make the snippet executable as is):
yaml = StringIO.new(<<~DATA)
common:
substitute:
foo: fee
DATA
def substitute_if_needed(data, original_value)
mapping = data.dig("common", "substitute", original_value)
mapping.nil? ? original_value : mapping
end
data = YAML.load(yaml)
substitute_if_needed(data, "foo") # => "fee"
substitute_if_needed(data, "bar") # => "bar"
I extract a JsonArray containing a list of string, and I want to validate by a regex each string inside this object.
Problem, I don't seem to find any answers on the Taurus' website.
Do you know how I can do it ?
Example below:
# Verification of value inside the JsonArray
extract-jsonpath:
names: $.names
- foreach: name in names
do:
- jsonpath: ${name} # if this JSONPATH is not found, assert will fail
validate: true # validate against an expected value
expected-value: "\\w" # value we're expecting to validate. [default: false]
regexp: true # if the value is regular expression, default: true
expect-null: false # expected value is null
invert: false # invert condition
I don't think it is possible with Taurus YAML syntax as:
foreach keyword generates a normal JMeter ForEach Controller
These jsonpath, validate, etc. are applied to a Sampler by default, they will not work if you add them just as children of the ForEach Controller
Assuming above points I would suggest adding a JSR223 PostProcesssor to perform all the checks. In Taurus it is being done via JSR223 Blocks like:
- url: https://api.example.com/v1/media/search
extract-jsonpath:
names: $.names
jsr223:'1.upto(vars.get("names_matchNr") as int,{if (vars.get("names_$it").matches("\\w+")) {prev.setSuccessful(false)}})'
See The Groovy Templates Cheat Sheet for JMeter article to get more ideas with regards to what can be done using Groovy scripts.
I'm just starting to learn YAML, and I'm not really finding a best practice for something that I'm trying to accomplish. Basically, I have an array of objects in my YAML file, and for production, I'd like to add 1 more entry to this array. So I basically want something like this (it's pseudo code because I know it's not valid YAML):
development:
array: &ARRAY
- name: item1
value: value1
- name: item2
value: value2
production:
<<: *ARRAY
array:
- name: item3
value: value3
Currently, I'm parsing my YAML files with Ruby, so I decided to handle this logic in Ruby. I'm doing something like this:
yaml_contents = YAML::load(yaml_string)
prod_values = yaml_contents['production']
prod_values['array'].push({:name => 'item3', :value => 'value3'})
However, that can make my loading script very hairy. Is there a better way of designing this?
I believe this question is related.
The << syntax is for merging maps (i.e. Hashes), not sequences. You could do something like this:
development: &ARRAY
- name: item1
value: value1
- name: item2
value: value2
production:
- *ARRAY
- name: item3
value: value3
When you load this the production array will have a nested array, so you would need to use flatten:
yaml_contents = YAML::load(yaml_string)
prod_values = yaml_contents['production'].flatten
If your actual data could involve nested hashes and you only want to flatten any arrays that appear as aliases in the Yaml you could write your own Psych visitor (probably a sub class of Psych::Visitors::ToRuby) and merge them in as you create the object graph, but I suspect simply calling flatten will be enough in this case.