I have a yaml file which is the start of a Neoload performance test script-as-code;
name: TestFile
servers:
- name: myserver
host: myhost.world.com
scheme: https
I want Jenkins to build the yaml file based on variables from previous steps/pipelines. I have seen writeYaml can do what I want, and it seems fairly easy to use. This is what I have so far;
script{
def map = ['name':'TestFile]
map.put('name','myserver')
writeYaml, file: filename, data: map
}
The def works fine, but I can't quite get a grasp on the syntax to add the 'servers' section to the map object. With what I have above, I just end up with name: myserver in the file.
I did try;
map.put("servers", new String[] {"name","myserver"})
But it doesn't compile (unexpected token "name"), I'm assuming because I'm trying to use the wrong language in a scripted pipeline.
I also appreciate that I could just do the equivalent of 'write line' to the file to generate a YAML, but this file gets a lot more complicated further down so I'd prefer to use this object based approach.
Given an existing Groovy Map with a key-value pair of ['name': 'TestFile], you can add keys and values to this Map in different syntax.
To generate the YAML that you want, you would need an additional key of servers with a one element array (- signifies an Array type in YAML). The member element of that array should be another Map with all of your key value pairs.
You would be looking to append ['servers': [['name': 'myserver', 'host': 'myhost.world.com', 'scheme': 'https']]] to your map. There are three [] in that syntax; the first and third signify Map and the second signifies List. servers is the key in your first Map, and the value is the List. The List's only element is the map of key value pairs. The three key value pairs constitute the map in that List element.
Given two common syntax for appending key value pairs to a Map, you could do either:
map['servers'] = [['name': 'myserver', 'host': 'myhost.world.com', 'scheme': 'https']]
or
map.servers = [['name': 'myserver', 'host': 'myhost.world.com', 'scheme': 'https']]
to achieve the desired result.
Related
I have a feeling this isn't possible, but I have a snippet of YAML that looks like the following:
.map_values: &my_map
a: 'D'
b: 'E'
a: 'F'
section:
stage: *my_map['b']
I would like stage to have the value of E.
Is this possible within YAML? I've tried just about every incarnation of substitution I can think of.
Since there is a duplicate key in your mapping, which is not allowed
in YAML 1.2 (and should at least throw a warning in YAML 1.1) this is
not going to work, but even if you correct that, you can't do that
with just anchors and aliases.
The only substitution like replacement that is available in YAML is the "Merge Key Language-Independent Type". That is indirectly referenced in the YAML spec, and not included in it, but available in most parsers.
The only thing that allows it to do is "update" a mapping with key value pairs of one or more other mappings, if the key doesn't already exist in the mapping. You use the special key << for that, which takes an alias, or a list of aliases.
There is no facility, specified in the YAML specification, to dereference particular keys.
There are some systems that use templates that generate YAML, but there are two main problems to apply these here:
the template languages themselves often are clashing with the indicators in the YAML syntax,
making the template not valid YAML
even if the template could be loaded as valid YAML, and the values extracted that are needed to
update other parts of the template, you would need to parse the input twice (once to get the
values to update the template, then to parse the updated template). Given the potential
complexity of YAML and the relative slow speed of its parsers, this can be prohibitive
What you can do is create some tag (e.g. !lookup) and have its constructor interpret that node.
Since the node has to be valid YAML again you have to decide on whether to use a sequence or a mapping.
You'll have to include some special syntax for the values in both cases, and also for the key
(like the << used in merges) in the case of mappings.
In the examples I left out the spurious single quotes, depending on
your real values you might of course need them.
Example using sequence :
.map_values: &my_map
a: D
b: E
c: F
section: !Lookup
- *my_map
- stage: <b>
Example using mapping:
.map_values: &my_map
a: D
b: E
c: F
section: !Lookup
<<: *my_map
stage: <b>
Both can be made to construct the data on the fly (i.e. no past
loading processing of your data structure necessary). E.g. using Python and
the sequence "style" in input.yaml:
import sys
import ruamel.yaml
from pathlib import Path
input = Path('input.yaml')
yaml = ruamel.yaml.YAML(typ='safe')
yaml.default_flow_style = False
#yaml.register_class
class Lookup:
#classmethod
def from_yaml(cls, constructor, node):
"""
this expects a two entry sequence, in which the first is a mapping X, typically using
an alias
the second entry should be an mapping, for which the values which have the form <key>
are looked up in X
non-existing keys will throw an error during loading.
"""
X, res = constructor.construct_sequence(node, deep=True)
yield res
for key, value in res.items():
try:
if value.startswith('<') and value.endswith('>'):
res[key] = X[value[1:-1]]
except AttributeError:
pass
return res
data = yaml.load(input)
yaml.dump(data, sys.stdout)
which gives:
.map_values:
a: D
b: E
c: F
section:
stage: E
There are a few things to note:
using <...> is arbitrary, you don't need a both beginning and an
end marker. I do recommend using some character(s) that has no
special meaning in YAML, so you don't need to quote your values. You can e.g. use some
well recognisable unicode point, but they tend to be a pain to type in an editor.
when from_yaml is called, the anchor is not yet fully constructed. So X is an empty dict
that gets filled later on. The constructed with yield implements a two step process: we first
give back res "as-is" back to the constructor, then later update it. The constructor stage of
the loader knows how to handle this automatically when it gets the generator instead a "normal" value.
the try .. except is there to handle mapping values that are not strings (i.e. numbers, dates, boolean).
you can do substitutions in keys as well, just make sure you delete the old key
Since tags are standard YAML, the above should be doable one way or another in any
YAML parser, independent of the language.
I ran into an odd issue when trying to modify a chef recipe. I have an attribute that contains a large hash of hashes. For each of those sub-hashes, I wanted to add a new key/value to a 'tags' hash within. In my recipe, I create a 'tags' local variable for each of those large hashes and assign the tags hash to that local variable.
I wanted to add a modification to the tags hash, but the modification had to be done at compile time since the value was dependent on a value stored in an input json. My first attempt was to do this:
tags = node['attribute']['tags']
tags['new_key'] = json_value
However, this resulted in a spec error that indicated I should use node.default, or the equivalent attribute assignment function. So I tried that:
tags = node['attribute']['tags']
node.normal['attribute']['tags']['new_key'] = json_value
While I did not have a spec error, the new key/value was not sticking.
At this point I reached my "throw stuff at a wall" phase and used the hash.merge function, which I used to think was functionally identical to hash['new_key'] for a single key/value pair addition:
tags = node['attribute']['tags']
tags.merge({ 'new_key' => 'json_value' })
This ultimately worked, but I do not understand why. What functional difference is there between the two methods that causes one to be seen as a modification of the original chef attribute, but not the other?
The issue is you can't use node['foo'] like that. That accesses the merged view of all attribute levels. If you then want to set things, it wouldn't know where to put them. So you need to lead off by tell it where to put the data:
tags = node.normal['attribute']['tags']
tags['new_key'] = json_value
Or just:
node.normal['attribute']['tags']['new_key'] = json_value
Beware of setting things at the normal level though, it is not reset at the start of each run which is probably what you want here, but it does mean that even if you remove the recipe code doing the set, the value will still be in place on any node that already ran it. If you want to actually remove things, you have to do it explicitly.
Say I have a YML file original.yml with an array of objects
array_in_yml:
- start: 1
- middle: 2
- end: 3
I included it in modified.yml
!include "original.yml"
array_in_yml: []
I am expecting this array to be empty when I load the modified.yml, but it seems to have 3 values as original.yml. How do I force/override the array to be empty?
The discussion about !include seems to lead a bit away from the actual question. Let's assume that in some unknown way, the !include line gets replaced with the content in original.yml. We would have:
array_in_yml:
- start: 1
- middle: 2
- end: 3
array_in_yml: []
This is not valid YAML, since every key in a dictionary must be unique, but you use the key array_in_yml twice. Your YAML processor might ignore this and simply assign the first value (which is a sequence of three items) to the key array_in_yml.
Now the important part: There is no way in YAML to modify previously given values. You cannot override a value given previously with a different one. What you want to do is outside the YAML spec and you would need some merging tool that does such replacements for you.
I am making my first bukkit plugin. I would like to programmatically create a YAML file that represents a HashMap. How can I set and get this data structure?
The HashMap parameters look like <Signature, Location>, where Signature is my class that stores 4 integers, and Location is org.bukkit.Location
I think I would like the YAML file to look like this, but I am not sure if this structure is best:
MyPlugin:
ListOfData:
- signature: [1,2,3,4] # this is a unique set of 4 integers
location: [122,64,254] # non-unique set of 3 integers
- signature: [4,2,1,2]
location: [91,62,101]
- signature: [3,3,1,3]
location: [190,64,321]
Signature can be modified as necessary, and I can create a wrapper for Location if necessary.
Thanks!
This is a suggested solution. I don't know if it is the best way...:)
You may want to consider this as your yaml structure:
MyPlugin:
ListOfData:
'[1,2,3,4]': '[122,64,254]'
'[4,2,1,2]': '[91,62,101]'
'[3,3,1,3]': '[190,64,321]'
anothersignature:anotherlocation
...
This will let you read the "ListOfData" in using the normal technique for reading hash map from a YAMLConfiguration (see below).
You'll have to treat the incoming information from the file as a HashMap of <String, String> and do any translation (e.g. turn 122,64,254 into a location) you need from there.
For reading a HashMap:
this.getConfig().getConfigurationSection("path.to.map").getValues(false)
For writing a HashMap (saveConfig() will still need to be called to write to disk):
this.getConfig().createSection("path.to.map", MyMap)
There's some details and subtleties here, its worth reading these carefully (same page, but different non-contiguous sections):
http://wiki.bukkit.org/Configuration_API_Reference#HashMaps
http://wiki.bukkit.org/Configuration_API_Reference#HashMaps_2
I'm using Jammit to package assets up for a Rails application and I have a few asset files that I'd like to be included in each of a few groups. For example, I'd like Sammy and its plugins to be in both my mobile and screen JS packages.
I've tried this:
sammy: &SAMMY
- public/javascripts/vendor/sammy.js
- public/javascripts/vendor/sammy*.js
mobile:
<<: *SAMMY
- public/javascripts/something_else.js
and this:
mobile:
- *SAMMY
but both put the Sammy JS files in a nested Array, which Jammit can't understand. Is there a syntax for including the elements of an Array directly in another Array?
NB: I realize that in this case there are only two elements in the SAMMY Array, so it wouldn't be too bad to give each an alias and reference both in each package. That's fine for this case, but quickly gets unmaintainable when there are five or ten elements that have a specific load order.
Closest solution I know of is this one:
sammy:
- &SAMMY1
public/javascripts/vendor/sammy.js
- &SAMMY2
public/javascripts/vendor/sammy*.js
mobile:
- *SAMMY1
- *SAMMY2
- public/javascripts/something_else.js
Alternatively, as already suggested, flatten the nested lists in a code snippet.
Note: according to yaml-online-parser, your first suggestion is not a valid use of << (used to merge keys from two dictionaries. The anchor then has to point to another dictionary I believe.
If you want mobile to be equal to sammy, you can just do:
mobile: *SAMMY
However if you want mobile to contain other elements in addition to those in sammy, there's no way to do that in YAML to the best of my knowledge.
Your example is valid YAML (a convenient place to check is YPaste), but it's not defined what the merge does. Per the spec, a merge key can have a value:
A mapping, in which case it's merged into the parent mapping.
A sequence of mappings, in which case each is merged, one-by-one, into the parent mapping.
There's no way of merging sequences on YAML level.
You can, however, do this in code. Using the YAML from your second idea:
mobile:
- *SAMMY
you'll get nested sequences - so flatten them! Assuming you have a mapping of such nested sequences:
data = YAML::load(File.open('test.yaml'))
data.each_pair { |key, value| value.flatten! }
(Of course, if you have a more complicated YAML file, and you don't want every sequence flattened (or they're not all sequences), you'll have to do some filtering.)
This solution is for Symfony/PHP only (considerations for other languages, see below)
Note about array keys from the PHP array manual page:
Strings containing valid decimal ints, unless the number is preceded by a + sign, will be cast to the int type. E.g. the key "8" will actually be stored under 8. [...]
This means that if you actually index your anchor array with integer keys, you can simply add new keys by continuing the initial list. So your solution would look like this:
sammy: &SAMMY
1: public/javascripts/vendor/sammy.js
2: public/javascripts/vendor/sammy*.js
mobile:
<<: *SAMMY
3: public/javascripts/something_else.js
You can even overwrite keys and still add new ones:
laptop:
<<: *SAMMY
1: public/javascripts/sammy_laptop.js
3: public/javascripts/something_else.js
In both cases the end result is a perfectly valid indexed array, just like before.
Other programming languages
Depending on your YAML implementation and how you iterate over your array, this could conceivably also be used in other programming languages. Though with a caveat.
For instance, in JS you can access numerical string keys by their integer value as well:
const sammy = {"1": "public/javascripts/vendor/sammy.js"}
sammy["1"]; // "public/javascripts/vendor/sammy.js"
sammy[1]; // "public/javascripts/vendor/sammy.js"
But you'd need to keep in mind, that your initial array is now an object, and that you would need to iterate over it accordingly, e.g.:
Object.keys(sammy).forEach(key => console.log(sammy[key]))
As it has been suggested, when you need to flatten a list, at least in ruby, it is trivial to add a "!flatten" type specifier to mobile and implement a class that extends Array, adds the yaml_tag and flattens the coder seq on init_with.