Golang yaml unmarshal structure - go

I would need some help to handle configuration files in golang.
I renamed every fields and values for privacy (just saying)
I have this yaml configuration file :
Label1:
field1: value1
field2: value2
field3: value3
field4: value4
Label2:
field1: value1
field2: value2
field3: value3
field4: value4
Label3:
field1: value1
field2: value2
field3: value3
field4: value4
I'm using https://github.com/kylelemons/go-gypsy/tree/master/yaml to manually decode this yaml file into the structure below (manually walking the list of nodes). I don't have control over how many "Label" items will be on that file so I need a generic way of parsing it (and it is generic using the library mentioned above)
type Obj struct {
Field1 string `yaml:"field1"`
Field2 string `yaml:"field2"`
Field3 string `yaml:"field3"`
Field4 bool `yaml:"field4"`
}
type ObjConfig struct {
Objs map[string]Obj
}
I have several other configuration files, much simpler, which I'm able to decode using annotations such as :
type conf1 struct {
Field1 int `yaml:"field1"`
Field2 string `yaml:"field2"`
}
I'm using the automatic yaml Unmarshal (through Viper framework) for these and it's working fine.
It wasn't a big deal mixing these 2 ways of handling configuration files until now. But now I need to retrieve the file from a spring config server that I did set up instead of having the files locally. I'm using this library to do so :
https://github.com/Piszmog/cloudconfigclient
It's working well for structures annotated with the yaml:"fieldX", I'm able to deserialize it directly.
But for the first structure of course it's not working and I need a workaround. The library (cloudconfigclient) is using the classic yaml decoder so I can't write my own custom decoder. Looking at the library code it uses this code to deserialize the configuration file (with resp.Body being the yaml configuration file content) :
var dest interface{}
yaml.NewDecoder(resp.Body).Decode(dest)
I'm able to change the configuraton file structure though, as long as I can get the same data in it. Maybe I could use something like
Labels:
- Label1:
field1: value1
field2:value2
....
- Label2:
field1:value1
....
and use the yaml map handling from unmarshal ?
I'm obviously not experienced with golang, I'd love to get some help here.
To sum up, the question is how to decode the first yaml file using yaml.Unmarshal with the classic decoder (by either specific annotations that I don't know of, or changing the yaml structure itself).

I got it working by mapping it to a generic map[string]interface{} type
var configResponse map[string]interface{}
err = configClient.GetFileFromBranch(BRANCH, CONF_DIR, CONF_FILE, &configResponse )
and then handling the cast myself into the structure
type ObjConfig struct {
Objs map[string]Obj
}
Not the most easy to read nor resilient to error solution but it's working

Related

regex as protobuf message field name?

can we define regular expression in protobuf field name? I send the request as list of dictinary in client.py file
"cur_cur_bin" : [{"cur_cur_bin1_bin3_bin1" : 4,"cur_cur_bin3_bin5_bin8" : 6} ]
I defined .proto file like,
int32 cur_cur_bin1_bin3_bin1 = 1;
}
message Message{
repeated cur_cur_BIN cur_cur_bin = 1;
}```
any one can explain how to define this type of field in .proto file dynamically. because
(bin1) having some range like (1 - [1-8]) same for (bin3) like (3 -[8-11]) like this.
No, as far as I know there is no mechanism to generate field names automatically or dynamically in protoc.
You can create messages dynamically through the python-protobuf reflection API, but that probably doesn't make sense for your usecase.
Instead define a reasonable .proto format, and then write some custom Python code to convert your JSON data to that.

Can't evaluate field <field> in type string

I'm used text/template in golang package to parse text with environment in json string.
Example: https://play.golang.org/p/uARnrOyG4Th
But I have an error: 2009/11/10 23:00:00 Execute: template: Person template:1:19: executing "Person template" at <.id>: can't evaluate field id in type string
Please help me with problem. I couldn't find the error based on the library's documents.
template can't directly apply to string, you should create a struct then unmarshal your JSON string to the struct instance. try the code
person := map[string]string{"id": "12345", "token": "NSJACSCS"}
output:
momo://pay?appid=12345&token=NSJACSCS
Program exited.

Parse yaml with keys provide errors

I’ve the following yaml which I need to parse,
I’ve tried with the following
Build-t:
before: test1
- value : dddd
- bbb: zzzz
after: test2
- value: bbb
- aaa: aaaa
I’ve tried with the following:
type root struct{
build type Build `yaml:"Build-t,omitempty"`
}
type Build struct {
Before map[string]interface{} `yaml:"before,omitempty"`
After map[string]interface{} `yaml:"after,omitempty"`
}
Now when I parse it I got error,
What I need is to get the values from the object before and after which are hardcoded value in the yaml
And all the others value under it are dynmically can be added therefor I put it as interface
btw, if I change the root to this its working and I see all the fields under the Build-t but the before and after are like keys ...
type root struct{
build type map[string]interface{} `yaml:"Build-t,omitempty"`
}
the error is:
line 6: cannot unmarshal !!str `test1` into map[string]interface {}
line 7: cannot unmarshal !!str `test2` into map[string]interface {}
see the yaml valid here
https://codebeautify.org/yaml-validator/cb705458
That sounds correct - the YAML is invalid. Did you mean something like this?
Build-t:
before:
test1:
- value: dddd
- bbb: zzzz
after:
test2:
- value: bbb
- aaa: aaaa
Remember that the whitespace is important, and it's a key-value structure - so your values can either be strings, or substructures - not both.
Also, that yaml validator... I can't seem to make it declare anything as invalid!

How do I make protobuf case-insensitive?

I have a protobuf contract like this,
message Car{
string carId = 1;
}
I generate java classes from this contract and use it to parse JSON request.
Now if my JSON has "CarID" or "carid" then protobuf generated java classes don't recognize that field. How do I make it case-insensitive?
The protobuff descriptor (.proto) are case insensitive. If you try to compile:
message Car{
string carId = 1;
string carid =2;
}
You will have the compilation error:
CARID_FIELD_NUMBER is already defined in ...
Also you have to know that for proto3, the JSON parser are dealing with lowerCamelCase. As stated on reference guide:
https://developers.google.com/protocol-buffers/docs/proto3#json
Use proto field name instead of lowerCamelCase name: By default proto3
JSON printer should convert the field name to lowerCamelCase and use
that as the JSON name. An implementation may provide an option to use
proto field name as the JSON name instead. Proto3 JSON parsers are
required to accept both the converted lowerCamelCase name and the
proto field name.
From your parser point of view "carID" and "CarID" are the same, because it will automatically convert "CarID" to "carID". But "carId" and "carid" will always be different.

Save YAML with inheritance in ruby

I have YAML file with inheritance and I want to add or edit a key programatically.
I load the YAML into hash using YAML.load method but when I save the hash back using YAML.dump I lose all the inheritance info.
Is there a way to edit the YAML in Ruby without losing the inheritance info?
YAML example:
main:
prod: &prod
key1: true
key2: 50
key3: "abc"
prod_v_3_5: &prod_v_3_5
<<: *prod
key2: 100
prod_v_3_6: &prod_v_3_6
<<: *prod_v_3_5
key2: 150
Code example:
config = Api.get(id)
yaml = YAML.load(config)
yaml["main"][section].store(key, value)
config = YAML.dump(yaml)
Api.set(id, config)
As far as I know (I also use that option to import and override) the YAML source is read and processed and then the hash elements are exposed and not linked internally. So the mechanism is copy paste override and not linking and overload.
I guess you have to modify the YAML source by opening the file and modify its content if you won't destroy your source.
To add new section:
config = Api.get(id)
config = "#{config}\n \n #{section}: &#{section}\n <<: *#{parent_section}"
To add new value:
config = Api.get(id)
matches = /^(.+)(\n #{section}:\s*&#{section}\s*\n )(<<:[^\n]+)?(.*)$/m.match(config)
config = "#{matches[1]}#{matches[2]}#{matches[3]}\n #{key}: #{value}#{matches[4]}\n"

Resources