I want to generate a yaml file using a struct's yaml tags, but I want to ignore the omitempty tags. So I figured I should go over the struct recursively with reflection, and get the information using Tag.GetTag("yaml").
This is fine, but it leaves me with the task of creating a yaml with this information.
I started writing this -
const Indent = " "
const NL = ":\n"
type Yaml struct {
Simple []string
Slices [][]Yaml
Nested map[string] *Yaml
}
But I actually hope there is a useful library to do this, as it seems like a relatively common task to do. Unfortunately,a search in Google didn't find anything.
Does such a library exist?
Thanks,
Related
I want to know which is the best way to create instances of a certain struct based on a map[string]string
My app should process huge files in CSV format and should create an instance of a struct for each row of the file.
I'm already using the encoding/csv/Reader from golang to read the CSV file and create an instance of map[string]string for each row in the file.
So given this file:
columnA, columnB, columnC
a, b, c
My own reader implementation will return this map (each row values with the header):
myMap := map[string]string{
"columnA": "a",
"columnB": "b",
"columnC": "c",
}
(this is just an example in real life the file contains a lot of columns and rows)
so.. at this point I need to create an instance of the struct that is related with the row contents, let say:
type MyStruct struct {
AColumn string
BColumn string
CColumn string
}
My question is what could be the best way to create the instance of the struct using the given map, I have already implemented a version that just copy each value from the map to the struct but the code ended up being very long and tedious:
s := &MyStruct{}
s.AColumn := m["columnA"]
s.AColumn := m["columnB"]
s.AColumn := m["columnC"]
...
I also consider using this library https://github.com/mitchellh/mapstructure but I don't know if using reflection could be the best approach considering that the file is huge and will be using reflection for each row.
Maybe there is no other option but I'm asking just in case someone knows a better approach.
Thanks in advance.
I would say that the idiomatic Go way would be just populating the struct's fields from your map. Go favors explicitness this approach is the more direct and the easiest to read. In other words, your approach is correct.
You could make it slightly nicer by initializing the struct directly:
s := &MyStruct{
AColumn: m["columnA"],
BColumn: m["columnB"],
CColumn: m["columnC"],
}
Now, if your structure has 100s of fields (which is an odd design choice), you may want to leverage some code generation. Otherwise, just go with the straightforward code - it's the best approach in the long term.
I already posted a library that I made for some stuff I have needed sometimes, I've made a MapToStruct fews months ago, I pushed that today to share with you the full library. The library is based in the usage of reflect, I still testing and implementing stuff, you will find some odd comments and these kind of things.
https://github.com/FedeMFernandez/goscript
I Hope it is useful
I need to change some values in YAML file from Go code. In my case, I need to change values.yaml file from Helm chart. Since that file can change, I do not structure of the whole file in advance (for example developers added new YAML sections in it in various projects). I just know how section that I want to change looks like.
I understand there is YAML library in Go (https://github.com/go-yaml/yaml). It will not do the job, because it assumes I know in advance structure of the file that I need to change. All examples are something like:
1. create struct
2. unmarshal YAML to struct
3. change
4. marshal and save back
It is not working for me since I do not know exact format of file, hence I cannot do step 1, create struct.
This is part of YAML file I am trying to change:
image:
repository: nginx
tag: stable
pullPolicy: IfNotPresent
I understand this can be done with help of interface{}, but I do not understand how. Assuming that I understand struct, marshal/unmarshal YAML files, can someone provide code that will:
1. Load YAML file that has at least 20 entries in it and is of unknown structure
2. Change only 1 entry (in my case I want to change tag number for image section)
3. Save it back.
Thanks a lot !
Something like this should work:
data, err := ioutil.ReadFile(file)
var v interface{}
err = yaml.Unmarshal(data, &v)
img, ok := v.["image"].(map[interface{}]interface{})
if ok {
img["tag"] = "somevalue"
}
The yaml library I use unmarshals into map[interface{}]interface{}. You need to add the necessary error checking, type assertions, etc.
When done, you can yaml.Marshal(v) and write the result.
The struct in the .pb.go file generated by .proto file has three additional fields and some other things.like this:
When converting this struct to json, if one field is empty, the field will not appear in json. Now I know it can be done using jsonpb.Marshaler.
m := jsonpb.Marshaler{EmitDefaults: true}
Now, I coverting struct to map[string]interface{}, put it in
InfluxDB. I have to convert struct to map[string]interface{}.The function NewPoint needs. like this:
I use structs.Map(value) function in go ,The transformed map has three additional fields, and running the program causes errors，like this:
{"error":"unable to parse 'txt,severity=1 CurrentValue=\"1002\",MetricAlias=\"CPU\",XXX_sizecache=0i,XXX_unrecognized= 1552551101': missing field value"}
When I remove these three fields, the program runs OK.These three fields are automatically generated, and I have a lot of structs.
What should I do?Thank you!
Protobuf generator adds some additional fields with names starting from XXX that are intended for optimizations. You can't change this behavior of protoc-gen-go.
The problem is in the way you convert struct to map[sting]interface{}. It's hard to figure out from which package exactly structs.Map comes from. Seems like it goes from here: https://github.com/fatih/structs/blob/master/structs.go#L89 - this code uses reflect to iterate through all fields of the structure and push them to map[sting]interface{}. You just need to write your own slightly modified version of FillMap routine that will omit XXX fields.
For testing, I often see go code read byte slices, which are parsed into structs using yaml, for example here:
https://github.com/kubernetes/kubernetes/blob/master/pkg/util/strategicpatch/patch_test.go#L74m
I just got bitten by not exporting my field names, resulting in an empty list which I iterated over in my test cases, thus assuming that all tests were passing (in hindsight, that should have been a red flag :)). There are other errors which are silently ignored by yaml unmarshaling, such as a key being misspelled and not matching a struct field exactly.
Is there a way to ensure that all the data in the byte slice was actually parsed into the struct returned by yaml.Unmarshal? If not, how do others handle this situation?
go-yaml/yaml
For anyone searching for a solution to this problem, the yaml.v2 library has an UnmarshalStrict method that returns an error if there are keys in the yaml document that have no corresponding fields in the go struct.
import yaml "gopkg.in/yaml.v2"
err := yaml.UnmarshalStrict(data, destinationStruct)
BurntSushi/toml
It's not part of the question, but I'd just like to document how to achieve something similar in toml:
You can find if there were any keys in the toml file that could not be decoded by using the metadata returned by the toml.decode function.
import "github.com/BurntSushi/toml"
metadata, err := toml.Decode(data, destinationStruct)
undecodedKeys := metadata.Undecoded()
Note that metadata.Undecoded() also returns keys that have not been decoded because of a Primitive value. You can read more about it here.
Json
The default go json library does not support this currently, but there is a proposal ready to be merged. It seems that it will be a part of go 1.10.
As you could probably tell from the below code I am working on a project which creates csv reports from data in mongoDB. After getting the data I need in, I need to structure the data into something more sensible then how it exists in the db, which is fairly horrendous (not my doing) and near impossible to print the way I need it. The structure that makes the most sense to me is a slice (for each document of data) of maps of the name of the data to a structure holding the data for that name. Then I would simply have to loop through the document and stuff values into the structs where they belong.
My implementation of this is
type mongo_essential_data_t struct {
caution string
citation string
caution_note string
}
mongo_rows_struct := make([]map[string]mongo_essential_data_t, len(mongodata_rows))
//setting the values goes like this
mongo_rows_struct[i][data_name].caution_note = fmt.Sprint(k)
//"i" being the document, "k" being the data I want to store
This doesn't work however. When doing "go run" it returns ./answerstest.go:140: cannot assign to mongo_rows_struct[i][data_name].caution_note. I am new to Go and not sure why I am not allowed to do this. I'm sure this is an invalid way to reference that particular data location, if it is even possible to reference it in Go. What is another way to accomplish this setting line? If it is too much work to accomplish this the way I want, I am willing to use a different type of data structure and am open to suggestions.
This is a known issue of Golang, known as issue 3117. You can use a temporary variable to get around it:
var tmp = mongo_rows_struct[i][data_name]
tmp.caution_note = fmt.Sprint(k)
mongo_rows_struct[i][data_name] = tmp
as per my understanding, when you write:
mongo_rows_struct[i][data_name]
compiler will generate code, which will return copy of mongo_essential_data_t struct(since struct in go is value type, not reference type), and
mongo_rows_struct[i][data_name].caution_note = fmt.Sprint(k)
will write new value to that copy. And after that copy will be discarded. Obviously, its not what you expect. So Go compiler generate error to prevent this misunderstanding.
In order to solve this problem you can:
1. Change definition of your data type to
[]map[string]*mongo_essential_data_t
2. Explicitly create copy of your struct, make changes in that copy and write it back to the map
data := mongo_rows_struct[i][data_name]
data.caution_note = fmt.Sprint(k)
mongo_rows_struct[i][data_name] = data
Of course, first solution is preferable because you will avoid unnecessary copying of data