I have json data like:
`[{"fea1":12345},{"fea2":23456}]`
I want to unmarshal them into Go structs.
Now I defined a struct like []map[string]int.
It works, but I think this is not the best way which processing pair data with map structure.
Processing large data set also cost much resource if using map structure.
Is there a more graceful way to implement it?
If you have predefined set of fields, you may use struct like this:
type Fea struct {
Fea1 int `json:"fea1,omitempty"`
Fea2 int `json:"fea2,omitempty"`
}
type Feas []Fea
var feas Feas
Then Unmarshal to feas. This way present fields would be filled, others would be empty.
Related
I am trying to do the following:
Deserialise a struct of fields into another struct of different format.
I am trying to do the transformation via an Hashmap as this seems to be the best suited way.
I am required to do this as a part of transformation of a legacy system, and this being one of the intermediary phases.
I have raised another question which caters to a subset of the same use-case, but do not think it gives enough overview, hence raising this question with more details and code.
(Rust converting a struct o key value pair where the field is not none)
Will be merging both questions once I figure out how.
Scenario:
I am getting input via an IPC through a huge proto.
I am using prost to deserialise this data into a struct.
My deserialised struct has n no of fields and can increase as well.
I need to transform this deserialised struct into a key,value struct. (shown ahead).
The incoming data, will mostly have a majority of null keys .i.e out of n fields, most likely only 1,2 have values at a given time.
Input struct after prost deserialisation:
Using proto3 so unable to define optional fields.
Ideally I would prefer a prost struct of options on every field. .i.e Option instead of string.
struct Data{
int32 field1,
int64 field2,
string field3,
...
}
This needs to be transformed to a genric struct as below for further use:
struct TransformedData
{
string Key
string Value
}
So,
Data{
field1: 20
field2: null
field3: null
field4: null
....
}
becomes
TransformedData{
key:"field1"
Value: "20"
}
Methods I have tried:
Method1
Add serde to the prost struct definiton and deserialise it into a map.
Loop over each item in a map to get values which are non-null.
Use these to create a struct
https://play.rust-lang.org/?version=stable&mode=debug&edition=2015&gist=9645b1158de31fd54976926c9665d6b4
This has its challenges:
Each primitve data type has its own default values and needs to be checked for.
Nested structs will result in object data type which needs to be handled.
Need to iterate over every field of a struct to determine non null values.
1&2 can be mitigated by setting an Optional field(yet to figure out how in proto3 and prost)
But I am not sure how to get over 3.
Iterating over a large struct to find non null fields is not scalable and will be expensive.
Method 2:
I am using prost reflects dynamic reflection to deserialise and get only specified value.
Doing this by ensuring each proto message has two extra fields:
proto -> signifying the proto struct being used when serializing.
key -> signifying the filed which has value when serializing.
let fd = package::FILE_DESCRIPTOR_SET;
let pool = DescriptorPool::decode(fd).unwrap();
let proto_message: package::GenericProto = prost::Message::decode(message.as_ref()).expect("error when de-serialising protobuf message to struct");
let proto = proto_message.proto.as_str();
let key = proto_message.key.as_str();
Then using key , I derive the key from what looks to be a map implemented by prost:
let message_descriptor = pool.get_message_by_name(proto).unwrap();
let dynamic_message = DynamicMessage::decode(message_descriptor, message.as_ref()).unwrap();
let data = dynamic_message.get_field_by_name(key).unwrap().as_ref().clone();
Here :
1.This fails when someone sends a struct with multiple fields filled.
2.This does not work for nested objects, as nested objects in turn needs to be converted to a map.
I am looking for the least expensive way to do the above.
I understand the solution I am looking for is pretty out there and I am taking a one size fits all approach.
Any pointers appreciated.
I have a standard response built in PHP for microservices that I want to replicate in Go.
In particular, the "data" element of the response array can have anything.
Example
$response['data'] = [];
$response['data']['entities'] = [ objects that implements toArray() ]
$response['data']['quantity'] = 1
Problem is that Data differs per endpoint.
So one option I was thinking is in each endpoint (controller, action, etc) declare a Data struct that I would use in the StandardResponse.Data struct.
But how can I achieve that? Using interfaces?
Sorry I am new to Go.
An interface{} is one way to do it. Something like:
type StandardResponse struct {
Error bool
ErrorMsg string
Data interface{}
}
Marshaling this to JSON will work assuming the underlying type of interface{} supports it.
I would like to know if there is easy way to convert from one struct to another in Go when one struct includes the other.
For example
type Type1 struct {
Field1 int
Field2 string
}
type Type2 struct {
Field1 int
}
I know that it can be handled like this
var a Type1{10, "A"}
var b Type2
b.Field1 = a.Field1
but if there are many fields, I will have to write numerous assignments. Is there any other way to handle it without multiple assignments?
In a word, is there anything like b = _.omit(a, 'Field2') in javascript?
Not directly, no. You can freely convert between identical types only.
You can get various levels of solutions to this type of problem:
writing the assignments out yourself (likely the best performance)
using reflection to copy from one to the other based on field names
something quick-and-dirty like marshalling one type to JSON then unmarshalling to the other type (which is basically using reflection under the hood with a plaintext middleman, so it's even less efficient, but can be done with little work on your part)
I am really stuck here with a seemingly trivial problem in Go:
I have a Golang microservice, which outputs data in json format. Let's say I have a simple struct with json tags for this result:
type Result struct {
Name string `json:"name"`
Age int `json:"age"`
}
In the code part where the data is actually pulled out of the database, I have a quite similar struct, like this:
type ResultBackend struct {
Name string `bson:"fullName"`
Age int `bson:"age"`
}
The struct fields are similar, except the different tags. I would like to keep things simple, and return just one struct from the backend service (ResultBackend), which can then be send as a JSON response, like this:
func process() Result {
var result ResultBackend
... do a MongoDB query here and store results in result variable ...
return result
}
This certainly would not work, because we have two different structs here.
Of course one solution would be to embed both tags in one struct like so:
type Result struct {
Name string `json:"name" bson:"fullName"`
Age int `json:"age bson:"age"`
}
and then use this struct in the main code and the "process" function.
This works, but this seems like "poisoning" the Result struct of the main code with the bson tags. What, for instance, if the backend result is a XML file? I'd have to add xml tags to the struct as well. Or maybe someday tags some very obsure database adapter.
This doesn't seem to be the cleanest approach in my eyes. I'd rather have a clean Result struct in the main code, and simply to a conversion from one struct to another.
Is there any easy way doing that, or do I really have to copy all the fields of a ResultBackend struct to a new Result struct and return that? Or I am trying to over-simplify my code here? :)
Cheers!
What I'd do is create a separate type for every "serialisation format" if you will. This approach has several advantages:
Your concerns are separated.
JSON un/marshalling doesn't interfere with BSON/XML, so you can add any kind of additional struct fields (like xml.Name for example).
You can actually create several such types for different APIs that need different parameters.
The only disadvantage I see is that it's more code and even more code to move data between those. Ultimately it's up to you and your application design. If you're sure that your JSON/BSON will stay the same, you can use one type. Otherwise, I'd recommend specialisation.
To those who want to add bson tags in golang struct, here is a tool which can be helpful. I have spent a whole day searching for this and I want others to save their valuable time.
You can convert the following struct
type Result struct {
Name string `json:"name"`
Age int `json:"age"`
}
into
type Result struct {
Name string `json:"name" bson:"fullName"`
Age int `json:"age bson:"age"`
}
With this tool, you can not only add bson tags but also
XML tags,
Define validation and scope
Format tag values
Apply transformation
skip unexported fields.
Go Modify Tags: https://github.com/fatih/gomodifytags
Cheers,
I want to encode pointers differently than from values. Currently, if we have a struct:
type Order struct {
Item Tool
AssociatedItem *Tool
}
both get inlined into a Order document inside mongo when marshalling.
I need to be able to perform my own serialization in the case of a *Tool. For instance, I could in this case only store the Id for the Too instead of the whole content. Unfortunately the overriding mechanism in mgo is to define a SetBSON() GetBSON for the Tool but it doesn't differentiate between pointers and non pointers.
What would be the best way to handle this?
Use a different type for "pointers", for example:
type SelectiveTool Tool
func (st *SelectiveTool) SetBSON(raw bson.Raw) error {
return raw.Unmarshal(s)
}
type Order struct {
Item Tool
AssociatedItem *SelectiveTool
}