I have a proxy service that translate protobuf into another struct. I
can just write some manual code to do that, but that is inefficient and boilerplate. I can also transform the protobuf data to JSON, and deserlize the JSON data into the destination struct, but the speed is slow and it is CPU heavy.
The Unmarshaler interface is now deprecated, and Message interface have internal types which I cannot implement in my project.
Is there a way I can do this now?
Psuedo code: basically, if Go's reflection supports setting and getting of struct / class fields by some sort of field identifier, then you can do this. Something like this in C# works, so long as the field types in the two classes are the same (because in C#, I'm doing object = object, which ends up being OK if they're the same actual type).
SourceStructType sourceStruct;
DestStructType destStruct;
foreach (Field sourceField in sourceStruct.GetType().GetFields())
{
Field destField = destStruct.GetType().FindFieldByName(sourceField.name);
destStruct.SetFieldValue(destField) = sourceStruct.GetFieldValue(sourceField);
}
If the structs are more complex - i.e. they have structs within them, then you'll have to recurse down into them. It can get fiddly, but once written you'll never have to write it ever again!
Related
I have a custom type, which is supposed to be an enumerator, and it implements json.Unmarshaler.
When I unmarshal into a slice, or a struct of this type, it works, like https://go.dev/play/p/DGg3nzo_VwN
When I unmarshal into a map containing this type, it breaks, like https://go.dev/play/p/YGgnRvr0agz
Crawling through the source code (go 1.18), I know how to make it work, just implement encoding.TextUnmarshaler, like https://go.dev/play/p/vY4E4snAY52 and I also have learned how the code works for maps: it checks if encoding.TextUnmarshaler is implemented, if yes, call its UnmarshalJSON before trying UnmarshalText. In other words, UnmarshalText must be present, however it will be ignored if UnmarshalJSON is also present.
Why is this precedence? Why only maps are treated this way?
From Go's decoder:
// Map key must either have string kind, have an integer kind,
// or be an encoding.TextUnmarshaler.
So you must implement TextUnmarshaller.
I have a map that uses an interface as the key. The map is defined like this MyMap map[Signature]Packets. The interface is Signature, and there will be two structs A and B that implement this interface. I am also using msgp to serialize these two structs.
My issue is that msgp automatically generates methods that use a pointer as the type of the function receiver, which I think will make the key Signature receive pointers. If that was the case, then the key would be different every single time since pointers are different, even though the underlying values are the same. So, every time, I would be creating a new entry instead of finding the existing one and modifying it.
I wonder:
Is there a way to force msgp to generate methods purely with function receivers of the concrete type? Currently, I can only modify the function receivers of auto-generated methods like MarshalMsg and UnmarshalMsg to the concrete type (A or B instead of *A or *B). By doing that, the key of the map is either of type A or of type B, and the map MyMap works fine. However, I know I should not modify the auto-generated code. So, I wonder whether there is an acceptable way to do that.
If there is no way to do 1., is there any workaround to solve this problem? I really need some polymorphic feature of the map's key with the use of msgp.
UPDATE 1 (Apr. 12):
Thanks for sharing your thoughts and offering solutions. Here are some details about my question.
The background is that the map is used for collecting different network events. The two structs implementing the interface Signature are EventSignatureIPv4 and EventSignatureIPv6
type EventSignatureIPv4 struct {
SourceIPv4 [4]byte
Port uint16
Traffic TrafficType
}
type EventSignatureIPv6 struct {
SourceIPv6 [16]byte
Port uint16
Traffic TrafficType
}
and Signature is holding common methods shared between IPv4 and IPv6 data. So, essentially, I want to collect and group corresponding IPv4/v6 events at the runtime. The key of the map is to identify the same source, and the value of the map is to collect events with different destinations.
The msgp library I am using is this one https://pkg.go.dev/github.com/tinylib/msgp#v1.1.5/msgp
Correct me if I am wrong. For compositions in Go, if one of the methods in the method set has a function receiver of the pointer type, then the instance would only be of the pointer type? So here, as I have
func (z *EventSignatureIPv6) MarshalMsg(b []byte) (o []byte, err error) {
/* Auto-generated code */
}
whenever I use Signature to receive the struct EventSignatureIPv6, the struct would only be of type *EventSignatureIPv6?
You are right, "two pointer values are equal if they point to the same variable.", so if you are looking to compare interfaces that may hold pointers to different types, e.g. *A and *B, you are already in trouble.
With that said, I don't think it's an amazing idea to use interface types as map keys in the first place, because you have to deal with some caveats, the first is that:
The comparison operators == and != must be fully defined for operands of the key type
And now you need to be careful about the types that implement the interface. In theory, nobody stops a client from implementing your interface on a defined type with underlying unhashable type, e.g. type UncomparableSignature []int
So you would probably have to add an unexported method on your interface, so that client code outside that package can't implement it. But still, nothing stops code within the same package from implementing it, so this is, at best, maintenance overhead.
Then if the interface holds pointers to zero-values, it's even dependant on the implementation of the specs:
Pointers to distinct zero-size variables may or may not be equal.
Furthermore, you open yourself up to pesky bugs, like variables of type Signature that holds a nil will overwrite each other's values:
var foo Signature
var bar Signature
myMap[foo] = &Packet{/*pretending to have value 1*/}
myMap[bar] = &Packet{/*pretending to have value 2*/}
fmt.Println(myMap[foo]) // 2
A possible solution is, you could replace the map key with a unique id, and you enforce implementors to provide it by declaring the appropriate method on the interface Signature (this still assumes that the implementors can be coordinated to provide unique ids across all of them):
type Signature interface {
UniqueIdent() uint64 // or string, if you prefer
// ...other methods
}
and then
packet := myMap[someSignature.UniqueIdent()]
I want to use the go structs that are generated by the gRPC compiler directly for database transactions but the problem is that only the json serialization field is set by gRPC.
Is there a way to either set additional serialization keys (like shown below) or is there another golang specific way to tell the database driver (sqlx on top of database/sql) that the json key format should be used?
Some example - The gRPC compiler creates the following struct:
type HelloWorld struct {
TraceId string `protobuf:"bytes,1,opt,name=trace_id,json=traceId,proto3" json:"trace_id,omitempty"`
...
What I would like to have:
type HelloWorld struct {
TraceId string `db:"trace_id" protobuf:"bytes,1,opt,name=trace_id,json=traceId,proto3" json:"trace_id,omitempty"`
...
A temporary workaround would be to write sql queries that use aliases (traceid instead of trace_id in this example) but it doesn't feel consistent and adds a lot of complexity.
I think that currently there is no built-in way of doing this. However, you might be interested in following this thread: https://github.com/golang/protobuf/issues/52
Other than that I think you can just create yet another struct for database access and make the mapping explicit which might be more readable.
Looking for a solution in gRPC protobuff to implement dynamic fields of different datatypes for an multi-tenant application.
Also there can be any number of dynamic fields based on tenant.
Using map in proto, I can define different set of map for each data type. Is there any optimized way to achieve this.
Any help on this is appreciated.
There are a few different ways of transferring dynamic content in protobuf. Which is ideal varies depending on your use case. The options are ordered by their dynamism. Less dynamic options normally have better performance.
Use google.protobuf.Any in proto3. This is useful when you want to store arbitrary protobuf messages and is commonly used to provide extension points. It replaces extensions from proto2. Any has a child message and its type, so your application can check at runtime if it understands the type. If your application does not know the type, then it can copy the Any but can't decode its contents. Any cannot directly hold scalar types (like int32), but each scalar has a wrapper message that can be used instead. Because each Any includes the type of the message as a string, it is poorly suited if you need lots of them with small contents.
Use the JSON mapping message google.protobuf.Value. This is useful when you want to store arbitrary schemaless JSON data. Because it does not need to store the full type of its contents, a Value holding a ListValue of number_values (doubles) will be more compact on-the-wire than repeated Any. But if a schema is available, an Any containing a message with repeated double will be more compact on-the-wire than Value.
Use a oneof that contains each permitted type. Commonly a new message type is needed to hold the oneof. This is useful when you can restrict the schema but values have a relationship, like if the position of each value in a list is important and the types in the list are mixed. This is similar to Value but lets you choose your own types. While technically more powerful than Value it is typically used to produce a more constrained data structure. It is equal to or more compact on-the-wire than Value. This requires knowing the needed types ahead-of-time. Example: map<string, MyValue>, where MyValue is:
message MyValue {
oneof kind {
int32 int_value = 1;
string string_value = 2;
}
}
Use a separate field/collection for each type. For each type you can have a separate field in a protobuf message. This is the approach you were considering. This is the most compact on-the-wire and most efficient in memory. You must know the types you are interested in storing ahead of time. Example: map<string, int32> int_values = 1; map<string, string> string_values = 2.
I am trying to use interface{} to Marshal and UnMarshal my xml in golang. The reason is that as in calling a soap server my soap envelope, header are same but i want to pass the different soap function by passing different structs. I have made a sample code in playground(not related to soap). I am able to marshal the xml with interface{} but unable to unmarshal.
Here is link Play Ground
Please tell me what i am doing wrong ?
You can't unmarshal to an empty interface since the empty interface doesn't have any exported fields to map the xml keys/values to. If you want share your same code to "dynamically" handle the message differently, you should store it in a string or byte array (a byte array is recommended in this case). You then process the data based on your needs.
http://play.golang.org/p/sPq0ZfAcU7