I want my custom SNMP agent to expose some metrics.
But I don't know how to deal with arrays of simple types (byte, int, double).
How can I write a MIB file to expose array of int ?
Is there a simpler way than the tables/entries/columns/indexes route ?
Related
I have a proxy service that translate protobuf into another struct. I
can just write some manual code to do that, but that is inefficient and boilerplate. I can also transform the protobuf data to JSON, and deserlize the JSON data into the destination struct, but the speed is slow and it is CPU heavy.
The Unmarshaler interface is now deprecated, and Message interface have internal types which I cannot implement in my project.
Is there a way I can do this now?
Psuedo code: basically, if Go's reflection supports setting and getting of struct / class fields by some sort of field identifier, then you can do this. Something like this in C# works, so long as the field types in the two classes are the same (because in C#, I'm doing object = object, which ends up being OK if they're the same actual type).
SourceStructType sourceStruct;
DestStructType destStruct;
foreach (Field sourceField in sourceStruct.GetType().GetFields())
{
Field destField = destStruct.GetType().FindFieldByName(sourceField.name);
destStruct.SetFieldValue(destField) = sourceStruct.GetFieldValue(sourceField);
}
If the structs are more complex - i.e. they have structs within them, then you'll have to recurse down into them. It can get fiddly, but once written you'll never have to write it ever again!
I have a map that uses an interface as the key. The map is defined like this MyMap map[Signature]Packets. The interface is Signature, and there will be two structs A and B that implement this interface. I am also using msgp to serialize these two structs.
My issue is that msgp automatically generates methods that use a pointer as the type of the function receiver, which I think will make the key Signature receive pointers. If that was the case, then the key would be different every single time since pointers are different, even though the underlying values are the same. So, every time, I would be creating a new entry instead of finding the existing one and modifying it.
I wonder:
Is there a way to force msgp to generate methods purely with function receivers of the concrete type? Currently, I can only modify the function receivers of auto-generated methods like MarshalMsg and UnmarshalMsg to the concrete type (A or B instead of *A or *B). By doing that, the key of the map is either of type A or of type B, and the map MyMap works fine. However, I know I should not modify the auto-generated code. So, I wonder whether there is an acceptable way to do that.
If there is no way to do 1., is there any workaround to solve this problem? I really need some polymorphic feature of the map's key with the use of msgp.
UPDATE 1 (Apr. 12):
Thanks for sharing your thoughts and offering solutions. Here are some details about my question.
The background is that the map is used for collecting different network events. The two structs implementing the interface Signature are EventSignatureIPv4 and EventSignatureIPv6
type EventSignatureIPv4 struct {
SourceIPv4 [4]byte
Port uint16
Traffic TrafficType
}
type EventSignatureIPv6 struct {
SourceIPv6 [16]byte
Port uint16
Traffic TrafficType
}
and Signature is holding common methods shared between IPv4 and IPv6 data. So, essentially, I want to collect and group corresponding IPv4/v6 events at the runtime. The key of the map is to identify the same source, and the value of the map is to collect events with different destinations.
The msgp library I am using is this one https://pkg.go.dev/github.com/tinylib/msgp#v1.1.5/msgp
Correct me if I am wrong. For compositions in Go, if one of the methods in the method set has a function receiver of the pointer type, then the instance would only be of the pointer type? So here, as I have
func (z *EventSignatureIPv6) MarshalMsg(b []byte) (o []byte, err error) {
/* Auto-generated code */
}
whenever I use Signature to receive the struct EventSignatureIPv6, the struct would only be of type *EventSignatureIPv6?
You are right, "two pointer values are equal if they point to the same variable.", so if you are looking to compare interfaces that may hold pointers to different types, e.g. *A and *B, you are already in trouble.
With that said, I don't think it's an amazing idea to use interface types as map keys in the first place, because you have to deal with some caveats, the first is that:
The comparison operators == and != must be fully defined for operands of the key type
And now you need to be careful about the types that implement the interface. In theory, nobody stops a client from implementing your interface on a defined type with underlying unhashable type, e.g. type UncomparableSignature []int
So you would probably have to add an unexported method on your interface, so that client code outside that package can't implement it. But still, nothing stops code within the same package from implementing it, so this is, at best, maintenance overhead.
Then if the interface holds pointers to zero-values, it's even dependant on the implementation of the specs:
Pointers to distinct zero-size variables may or may not be equal.
Furthermore, you open yourself up to pesky bugs, like variables of type Signature that holds a nil will overwrite each other's values:
var foo Signature
var bar Signature
myMap[foo] = &Packet{/*pretending to have value 1*/}
myMap[bar] = &Packet{/*pretending to have value 2*/}
fmt.Println(myMap[foo]) // 2
A possible solution is, you could replace the map key with a unique id, and you enforce implementors to provide it by declaring the appropriate method on the interface Signature (this still assumes that the implementors can be coordinated to provide unique ids across all of them):
type Signature interface {
UniqueIdent() uint64 // or string, if you prefer
// ...other methods
}
and then
packet := myMap[someSignature.UniqueIdent()]
Looking for a solution in gRPC protobuff to implement dynamic fields of different datatypes for an multi-tenant application.
Also there can be any number of dynamic fields based on tenant.
Using map in proto, I can define different set of map for each data type. Is there any optimized way to achieve this.
Any help on this is appreciated.
There are a few different ways of transferring dynamic content in protobuf. Which is ideal varies depending on your use case. The options are ordered by their dynamism. Less dynamic options normally have better performance.
Use google.protobuf.Any in proto3. This is useful when you want to store arbitrary protobuf messages and is commonly used to provide extension points. It replaces extensions from proto2. Any has a child message and its type, so your application can check at runtime if it understands the type. If your application does not know the type, then it can copy the Any but can't decode its contents. Any cannot directly hold scalar types (like int32), but each scalar has a wrapper message that can be used instead. Because each Any includes the type of the message as a string, it is poorly suited if you need lots of them with small contents.
Use the JSON mapping message google.protobuf.Value. This is useful when you want to store arbitrary schemaless JSON data. Because it does not need to store the full type of its contents, a Value holding a ListValue of number_values (doubles) will be more compact on-the-wire than repeated Any. But if a schema is available, an Any containing a message with repeated double will be more compact on-the-wire than Value.
Use a oneof that contains each permitted type. Commonly a new message type is needed to hold the oneof. This is useful when you can restrict the schema but values have a relationship, like if the position of each value in a list is important and the types in the list are mixed. This is similar to Value but lets you choose your own types. While technically more powerful than Value it is typically used to produce a more constrained data structure. It is equal to or more compact on-the-wire than Value. This requires knowing the needed types ahead-of-time. Example: map<string, MyValue>, where MyValue is:
message MyValue {
oneof kind {
int32 int_value = 1;
string string_value = 2;
}
}
Use a separate field/collection for each type. For each type you can have a separate field in a protobuf message. This is the approach you were considering. This is the most compact on-the-wire and most efficient in memory. You must know the types you are interested in storing ahead of time. Example: map<string, int32> int_values = 1; map<string, string> string_values = 2.
I have a matrix struct written in Go. That matrix struct has a bunch of methods. I want to be able to efficiently compute matrix operations but I also want to be able to send it over the wire in order to distribute the computation.
I currently have the matrix and its methods separate from the protobuf definition. When I need to send it over the wire I have to create a new pb.Matrix{} from the existing Matrix{} struct and then make my grpc call. That seems like a waste. So, is it a waste? And should I just be defining my matrix struct as a protobuf definition and then use embedding to define operations on it? Or is it better to keep them separate from each other?
In terms of architecture, I'd keep them separate. That would agree with the Single Responsibility Principle. In one of my projects we use this form:
type Foo struct { ... }
func NewFooFromProto(f *myproto.Foo) *Foo { ... }
func (f *Foo) ToProto() *myproto.Foo { ... }
For a given type Data , I would like to define a set of filters, each processing Data in a certain way. Some filters only need the data to be processed, other may need additional parameters.
type Data struct {
...
}
I want to be able to define a list of filters, and apply them sequentially to an instance of Data. To acheive this, I defined a Filter interface :
type Filter interface {
Apply (d *Data) error
}
To define a filter, all I have to do is create a new type and define the Apply method for it.
Now, let's say I have a filter that does not need any additional information. Is it good practice to define it as an empty struct ?
type MySimpleFilter struct {}
func (f *MySimpleFilter) Apply (d *Data) {
...
}
I'd argue this is good practice if you have no use for a Field, especially compared to using another type (i.e. type MySimpleFilter int) because an empty struct uses no space:
https://codereview.appspot.com/4634124
and it can still fulfill interface contracts (hence can be more useful than a functional approach in some cases).
This can also be a good idiom when using a map that you have no use for the value (i.e. map[string]struct{}). See this discussion for details:
https://groups.google.com/forum/?fromgroups=#!topic/golang-nuts/lb4xLHq7wug
This is a question that doesn't have a clear answer since it's a matter of taste. I'd say it is good practice because it makes MySimpleFilter symmetrical to the other filters, which makes it easier to understand the code.