I want to create a protobuf which returns a map containing custom keys. When I try to make one, I get an error: Key in map fields cannot be float/double, bytes or message types. I want to know if there is a workaround for it.
I am trying to create a map of following type (as an illustration):
message Name {
string first_name = 1;
string last_name = 2;
}
message GetNameResp {
map<Name, string> name_info = 1;
}
So, can the Name here be a possible key to the map? The only solution I could find was to serialize the map and then return as repeated values.
Thanks!
Map keys may not be message types:
https://developers.google.com/protocol-buffers/docs/proto3#maps
You would have to work around this in code.
You could convert Name to string:
serialize the protobuf message
marshaling to JSON
hashing the value (though you'd potentially have collisions)
Related
If a field stores a specific value and is interpreted in a specific manner, is it possible to change this interpretation in a backwards compatible way?
Let's say I have a field that stores values of different data types.
The most generic case is to store it as a byte array and let the apps encode and decode it to the correct data type.
Common cases for data types are integers and strings, so support for those types is present.
Using a oneof structure this looks as follows:
message Foo
{
...
oneof value
{
uint32 integer = 1;
string text = 2;
bytes data = 3;
}
}
Applications that want to store an ip prefix in the value field, have to use the generic data field and do the encoding and decoding correctly.
If I now want to add support for ip prefixes to the Foo message itself so the apps don't have to deal with the encoding and decoding anymore, I could add a new field to the oneof structure with an IpPrefix datatype:
message Foo
{
...
oneof value
{
uint32 integer = 1;
string text = 2;
bytes data = 3;
IpPrefix ip_prefix = 4;
}
}
Even though this makes life easier for the apps, I believe it breaks backwards compatibility.
If a sending app has support for the new field, it will put its ip prefix value in the ip_prefix field.
But if a receiving app does not have support for this new field yet, it will ignore the field.
It will look for the ip prefix value in the data field, as it always did, but it won't find it there.
So the receiving app can no longer correctly read the ip prefix value anymore.
Is there a way to make this scenario somehow backwards compatible?
PS: I realize this is a somewhat vague and perhaps unrealistic example. The exact case I need it for is for the representation of RADIUS attributes in a protobuf message. These attributes are in essence a byte array that is sent over the network, but the bytes in the array have meaning and could be stored as different fields in the protobuf message. A basic attribute exists of a Type field and a Value field where the value field can be a string, integer, ip address... From time to time new datatypes (even complex ones) are added and I would like to be able to add new datatypes in a backwards compatible way.
There are two ways to go about this:
1. Enforce an update schedule, readers before writers
Add the new type of field to the .proto definition, but document that it should not be used except for testing and reception. Document that all readers of the message must support both the old and the new field by a specific milestone/date, after which the writers can start using it. Eventually you can deprecate the old field and new readers don't need to support it anymore.
2. Have both fields during the transition period
message Foo
{
...
oneof value
{
uint32 integer = 1;
string text = 2;
bytes data = 3;
}
IpPrefix ip_prefix = 4;
}
Document that writers should set both data and ip_prefix during the transition period. The readers can start using ip_prefix as soon as writers have added support, and can optionally fall back to data.
Later, you can deprecate data and move ip_prefix to inside the oneof without breaking compatibility.
I am trying to do the following:
Deserialise a struct of fields into another struct of different format.
I am trying to do the transformation via an Hashmap as this seems to be the best suited way.
I am required to do this as a part of transformation of a legacy system, and this being one of the intermediary phases.
I have raised another question which caters to a subset of the same use-case, but do not think it gives enough overview, hence raising this question with more details and code.
(Rust converting a struct o key value pair where the field is not none)
Will be merging both questions once I figure out how.
Scenario:
I am getting input via an IPC through a huge proto.
I am using prost to deserialise this data into a struct.
My deserialised struct has n no of fields and can increase as well.
I need to transform this deserialised struct into a key,value struct. (shown ahead).
The incoming data, will mostly have a majority of null keys .i.e out of n fields, most likely only 1,2 have values at a given time.
Input struct after prost deserialisation:
Using proto3 so unable to define optional fields.
Ideally I would prefer a prost struct of options on every field. .i.e Option instead of string.
struct Data{
int32 field1,
int64 field2,
string field3,
...
}
This needs to be transformed to a genric struct as below for further use:
struct TransformedData
{
string Key
string Value
}
So,
Data{
field1: 20
field2: null
field3: null
field4: null
....
}
becomes
TransformedData{
key:"field1"
Value: "20"
}
Methods I have tried:
Method1
Add serde to the prost struct definiton and deserialise it into a map.
Loop over each item in a map to get values which are non-null.
Use these to create a struct
https://play.rust-lang.org/?version=stable&mode=debug&edition=2015&gist=9645b1158de31fd54976926c9665d6b4
This has its challenges:
Each primitve data type has its own default values and needs to be checked for.
Nested structs will result in object data type which needs to be handled.
Need to iterate over every field of a struct to determine non null values.
1&2 can be mitigated by setting an Optional field(yet to figure out how in proto3 and prost)
But I am not sure how to get over 3.
Iterating over a large struct to find non null fields is not scalable and will be expensive.
Method 2:
I am using prost reflects dynamic reflection to deserialise and get only specified value.
Doing this by ensuring each proto message has two extra fields:
proto -> signifying the proto struct being used when serializing.
key -> signifying the filed which has value when serializing.
let fd = package::FILE_DESCRIPTOR_SET;
let pool = DescriptorPool::decode(fd).unwrap();
let proto_message: package::GenericProto = prost::Message::decode(message.as_ref()).expect("error when de-serialising protobuf message to struct");
let proto = proto_message.proto.as_str();
let key = proto_message.key.as_str();
Then using key , I derive the key from what looks to be a map implemented by prost:
let message_descriptor = pool.get_message_by_name(proto).unwrap();
let dynamic_message = DynamicMessage::decode(message_descriptor, message.as_ref()).unwrap();
let data = dynamic_message.get_field_by_name(key).unwrap().as_ref().clone();
Here :
1.This fails when someone sends a struct with multiple fields filled.
2.This does not work for nested objects, as nested objects in turn needs to be converted to a map.
I am looking for the least expensive way to do the above.
I understand the solution I am looking for is pretty out there and I am taking a one size fits all approach.
Any pointers appreciated.
I am new to golang and I have one issue which I think community can help me to solve it.
I have one data structure like below
type ParentIDInfo struct {
PCOrderID string `json:"PCorderId",omitempty"`
TableVarieties TableVarietyDC `json:"tableVariety",omitempty"`
ProduceID string `json:"PRID",omitempty"`
}
type PCDCOrderAsset struct {
PcID string `json:"PCID",omitempty"`
DcID string `json:"DCID",omitempty"`
RequiredDate string `json:"requiredDate",omitempty"`
Qty uint64 `json:"QTY",omitempty"`
OrderID string `json:"ORDERID",omitempty"`
Status string `json:"STATUS",omitempty"`
Produce string `json:"Produce",omitempty"`
Variety string `json:"VARIETY",omitempty"`
Transports []TransportaionPCDC `json:"Transportaion",omitempty"`
ParentInfo []ParentIDInfo `json:"ParentInfo",omitempty"`
So I have issue to access the PCOrderID which inside the []ParentIDInfo . I have tried below however I getting error as "pcdcorder.ParentInfo.PCOrderID undefined (type []ParentIDInfo has no field or method PCOrderID)"
keyfarmercas = append(keyfarmercas, pcdcorder.ParentInfo.PCOrderID)
Any help will be very good
Thanks in advance
PCDCOrderAsset.ParentInfo is not a struct, it does not have a PCOrderID field. It's a slice (of element type ParentIDInfo), so its elements do, e.g. pcdcorder.ParentInfo[0].PCOrderID.
Whether this is what you want we can't tell. pcdcorder.ParentInfo[0].PCOrderID gives you the PCOrderID field of the first element of the slice. Based on your question this may or may not be what you want. You may want to append all IDs (one from each element). Also note that if the slice is empty (its length is 0), then pcdcorder.ParentInfo[0] would result in a runtime panic. You could avoid that by first checking its length and only index it if its not empty.
In case you'd want to add ids of all elements, you could use a for loop to do that, e.g.:
for i := range pcdorder.ParentInfo {
keyfarmercas = append(keyfarmercas, pcdcorder.ParentInfo[i].PCOrderID)
}
I am creating a proto3 schema that allow multiple/arbitrary data in incoming jsonObject. I would like to convert the incoming json object in one shot.
for example
{"key1":"value",
"key2": { //schema A}
}
I also want to support for schema B for key2 in different request.
{"key1":"value",
"key2": { //schema B}
}
I tried couple of different approach such as oneof but for oneof it needs different key names, since I am using the same key2, it does not work for me in this case.
and here is the schema.
message IncomingRequest {
string key1 = 1;
//google.protobuf.Any key2 = 2; --> not working
oneof message{
A payload = 2;
B payload = 3; --> duplicate key
}
}
Anyone know how to achieve this?
Two ways I can think of:
Message types for each request
If you know, based on the request (e.g. HTTP URL called), if it should be schema A or B, I recommend to create separate message types for each request. This might result in more proto types you have to define, but will be simple to use in the actual code you have to write to consume the payloads.
Struct type
If you really want/have to reuse the same message type you can use the Struct proto type to encode/decode any JSON structure.
message IncomingRequest {
string key1 = 1;
google.protobuf.Struct key2 = 2;
}
Although from the proto type definition it doesn't look like it does what you want, Protobuf decoders/encoders will handle this type in a special way to give you the behavior wanted.
The issue with this option is that what you gain in flexibility in the proto, you lose in expressiveness in the generated code, as you have to do a lot edge-case checking if a specific value/type is set.
I am using protocol buffers defined like this:
message Index {
message albums {
repeated string name = 1;
}
map<string, albums> artists_albums= 1;
map<int32, albums> year_albums = 2;
}
It generates go code like this:
type Index struct {
ArtistsAlbums map[string]*IndexAlbums
YearAlbums map[int32]*IndexAlbums
}
How can I make it generate map values of type IndexAlbums instead of *IndexAlbums?
If you use gogoprotobuf then there is an extension that will allow that
map<string, albums> artists_albums = 1 [(gogoproto.nullable) = false];
With regular goprotobuf I don't believe there is a way.
nullable, if false, a field is generated without a pointer (see warning below).
Warning about nullable: According to the Protocol
Buffer specification, you should be able to tell whether a field is
set or unset. With the option nullable=false this feature is lost,
since your non-nullable fields will always be set. It can be seen as a
layer on top of Protocol Buffers, where before and after marshalling
all non-nullable fields are set and they cannot be unset.