I am looking for a xpath like query language for protobuf messages. For example, for the Person message shown below [ borrowed from the Developer guide ]
message Person {
required string name = 1;
required int32 id = 2;
optional string email = 3;
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
message PhoneNumber {
required string number = 1;
optional PhoneType type = 2 [default = HOME];
}
repeated PhoneNumber phone = 4;
}
I would like to have methods like
XPBQuery.get(person, "$.id") ==> returns the id
XPBQuery.get(person, "$.name") ==> returns the name
XPBQuery.get(person, "$.phone.number[0]") ==> returns the first phone number
One way is to convert the proto to Json and use a JsonPath/JsPath API's. But it may be expensive to convert to Json everytime especially for large Proto objects.
Any help is much appreciated.
Thanks,
Irfan
Support for that is coming in protobuf v3: https://github.com/google/protobuf/blob/4644f99d1af4250dec95339be6a13e149787ab33/src/google/protobuf/field_mask.proto
While looking for a solution to a similar problem I discovered:
PbQuery (python)
protobuf-utils (java)
protobuf-el (java)
(I did not use those libraries as my target language is C++, but hope this might help someone else)
Good luck!
Related
I have quick question for modifying type of field for protobuf.
I have create the int64 id something like below.
message test {
int64 id = 1;
}
And I find out I can not set this id as null because of the type.
Therefore, I would like to modify it from int64 to int64value but do not sure is there any standard way for this kind of operation.
message test {
// int64 id = 1;
int64value id = 2;
}
or
message test {
int64 id = 1 [deprecated=true];
int64value id = 2;
}
Thanks and open to any kind of input!
I would like to get more standard way for this kind of operation.
You can reference values by field number or by name. So your first approach works. As long as you do not need backwards compatibility you could just change the id from int to your id type. This would result in a cleaner code.
For the use of the deprecated Attribut you could check out this:
Google protobuf 3: deprecated a field, but cannot remove the dependencies?
This post describes what the numbered tags in proto files are for, essentially match fields when serializing and deserializing the data. My question is: what happens if I change the number of an existing field?
Taking the same example, say this is the original data
message SearchRequest {
required string query = 1;
// Pagination fields
optional int32 page_number = 2;
optional int32 result_per_page = 3;
}
And I want to add a new field, which logically makes sense to put before the pagination fields - Can I re-enumerate the fields as below, if I'm not using the page_number and result_per_page fields yet (though the message type SearchRequest is in use)?
message SearchRequest {
required string query = 1;
optional int32 new_data = 2;
// Pagination fields
optional int32 page_number = 3;
optional int32 result_per_page = 4;
}
Should I have given the pagination fields higher numbers from the start to allow for new fields?
The documentation says
These field numbers are used to identify your fields in the message
binary format, and should not be changed once your message type is in
use.
Changing field numbers is almost always a bad idea - you will get very odd behaviour unless all clients and servers are.deployed at exactly the same time and there is no persisted payload data anywhere (in files, databases, etc). In particular, the serializer will try to interpret data as a different thing: in some cases this will cause a serialization failure and in other cases it will silently and happily deserialize the data with garbage meaning, causing chaos.
There is no reason not to simply use field 5.
I am in the process of migrating a legacy application (bit of microservices and monolith) into using GRPC. The entire code base is currently in GO.
I have started to model my messages and they are looking very similar to my structs that I am using in my application code. It seems odd that I have the same objects defined twice but also seems odd to use message objects as core structs. Seems that I will have a lot of memory intensive marshaling of data though. Below is an example of a message and a struct and how similar they are.
Are there any recommendations/best practices on deciding how to model my messages? Should they be aligned with my core structs? Should I not care about all the marshalling that has to happen from my Golang Structs into messages?
Forgive me if I am asking such a basic question as I am very new to protocol buffers and GRPC.
Message Proto Def:
message Profile {
string UID = 1;
string ContactEmail = 2;
google.protobuf.Timestamp DateOfBirth = 3;
float WeightInKilos = 4;
string Gender = 5;
string Unit = 6;
string CurrentStatus = 7;
string Country = 8;
string ExperienceType = 9;
google.protobuf.Timestamp DateJoined = 10;
repeated Ability Abilities = 11;
repeated Role Roles = 12;
repeated TermsAndConditionsAcceptance TermsAndConditionsAcceptances = 13;
string TimeZone = 14;
repeated BaselineTestResults BaselineTests =15;
//Excluded UpdatedDate as other domains shouldn't need it
string FirstName =16;
string LastName =17;
string DisplayName = 18;
string State = 19;
repeated google.protobuf.Any Preferences = 20;
Thresholds Thresholds = 21;
string StripeCustomerID = 22;
}
Struct Def:
type Profile struct {
UID string `json:"UID" firestore:"UID"`
ContactEmail string `json:"ContactEmail,omitempty" firestore:"ContactEmail"`
DateOfBirth time.Time `json:"DateOfBirth,omitempty" firestore:"DateOfBirth"`
WeightInKilos float64 `json:"WeightInKilos,omitempty" firestore:"WeightInKilos"`
Gender string `json:"Gender,omitempty" firestore:"Gender"`
Unit string `json:"Unit,omitempty" firestore:"Unit"`
CurrentStatus string `json:"CurrentStatus,omitempty" firestore:"CurrentStatus"`
Country string `json:"Country,omitempty" firestore:"Country"`
ExperienceType string `json:"ExperienceType,omitempty" firestore:"ExperienceType"`
DateJoined time.Time `json:"DateJoined,omitempty" firestore:"DateJoined"`
Abilities []Ability `json:"Abilities,omitempty" firestore:"Abilities"`
Goals Goals `json:"Goals,omitempty" firestore:"Goals"`
Roles []Role `json:"Roles,omitempty" firestore:"Roles"`
TermsAndConditionsAcceptances []TermsAndConditionsAcceptance `json:"TermsAndConditionsAcceptances,omitempty" firestore:"TermsAndConditionsAcceptances"`
TimeZone string `json:"TimeZone,omitempty" firestore:"TimeZone"`
BaselineTests []BaselineTestResults `json:"BaselineTests,omitempty" firestore:"BaselineTests"`
UpdatedDate time.Time `json:"UpdatedDate,omitempty" firestore:"UpdatedDate"`
FirstName *string `json:"FirstName,omitempty" firestore:"FirstName"`
LastName string `json:"LastName,omitempty" firestore:"LastName"`
DisplayName string `json:"DisplayName,omitempty" firestore:"DisplayName"`
State string `json:"State"`
Preferences map[string]interface{} `json:"Preferences"`
Thresholds Thresholds `json:"Thresholds"`
StripeCustomerID string `json:"-"` //Tells it to ignore exporting
}
Though a very basic question but a very good question too. People generally skip it and have to struggle with codebase later as it grows.
Of course using message as core structs will be helpful in cases where the data you need to process the request is completely available in message. But in cases where you need to fetch data from other sources will not be helpful.
Normally a message data is dealt by boundary-layer/controller which further uses multiple services to create a response. So a service-layer/logic-layer is independent of a controller layer and the same should happen with message structs and core structs.
Always try to reduce coupling between layers so that they become reusable.
So if you have a layer which deals with database, you should have separate structs for that too.
you don't need to create structs in app code. just use messages of proto file.
Here is a quick start for GRPC in Go.
https://grpc.io/docs/quickstart/go/
Below is the service spec:
service Cooler {
rpc saveThing (stream SaveRequest) returns (SaveReply);
}
I need to stream messages to to Cooler.saveThing(). All the SaveRequests have a common field author and unique fields per a Thing are price and name. How can I send the author only once?
Not working attempt - Multiple inputs
It would be a solution but it is not supported by protobuf yet.
service Cooler {
rpc saveThing (stream SaveRequest, Author) returns (SaveReply);
}
Not working attempt - Nested message
Every received element of SaveRequest will still contain author and an array of Things.
message SaveRequest {
message Thing {
int price = 1;
string name = 2;
}
repeated Thing things = 1;
string author = 2;
}
Possible solution
Grpc headers.
Question
How can I send the author only once?
I'd like to update a message in Protocol Buffers:
message Person {
string name = 1;
}
Now, suppose that I don't want a name for a Person, but only its address:
message Person {
string address = 1;
}
Now, the id could remain 1 since the type is always a string, but I was wondering if it's better to rewrite the message in this way:
message Person {
string address = 2;
reserved 1;
}
in order to have more readability between versions.
you can just change the field name safely(if you want to keep same id and same type), please check below post would help you.
Protocol buffer: does changing field name break the message?
and also in my opion it is always good have
required or optional
annotation to the message fields