I am in the process of migrating a legacy application (bit of microservices and monolith) into using GRPC. The entire code base is currently in GO.
I have started to model my messages and they are looking very similar to my structs that I am using in my application code. It seems odd that I have the same objects defined twice but also seems odd to use message objects as core structs. Seems that I will have a lot of memory intensive marshaling of data though. Below is an example of a message and a struct and how similar they are.
Are there any recommendations/best practices on deciding how to model my messages? Should they be aligned with my core structs? Should I not care about all the marshalling that has to happen from my Golang Structs into messages?
Forgive me if I am asking such a basic question as I am very new to protocol buffers and GRPC.
Message Proto Def:
message Profile {
string UID = 1;
string ContactEmail = 2;
google.protobuf.Timestamp DateOfBirth = 3;
float WeightInKilos = 4;
string Gender = 5;
string Unit = 6;
string CurrentStatus = 7;
string Country = 8;
string ExperienceType = 9;
google.protobuf.Timestamp DateJoined = 10;
repeated Ability Abilities = 11;
repeated Role Roles = 12;
repeated TermsAndConditionsAcceptance TermsAndConditionsAcceptances = 13;
string TimeZone = 14;
repeated BaselineTestResults BaselineTests =15;
//Excluded UpdatedDate as other domains shouldn't need it
string FirstName =16;
string LastName =17;
string DisplayName = 18;
string State = 19;
repeated google.protobuf.Any Preferences = 20;
Thresholds Thresholds = 21;
string StripeCustomerID = 22;
}
Struct Def:
type Profile struct {
UID string `json:"UID" firestore:"UID"`
ContactEmail string `json:"ContactEmail,omitempty" firestore:"ContactEmail"`
DateOfBirth time.Time `json:"DateOfBirth,omitempty" firestore:"DateOfBirth"`
WeightInKilos float64 `json:"WeightInKilos,omitempty" firestore:"WeightInKilos"`
Gender string `json:"Gender,omitempty" firestore:"Gender"`
Unit string `json:"Unit,omitempty" firestore:"Unit"`
CurrentStatus string `json:"CurrentStatus,omitempty" firestore:"CurrentStatus"`
Country string `json:"Country,omitempty" firestore:"Country"`
ExperienceType string `json:"ExperienceType,omitempty" firestore:"ExperienceType"`
DateJoined time.Time `json:"DateJoined,omitempty" firestore:"DateJoined"`
Abilities []Ability `json:"Abilities,omitempty" firestore:"Abilities"`
Goals Goals `json:"Goals,omitempty" firestore:"Goals"`
Roles []Role `json:"Roles,omitempty" firestore:"Roles"`
TermsAndConditionsAcceptances []TermsAndConditionsAcceptance `json:"TermsAndConditionsAcceptances,omitempty" firestore:"TermsAndConditionsAcceptances"`
TimeZone string `json:"TimeZone,omitempty" firestore:"TimeZone"`
BaselineTests []BaselineTestResults `json:"BaselineTests,omitempty" firestore:"BaselineTests"`
UpdatedDate time.Time `json:"UpdatedDate,omitempty" firestore:"UpdatedDate"`
FirstName *string `json:"FirstName,omitempty" firestore:"FirstName"`
LastName string `json:"LastName,omitempty" firestore:"LastName"`
DisplayName string `json:"DisplayName,omitempty" firestore:"DisplayName"`
State string `json:"State"`
Preferences map[string]interface{} `json:"Preferences"`
Thresholds Thresholds `json:"Thresholds"`
StripeCustomerID string `json:"-"` //Tells it to ignore exporting
}
Though a very basic question but a very good question too. People generally skip it and have to struggle with codebase later as it grows.
Of course using message as core structs will be helpful in cases where the data you need to process the request is completely available in message. But in cases where you need to fetch data from other sources will not be helpful.
Normally a message data is dealt by boundary-layer/controller which further uses multiple services to create a response. So a service-layer/logic-layer is independent of a controller layer and the same should happen with message structs and core structs.
Always try to reduce coupling between layers so that they become reusable.
So if you have a layer which deals with database, you should have separate structs for that too.
you don't need to create structs in app code. just use messages of proto file.
Here is a quick start for GRPC in Go.
https://grpc.io/docs/quickstart/go/
Related
I have quick question for modifying type of field for protobuf.
I have create the int64 id something like below.
message test {
int64 id = 1;
}
And I find out I can not set this id as null because of the type.
Therefore, I would like to modify it from int64 to int64value but do not sure is there any standard way for this kind of operation.
message test {
// int64 id = 1;
int64value id = 2;
}
or
message test {
int64 id = 1 [deprecated=true];
int64value id = 2;
}
Thanks and open to any kind of input!
I would like to get more standard way for this kind of operation.
You can reference values by field number or by name. So your first approach works. As long as you do not need backwards compatibility you could just change the id from int to your id type. This would result in a cleaner code.
For the use of the deprecated Attribut you could check out this:
Google protobuf 3: deprecated a field, but cannot remove the dependencies?
This post describes what the numbered tags in proto files are for, essentially match fields when serializing and deserializing the data. My question is: what happens if I change the number of an existing field?
Taking the same example, say this is the original data
message SearchRequest {
required string query = 1;
// Pagination fields
optional int32 page_number = 2;
optional int32 result_per_page = 3;
}
And I want to add a new field, which logically makes sense to put before the pagination fields - Can I re-enumerate the fields as below, if I'm not using the page_number and result_per_page fields yet (though the message type SearchRequest is in use)?
message SearchRequest {
required string query = 1;
optional int32 new_data = 2;
// Pagination fields
optional int32 page_number = 3;
optional int32 result_per_page = 4;
}
Should I have given the pagination fields higher numbers from the start to allow for new fields?
The documentation says
These field numbers are used to identify your fields in the message
binary format, and should not be changed once your message type is in
use.
Changing field numbers is almost always a bad idea - you will get very odd behaviour unless all clients and servers are.deployed at exactly the same time and there is no persisted payload data anywhere (in files, databases, etc). In particular, the serializer will try to interpret data as a different thing: in some cases this will cause a serialization failure and in other cases it will silently and happily deserialize the data with garbage meaning, causing chaos.
There is no reason not to simply use field 5.
I'd like to update a message in Protocol Buffers:
message Person {
string name = 1;
}
Now, suppose that I don't want a name for a Person, but only its address:
message Person {
string address = 1;
}
Now, the id could remain 1 since the type is always a string, but I was wondering if it's better to rewrite the message in this way:
message Person {
string address = 2;
reserved 1;
}
in order to have more readability between versions.
you can just change the field name safely(if you want to keep same id and same type), please check below post would help you.
Protocol buffer: does changing field name break the message?
and also in my opion it is always good have
required or optional
annotation to the message fields
I am looking for a xpath like query language for protobuf messages. For example, for the Person message shown below [ borrowed from the Developer guide ]
message Person {
required string name = 1;
required int32 id = 2;
optional string email = 3;
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
message PhoneNumber {
required string number = 1;
optional PhoneType type = 2 [default = HOME];
}
repeated PhoneNumber phone = 4;
}
I would like to have methods like
XPBQuery.get(person, "$.id") ==> returns the id
XPBQuery.get(person, "$.name") ==> returns the name
XPBQuery.get(person, "$.phone.number[0]") ==> returns the first phone number
One way is to convert the proto to Json and use a JsonPath/JsPath API's. But it may be expensive to convert to Json everytime especially for large Proto objects.
Any help is much appreciated.
Thanks,
Irfan
Support for that is coming in protobuf v3: https://github.com/google/protobuf/blob/4644f99d1af4250dec95339be6a13e149787ab33/src/google/protobuf/field_mask.proto
While looking for a solution to a similar problem I discovered:
PbQuery (python)
protobuf-utils (java)
protobuf-el (java)
(I did not use those libraries as my target language is C++, but hope this might help someone else)
Good luck!
I read somewhere that it was best practice to use structs when passing around model data using Swift.
I have been doing so but I have been wondering whether there is anything that I can do in regards to the creation of large (and growing) user data objects like this:
struct UserAccount {
var id: String?
let authId: String
let emailAddress: String
let mobilePhoneNumber: String
let firstName: String
let lastName: String
let countryCode: String
let homepageUrl: String
let tagline: String
let pictureUrl: String
let accountState: AccountState = .NoAccount
// ...
}
This is more or less what I use when I create a user account, but later on it feels cumbersome and wrong to have to instantiate gigantic objects in code. I am parsing JSON responses using json-swift but then having to instantiate the models separately, like so:
let id = jsonData["id"].string!
let authId = jsonData["authId"].string!
let emailAddress = jsonData["emailAddress"].string!
let mobilePhoneNumber = jsonData["mobilePhoneNumber"].string!
let firstName = jsonData["firstName"].string!
let lastName = jsonData["lastName"].string!
let countryCode = jsonData["countryCode"].string!
let homepageUrl = jsonData["homepageUrl"].string!
let tagline = jsonData["tagline"].string!
let pictureUrl = jsonData["pictureUrl"].string!
let accountState = convertAccountStateStringToEnum(jsonData["accountState"].string!)
let userAccount = UserAccount(
id: id,
authId: authId,
emailAddress: emailAddress,
mobilePhoneNumber: mobilePhoneNumber,
firstName: firstName,
lastName: lastName,
countryCode: countryCode,
homePageUrl: homepageUrl,
tagline: tagline,
pictureUrl: pictureUrl,
accountState: accountState
)
It might seem absurd that above I've instantiated the variables before I instantiate the struct, but the reason I did so is that when the IDE gives me type coercion errors from within a struct it is very difficult to understand what is wrong, and so this allows me to troubleshoot it quicker when I am making model changes. Any thoughts around this?
My Question:
Later on that user object is likely to contain a lot more data on the server side and instantiating use models with 50+ lines seems like a bad idea, therefore:
Are there solutions to create structs in some simpler way? Does the pattern have a name?
Should I be creating User models that are related to specific tasks? For example, a UserAccount model to GET or PUT a profile might be different from the one used to authenticate, get the user settings or list the most popular accounts on my service.
Do people just leave it as it is and define mappers so that there is no redundancy? If so - what's a good example of this?
Thanks.
When faced with this, I would build another initializer for the model that takes the JSON data:
struct UserAccount {
// ...
init(jsValue: JSValue) {
id = jsonData["id"].string!
authId = jsonData["authId"].string!
emailAddress = jsonData["emailAddress"].string!
mobilePhoneNumber = jsonData["mobilePhoneNumber"].string!
firstName = jsonData["firstName"].string!
lastName = jsonData["lastName"].string!
countryCode = jsonData["countryCode"].string!
homepageUrl = jsonData["homepageUrl"].string!
tagline = jsonData["tagline"].string!
pictureUrl = jsonData["pictureUrl"].string!
accountState = convertAccountStateStringToEnum(jsonData["accountState"].string!)
}
}
Then you can simply create new UserAccount instances from your JSON data:
let userAccount = UserAccount(jsValue: jsonData)
A few thoughts:
Can you share the reference that suggests its best practice to use structs rather than classes when passing around model data? I would have thought that passing around structs (especially big ones) by value would be less efficient than passing around class instances by reference.
See Choosing between classes and structures in The Swift Programming Language: Classes and Structures which suggests you want to use structures when:
The structure’s primary purpose is to encapsulate a few relatively simple data values.
It is reasonable to expect that the encapsulated values will be copied rather than referenced when you assign or pass around an instance of that structure.
Any properties stored by the structure are themselves value types, which would also be expected to be copied rather than referenced.
The structure does not need to inherit properties or behavior from another existing type.
Your user object does not seem to match this criteria. I would have thought the class approach is more logical.
BTW, if using classes, you can also simplify the code to instantiate an object if the class is KVO compliant. For example, you can iterate through an array of key names, grabbing the value associated with each key name from the JSON and then using setValue:forKey: to set the object's property for that particular key.
For more information about KVO-compliance see the Key-Value Observing Programming Guide: KVO Compliance. For more information about how to make Swift object KVO-compliant, see Key-Value Observing section of Using Swift with Cocoa and Objective-C: Adopting Cocoa Design Patterns.
In terms of separate user models for different tasks, that doesn't makes sense for me. You might have different objects for different functional purposes (e.g. accounts/authentication objects) which might reference the user objects, but it doesn't seem to make sense to have different types of user classes/structures.