Proto2 encode/decode issues after adding new message - protocol-buffers

I'm fairly new to protocol buffers but have been trying to learn them as a means of sending data via MQTT. So far, I've been fine with creating proto messages and compiling them for python runtime, until I started to notice incompatibility between versions of my protobufs.
When I add a message type (no changes to existing messages/fields) to my server-side proto definitions without updating my client side proto definitions, decoding messages sent to the server give me non-deterministic results.
Here is an example of what I'm talking about:
Client proto:
message Wrapper {
optional uint32 id = 1;
optional string name = 2;
oneof payload {
Event event = 3;
Command command = 4;
}
}
message Event {
uint32 event_id = 1;
oneof event_payload {
LoginEvent login_event = 2;
LogoffEvent logoff_event = 3;
}
}
Server Proto:
message Wrapper {
optional uint32 id = 1;
optional string name = 2;
oneof payload {
Event event = 3;
Command command = 4;
}
message Event {
uint32 event_id = 1;
oneof event_payload {
LoginEvent login_event = 2;
LogoffEvent logoff_event = 3;
NewUserEvent new_user_event = 4;
}
}
I will encode and send a message from the client:
message Wrapper {
id = 12345;
name = John;
event = {
login_event = ...
}
}
And will decode the message on the server and get:
message Wrapper {
id = 12345;
name = John;
event = {
logoff_event = ...
}
}
NOTE: The decoded message type isn't deterministic and changes between messages
Can someone explain why adding an event type seems to screw up decode? Or any best-practices I should obey to improve version compatibility? Thanks in advance!

This may be a side-effect of the Python implementation rather than a bug.
You don't include the code you're using to (un)marshal the protobufs.
In your example, does logoff_event contain a LogoffEvent message type?
You say non-deterministic? Do you see different event_payload messages types on the server for the same message sent by the client?
What is returned by:
msg = Wrapper() # Your decoded message
assert msg.event.WhichOneof("event_payload")
# or
assert msg.event.HasField("login_event")
See Python Generated Code: OneOf
Update 210927
I'm unable to repro the behavior you observe; it works as expected for me.
Client:
import client_pb2
msg = client_pb2.Wrapper()
msg.id=0
msg.name="Test"
msg.event.event_id=1
msg.event.login_event.y="Hello Freddie"
print(msg)
f=open("message","wb")
f.write(msg.SerializeToString())
f.close()
Yields:
id: 0
name: "Test"
event {
event_id: 1
login_event {
y: "Hello Freddie"
}
}
Server:
import server_pb2
msg = server_pb2.Wrapper()
f=open("message","rb")
msg.ParseFromString(f.read())
f.close
assert msg.event.WhichOneof("event_payload")
assert msg.event.HasField("login_event")
print(msg)
Yields:
id: 0
name: "Test"
event {
event_id: 1
login_event {
y: "Hello Freddie"
}
}

Related

MassTransit StateMachine - How to publish messages, where the message type can be identified only at runtime

I have StateMachine, After each EventCompletion and State Changes. It publishes one or more Commands using StateMachine.Context.Publish like below.
if (req.GetType() == typeof(ValuationRun_WfReq))
{
var obj = PreparePayLoad<ValuationRun_WfReq>(wfTask, req, context.Instance.CorrelationId);
obj.CorrelationId = context.Instance.CorrelationId;
await context.Publish(obj);
}
else if (req.GetType() == typeof(MonthEndStep2_WfReq))
{
var obj = PreparePayLoad<MonthEndStep2_WfReq>(wfTask, req, context.Instance.CorrelationId);
obj.CorrelationId = context.Instance.CorrelationId;
await context.Publish(obj);
}
else if (req.GetType() == typeof(MonthEndStep5_WfReq))
{
var obj = PreparePayLoad<MonthEndStep5_WfReq>(wfTask, req, context.Instance.CorrelationId);
obj.CorrelationId = context.Instance.CorrelationId;
await context.Publish(obj);
}
As each published message should go to its own queue. I have to expicitly add the If Conditions to check the Type and Publish with that Type.
This way I have more than 20 Type and adding If Conditions for each message type.
This I have to change to identify required Message Type from AppSettings and publish the right queue.

Why has methods are not generated using protoc 3.17.* if in protobuf repo they say it should?

Reading here: https://github.com/protocolbuffers/protobuf/blob/master/docs/field_presence.md#go-example the Golang example:
m := GetProto()
if (m.HasFoo()) {
// Clear the field:
m.Foo = nil
} else {
// Field is not present, so set it.
m.Foo = proto.Int32(1);
}
if I use:
protoc pkg/user.proto --go_out=. --go_opt=module=my_app --go-grpc_out=. --go-grpc_opt=module=my_app
with:
syntax = "proto3";
package example;
message MyPlayer {
uint64 id = 1;
optional string description = 2;
uint32 qty = 3;
optional uint64 age = 4;
}
it doesn't generate any has<T>() method.
Why?
// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
// protoc-gen-go v1.26.0
// protoc v3.17.3
Am I wrong if I use MyPlayer generated proto fields instead of methods?
Example:
if MyPlayer.description != nil {
description = *MyPlayer.description
}
instead of
description = MyPlayer.GetDescription()
which is not what I want (I want to detect nil values).
That docs is wrong, as reported here: https://github.com/golang/protobuf/issues/1336:
The documentation on https://github.com/protocolbuffers/protobuf/blob/master/docs/field_presence.md#go-example is incorrect. Using "optional" in proto3 makes the field be generated just as it would in proto2.
That doc is wrong. There are no Has methods in the generated code. Test for presence by comparing fields to nil.
Rewriting those examples correctly:
// Field foo does not have presence.
// If field foo is not 0, set it to 0.
// If field foo is 0, set it to 1.
m := GetProto()
if m.Foo != 0 {
// "Clear" the field:
m.Foo = 0
} else {
// Default value: field may not have been present.
m.Foo = 1
}
// Field foo has presence.
// If foo is set, clear it.
// If foo is not set, set it to 1.
m := GetProto()
if m.Foo != nil {
// Clear the field:
m.Foo = nil
} else {
// Field is not present, so set it.
m.Foo = proto.Int32(1)
}
PR to fix that doc:
protocolbuffers/protobuf#8788

Unmarshal custom types with jsonpb

What's the best way to convert this json object to protobuf?
JSON:
{
"name": "test",
"_list": {
"some1": { "value": 1 },
"some2": [
{ "value": 2 },
{ "value": 3 },
]
}
}
Proto:
message Something {
string name = 1;
message ListType {
repeated string = 1;
}
map<string, ListType> _list = 2;
}
Without having the _list in the message I would use jsonpb.Unmarsal, but I can't think of a way to define the Unmarshaler interface on a type that is generated in a diff package.
I also thought of having _list as a Any (json.RawMessage) and handle it after the Unmarshal (but can't make this to work; err message: Any JSON doesn't have '#type')
With _list being inconsistent (not just a list of strings/map of values/etc) and you mentioning you looked into using Any you could consider making your message:
message Something {
string name = 1;
google.protobuf.Struct _list = 2;
}
https://github.com/golang/protobuf/blob/master/ptypes/struct/struct.proto
With that you can marshal/unmarshal json to/from proto messages using github.com/golang/protobuf/jsonpb which is actually designed for use with the grpc gateway but you can use it too

How to write text format when proto including google::protobuf::Any

My proto is something like:
message PlatformConfig {
google.protobuf.Any source_adapter_config = 1;
};
and I want to set proto:
message SessionBundleSourceAdapterConfig {
SessionBundleConfig config = 1;
}
into source_adapter_config, How to write the this TextFormat of protobuf.
Finally, I found that Any type need a type_url to specify type, for example:
platform_configs {
key: "tensorflow"
value {
source_adapter_config {
type_url:"type.googleapis.com/tensorflow.serving.SavedModelBundleSourceAdapterConfig"
value: "..."
}
}
}

Opentok sending signal ios

I got a problem with OpenTok, I got a session of OTSession and I want to call the method signalWithType so I can send a chat message.
In the start I have
var session : OTSession?
And then in my method where I want to send chat message from textField I get the error 'Could not find memember 'signalWithType'
func textFieldShouldReturn(textField: UITextField) -> Bool {
self.view.endEditing(true)
let message = sendMessageField.text
sendMessageField.text = ""
var type = ""
var maybeError : OTError?
session?.signalWithType(type, string: message, connection: nil, error: maybeError)
if let error = maybeError {
println(error)
} else {
println("besked blev sendt")
}
return false
}
I can't find out why it says it as I pretty sure I got the right types and that.
I have not have other problems with calling methods from session..

Resources