Can you assign multiple different value types to one field in a repeated Protobuf message? - protocol-buffers

I'm trying to reverse engineer a client that uploads an audio file to a server, and then uploads the file's metadata in a separate request afterwards. The metadata is serialized in Protobuf, and it uses a fairly simple and readable structure. Here's what it looks like after protoc --decode_raw:
1 {
1: "title"
2: "This is the audio title"
}
1 {
1: "tags"
2 {
}
}
1 {
1: "loc"
2: "This is the location"
}
1 {
1: "transcription"
2: "A transcript"
}
1 {
1: "mapping"
2 {
1 {
1: 6
2 {
3: 840
}
}
2: 6
}
}
It seems to be just a repeated message at field 1 that holds a key-value pair each time, but sometimes the value is a string, and sometimes it's a more complex message. How can they assign both a string and a message to field 2 if Protobuf only lets you use one value type per field? If I'm going to craft my own request, I need something like this:
message KeyValuePair {
string key = 1;
oneof value {
string str_value = 2;
MessageValue msg_value = 2;
}
}
But that doesn't work, because Field number 2 has already been used in "package.name" by field "str_value". Any ideas? I'll be using Python to create and send the request data.

There is an official way to achieve this: google.protobuf.Any
If a protobuf schema defines an any at the top level, e.g.:
message Root {
repeated google.protobuf.Any value = 1;
}
message Title {
string title= 2;
}
message Tags {
string name = 1;
repeated string tags = 2;
}
Then messages of any Protobuf-defined type can be serialized in a list.
However, I don't think that's what the existing code is doing:
the raw output for Any normally includes the type.googleapis.com type url
Using Any, the title / loc fields would be encapsulated in a nested object, not a string at the same level.
E.g.:
1 {
1: "type.googleapis.com/Title"
2 {
1: "the title"
}
}

Related

Is there a way to use a proto oneof field as a type in another message?

Suppose I have a proto message like this:
message WorkflowParameters {
oneof parameters {
WorkflowAParams a = 1;
WorkflowBParams b = 2;
}
}
And I want to have another message where the type of workflow can be specified. Something like this:
message ListWorkflowsRequest {
// The type of workflows to fetch
WorkflowParameters.parameters workflow_type = 1;
}
The above doesn't work (it throws "WorkflowParameters.parameters" is not a type.) What's the recommended way of doing this?
It's not possible. oneof is only a thin syntatic sugar/behavior change, and has no effect on the actual schema. It affects the generated code's behavior, but not the serialized format. In the following example, these two messages are interchangeable and wire-compatible:
message WorkflowParameters {
oneof parameters {
WorkflowAParams a = 1;
WorkflowBParams b = 2;
}
}
message WorkflowParameters2 {
WorkflowAParams a = 1;
WorkflowBParams b = 2;
}
Now, if you just want to specify which part of a oneof will be set, you could theoretically use the generated code constants, and a simple int field:
message ListWorkflowsRequest {
// The field number of WorkflowParameters that should be filled.
int32 workflow_type = 1;
}
All language generators should have convenient enough constants created, like WorkflowParameters::A_FIELD_NUMBER for C++.

Sending ListBlog response in unary streaming

I am trying to return list of items from grpc without streaimg API and not finding any solution or example , I know we use this kind of proto file when using streaming
message ListBlogResponse {
Blog blog = 1;
}
and then Blog
message Blog {
string id = 1;
string author_id = 2;
string title = 3;
string content = 4;
}
but I want to send response once without using any streaming some thing like this :
return &api.ListBlogResponse{
Blog: items,
}, nil
what will be the protofile for this ?
You need a message containing multiple blogs:
message BlogEntries {
repeated Blog blog = 1;
}
Then you can return:
return &BlogEntries{Blog:entries},nil

How to add message type as object in ProtoBuf (gRPC) - Proto3 Syntax?

How to send message type as object in ProtoBuf - Proto3 Syntax?
I want to transfer object instead of string or number.
Example
{
name: 'One',
date: 'date',
some: 'some',
...
...
}
syntax = "proto3";
package db;
service Proxy
{
rpc myFunction(Request) returns (Response);
}
message Request
{
string name = 1;
}
message Response
{
object keyvalue = 1;
}
Here, I am getting error
throw Error("no such Type or Enum '" + path + "' in " + this);
^
Error: no such Type or Enum 'object' in Type
--
Workaround
I can convert it to string in server side, and then I can JSON.parse() at client.
But I want to know, if there any better way to do it.
protocol-buffer does not support the object data type!
But you can emulate your data as hierarchically by using protocol buffer message type itself.
syntax = "proto3";
package db;
service Proxy
{
rpc myFunction(Request) returns (Response);
}
message Request
{
string name = 1;
}
message Response
{
message obj {
string key1 = 1;
string key2 = 2
}
obj keyvalue = 1; // Here you have created your own type obj.
}
In the above example, you can see that the Response message now has "keyvalue" field of type obj(which is a custom type you have just built).
Now you will pass Object in a callback from the server instead of primitive type.
callback(null, { keyvalue: { key1: "value1", key2: "value2" } });
Let's say if keys are unknown to you but key/value pair data type is same and known to you then, in this case, you can use
map<type, type>
message Response
{
map<string, string> keyvalue = 1;
}
callback(null, { keyvalue: { "key1": "value1", "key5": "value2" } });
References:-
https://developers.google.com/protocol-buffers/docs/proto3#other
https://developers.google.com/protocol-buffers/docs/proto#scalar
https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Struct
Depending on what you're trying to accomplish, you can likely use one of three types.
The bytes type represents an arbitrary string of bytes. You can choose to encode your types in anyway you feel appropriate.
The any type represents an arbitrary protobuf message, allowing the client or server to encode non-predefined messages. This is a string of bytes, with a unique identifier per message type and language implementations will have ways to unpack the message.
You mention using Javascript. You could likely define a Javascript object message using a combination of oneof for the basic types and map for object types.
If all of the keys and values you want to send have the same type, you can use a map type. Your example shows string keys and string values, so that could use the type map<string, string>.
If you want to send arbitrary object data with similar structure to JSON objects, you can use the google.protobuf.Struct message type.

grpc and protobuf with variable keys

I have a hash of key values pairs.
I do not have the keys, nor the values.
I can assume most will be there, but sometimes keys are removed and others are added. Is there any way for me to have a message with variable keys?
{
"knownkey1": "value",
"knownkey2": {
"unknown-key1": "value",
"unknown-key2": "value"
}
}
or is the best ways to just serialize it using json stringify in the message? i would think that would defeat the whole purpose of using grpc.
message Rate {
string ticker = 1;
string value = 2;
}
message GetAllResponse {
string lastUpdated = 1;
repeated Rate payload = 2;
}
Looks like you can just use the maps type as outlined here:
https://developers.google.com/protocol-buffers/docs/proto3#maps

Overriding "root" object value of GraphQL Union/Interface type

I have an Apollo GraphQL service that delegates to an internal gRPC service. This service has an endpoint which returns a message that contains a oneof, which I'm mapping to a Union in GraphQL.
This is straightforward, but there's a fair degree of boilerplate involved when implementing the resolvers. Suppose I have the following protobuf message definition:
message MyUnionMessage {
oneof value {
UnionType1 type1 = 1;
UnionType1 type2 = 3;
UnionType1 type3 = 4;
}
}
message UnionType1 {<type 1 props>}
message UnionType2 {<type 2 props>}
message UnionType3 {<type 3 props>}
My corresponding GraphQL schema looks something like this:
union MyUnionType = UnionType1 | UnionType2 | UnionType3
type UnionType1 {<type 1 props>}
type UnionType1 {<type 2 props>}
type UnionType1 {<type 3 props>}
In the javascript binding for gRPC, a MyUnionMessage object will have two properties: value which is a string indicating which type of value is contained, and a property named for the type. So, if I had a MyUnionMessage containing a UnionType2, for example, the object would look like this:
{
value: 'type2',
type2: {...}
}
This is nice for implementing __resolveType, since I can do a simple switch on the value in value, but I then have to write a resolver for all of the fields of all of the concrete types.
What I'm looking for is to be able to so something like this:
resolvers = {
MyUnionType: {
__resolveType(obj) {
switch(obj.value) {
case 'type1': return 'UnionType1';
case 'type2': return 'UnionType2';
case 'type3': return 'UnionType3';
default: return null;
},
__resolveValue(obj) {
return obj[obj.value];
},
},
};
Basically, I want to write a "resolver" at the level of the generic union (or interface) type that transforms the object before it's passed to the concrete resolver.
Is such a thing possible?
I'd wager that this sort of scenario is typically solved by transforming the data before it hits the __resolveType logic. For example, say you had a Query field that returned a list of MyUnionType. Your resolver for that field might look something like:
function resolve (arr) {
return arr.map(obj => {
return {
...obj[obj.value]
type: obj.value // or whatever field name that won't cause a collision
}
})
}
You then switch on type inside of __resolveType and you're good to go. Of course, that means if you have multiple fields that return a MyUnionType, you'll want to extract that logic into a utility function that can be used by each resolver.
I don't think there's not really a way to do what you're trying to do with the existing API. You could, of course, do something like this:
const getUnionType(obj) {
switch(obj.value) {
case 'type1': return 'UnionType1';
case 'type2': return 'UnionType2';
case 'type3': return 'UnionType3';
default: {
throw new Error(`Unrecognized type ${obj.value}`)
}
}
}
const resolvers = {
MyUnionType: {
__resolveType(obj) {
const type = getUnionType(obj)
Object.assign(obj, obj[obj.value])
return type
},
},
};
This works, but keep in mind it is a bit fragile since it assumes resolveType will always get the same root value as the resolve function, which could hypothetically change in the future.

Resources