I have the following protobuf definition:
service MyService {
rpc ServiceMethod (ServiceMethodRequest) returns (ServiceMethodResponse) {}
}
message ServiceMethodRequest{
string requestParam = 1;
}
message ServiceMethodResponse{
Error error = 1;
SomeObjectList data = 2;
}
message Error{
string code = 1;
string errorMessage = 2;
}
message SomeObject {
string myobject = 1;
}
message SomeObjectList {
repeated SomeObject myobjects = 1;
}
As you can see I want to return formatted response so my API has some standardized way of responding. I formatted my response like this (this is in JSON format because of readability):
{"error":{"code":"-1","errorMessage":""},"data":{"myobjects":[{"myobject":"some string"},{"myobject":"another string"}]}}
But on the client side I am constantly getting:
{ Error: 2 UNKNOWN: Unknown Error
at Object.exports.createStatusError (.../node_modules/grpc/src/common.js:91:15)
at Object.onReceiveStatus (.../node_modules/grpc/src/client_interceptors.js:1204:28)
at InterceptingListener._callNext (.../node_modules/grpc/src/client_interceptors.js:568:42)
at InterceptingListener.onReceiveStatus (.../node_modules/grpc/src/client_interceptors.js:618:8)
at callback (.../node_modules/grpc/src/client_interceptors.js:845:24)
code: 2,
metadata: Metadata { _internal_repr: {}, flags: 0 },
details: 'Unknown Error' }
when I try to console.log response object.
What am I doing wrong?
Also, is there any good book on grpc with some examples (C, C#, Java, Python, JavaScript)?
Thanks in advance!
server.js
callback(null, message);
Related
I have 3 protos as follow:
1 - record.proto
message Record {
int64 primaryKey = 1;
int64 createdAt = 2;
int64 updatedAt = 3;
}
2 - user.proto
import "record.proto";
message User {
Record record = 31;
string name = 32;
string email = 33;
string password = 34;
}
3 - permissions.proto
import "record.proto";
message Permissions{
Record record = 31;
string permission1= 32;
string permission2= 33;
}
Question1:
Is there a way to implement a grpc server and client in golang that takes specifically Record as request but entertains both later types. i.e User and Permissions.
Something like:
service DatabaseService {
rpc Create(Record) returns (Response);
}
So that I can send grpc client request as both follow:
Create(User) returns (Response);
Create(Permissions) returns (Response);
You can use oneof which allows the client to send either User or Permissions.
Refer https://developers.google.com/protocol-buffers/docs/proto3#oneof
message Request {
oneof request {
User user = 1;
Permissions permissions = 2;
}
}
So client can fill any of them in the request.
service DatabaseService {
rpc Create(Request) returns (Response);
}
I am writing an application in Go that implements a gRPC API.
Here is the ProtoBuff specification that describes my endpoint's response message:
rpc MyEndpoint(RequestMessage) returns (ResponseMessage) {
option (google.api.http) = {
post: "/MyEndpoint"
body: "*"
};
}
message ResponseMessage {
message SubMessage {
int32 a = 1;
map<string, int32> b = 2;
}
string c = 1;
map<string, SubMessage> d = 2;
}
I am writing a test case to verify the response from the endpoint is what I expected it to be.
Here is my code to do that:
expectedResponse := ResponseMessage{
C: "string1",
D: map[string]*ResponseMessage_SubMessage{
"string2": &ResponseMessage_SubMessage{
A: 1,
B: map[string]int32{
"string3": 2,
},
},
},
}
fmt.Println("responseFromEndpoint = ", responseFromEndpoint)
fmt.Println("expectedResponse = ", expectedResponse)
assert.Equal(t, expectedResponse, responseFromEndpoint) // This is line #230 in file my_test_file.go
When I run this test-case, I get this output:
responseFromEndpoint = map[c:string1 d:map[string2:map[a:1 b:map[string3:2]]]]
expectedResponse = {{{} [] [] <nil>} 0 [] string1 map[string2:a:1 b:{key:"string3" value:2}]}
And the test case fails like this:
MyTestClass/MyTestCase: my_test_file.go:230:
Error Trace: my_test_file.go:230
Error: Not equal:
expected: ResponseMessage(ResponseMessage{state:impl.MessageState{NoUnkeyedLiterals:pragma.NoUnkeyedLiterals{}, DoNotCompare:pragma.DoNotCompare{}, DoNotCopy:pragma.DoNotCopy{}, atomicMessageInfo:(*impl.MessageInfo)(nil)}, sizeCache:0, unknownFields:[]uint8(nil), Name:"string1", SubMessages:map[string]*ResponseMessage_SubMessage{"string2":(*ResponseMessage_SubMessage)(0xc000098180)}})
actual : map[string]interface {}(map[string]interface {}{"c":"string1", "d":map[string]interface {}{"string2":map[string]interface {}{"a":1, "b":map[string]interface {}{"string3":2}}}})
How can I modify my test case so that I can ensure the endpoint responded with the data that I expected it to?
I have a method that I would like defined called FindAll which requires no parameters. ProtoC is complaining.
Expected type name.
This is for line:
rpc findAll () returns (BenchmarksList);
syntax = "proto3";
package helloWorldGRPC;
service HelloWorldGRPCService {
rpc findById (BenchmarksById) returns (Benchmarks);
rpc findAll () returns (BenchmarksList);
}
message BenchmarksById {
string id = 1;
}
message BenchmarksList {
repeated Benchmarks benchmarks = 1;
}
message Benchmarks {
string trans_id = 1;
string protocol = 2;
string database = 3;
string updated_at = 4;
string created_at = 5;
repeated Action actions = 6;
}
message Action {
string trans_id = 1;
int32 payload_length = 2;
string payload = 3;
string status = 4;
string updated_at = 5;
string created_at = 6;
}
The preferred way is to pass Empty - as tooling may recognize and optimize for that scenario. But in reality, there is nothing "special" about that type and any message - empty or otherwise - will suffice.
Something mysterious is happening for me:
What I wanted to do:
1. Save a Protocol Buffers object as SequenceFile format.
2. Read this SequenceFile text and extract the field that I need.
The mystery part is:
One field that I wanted to retrieve is always null.
Product_Perf is the field that I wanted to extract from SequencFiles that is always missing.
Here's my protocol buffers schema:
message ProductJoin {
Signals signals = 1;
int64 id = 2;
}
message Signals {
ProductPerf product_perf = 1;
}
message ProductPerf {
int64 impressions = 1;
}
Here's how I save the protocol buffers as SequenceFiles:
JavaPairRDD<BytesWritable, BytesWritable> bytesWritableJavaPairRdd =
flattenedPjPairRdd.mapToPair(
new PairFunction<Tuple2<Long, ProductJoin>, BytesWritable, BytesWritable>() {
#Override
public Tuple2<BytesWritable, BytesWritable> call(Tuple2<Long, ProductJoin> longProductJoinTuple2) throws Exception {
return new Tuple2<>(
new BytesWritable(longProductJoinTuple2._2().getId().getBytes()),
new BytesWritable(longProductJoinTuple2._2().toByteArray()));
}
}
//dump SequenceFiles
bytesWritableJavaPairRdd.saveAsHadoopFile(
"/tmp/path/",
BytesWritable.class,
BytesWritable.class,
SequenceFileOutputFormat.class
);
Below is the code that how I read the SequenceFile:
sparkSession.sparkContext()
.sequenceFile("tmp/path", BytesWritable.class, BytesWritable.class)
.toJavaRDD()
.mapToPair(
bytesWritableBytesWritableTuple2 -> {
Method parserMethod = clazz.getDeclaredMethod("parser");
Parser<T> parser = (Parser<T>) parserMethod.invoke(null);
return new Tuple2<>(
Text.decode(bytesWritableBytesWritableTuple2._1().getBytes()),
parser.parseFrom(bytesWritableBytesWritableTuple2._2().getBytes()));
}
);
I would like to know how to transform a Protobuf Any Type to the original Protobuf message type and vice versa. In Java from Message to Any is easy:
Any.Builder anyBuilder = Any.newBuilder().mergeFrom(protoMess.build());
But how can I parse that Any back to the originial message (e.g. to the type of "protoMess")? I could probably parse everything on a stream just to read it back in, but that's not what I want. I want to have some transformation like this:
ProtoMess.MessData.Builder protoMessBuilder = (ProtoMess.MessData.Builder) transformToMessageBuilder(anyBuilder)
How can I achieve that? Is it already implemented for Java? The Protobuf Language Guide says there were pack and unpack methods, but there are none in Java.
Thank you in Advance :)
The answer might be a bit late but maybe this still helps someone.
In the current version of Protocol Buffers 3 pack and unpack are available in Java.
In your example packing can be done like:
Any anyMessage = Any.pack(protoMess.build()));
And unpacking like:
ProtoMess protoMess = anyMessage.unpack(ProtoMess.class);
Here is also a full example for handling Protocol Buffers messages with nested Any messages:
ProtocolBuffers Files
A simple Protocol Buffers file with a nested Any message could look like:
syntax = "proto3";
import "google/protobuf/any.proto";
message ParentMessage {
string text = 1;
google.protobuf.Any childMessage = 2;
}
A possible nested message could then be:
syntax = "proto3";
message ChildMessage {
string text = 1;
}
Packing
To build the full message the following function can be used:
public ParentMessage createMessage() {
// Create child message
ChildMessage.Builder childMessageBuilder = ChildMessage.newBuilder();
childMessageBuilder.setText("Child Text");
// Create parent message
ParentMessage.Builder parentMessageBuilder = ParentMessage.newBuilder();
parentMessageBuilder.setText("Parent Text");
parentMessageBuilder.setChildMessage(Any.pack(childMessageBuilder.build()));
// Return message
return parentMessageBuilder.build();
}
Unpacking
To read the child message from the parent message the following function can be used:
public ChildMessage readChildMessage(ParentMessage parentMessage) {
try {
return parentMessage.getChildMessage().unpack(ChildMessage.class);
} catch (InvalidProtocolBufferException e) {
e.printStackTrace();
return null;
}
}
EDIT:
If your packed messages can have different Types, you can read out the typeUrl and use reflection to unpack the message. Assuming you have the child messages ChildMessage1 and ChildMessage2 you can do the following:
#SuppressWarnings("unchecked")
public Message readChildMessage(ParentMessage parentMessage) {
try {
Any childMessage = parentMessage.getChildMessage();
String clazzName = childMessage.getTypeUrl().split("/")[1];
String clazzPackage = String.format("package.%s", clazzName);
Class<Message> clazz = (Class<Message>) Class.forName(clazzPackage);
return childMessage.unpack(clazz);
} catch (ClassNotFoundException | InvalidProtocolBufferException e) {
e.printStackTrace();
return null;
}
}
For further processing, you could determine the type of the message with instanceof, which is not very efficient. If you want to get a message of a certain type, you should compare the typeUrl directly:
public ChildMessage1 readChildMessage(ParentMessage parentMessage) {
try {
Any childMessage = parentMessage.getChildMessage();
String clazzName = childMessage.getTypeUrl().split("/")[1];
if (clazzName.equals("ChildMessage1")) {
return childMessage.unpack("ChildMessage1.class");
}
return null
} catch (InvalidProtocolBufferException e) {
e.printStackTrace();
return null;
}
}
Just to add information in case someone has the same problem ... Currently to unpack you have to do (c# .netcore 3.1 Google.Protobuf 3.11.4)
Foo myobject = anyMessage.Unpack<Foo>();
I know this question is very old, but it still came up when I was looking for the answer. Using #sundance answer I had to answer do this a little differently. The problem being that the actual message was a subclass of the actual class. So it required a $.
for(Any x : in.getDetailsList()){
try{
String clazzName = x.getTypeUrl().split("/")[1];
String[] split_name = clazzName.split("\\.");
String nameClass = String.join(".", Arrays.copyOfRange(split_name, 0, split_name.length - 1)) + "$" + split_name[split_name.length-1];
Class<Message> clazz = (Class<Message>) Class.forName(nameClass);
System.out.println(x.unpack(clazz));
} catch (Exception e){
e.printStackTrace();
}
}
With this being the definition of my proto messages
syntax = "proto3";
package cb_grpc.msg.Main;
service QueryService {
rpc anyService (AnyID) returns (QueryResponse) {}
}
enum Buckets {
main = 0;
txn = 1;
hxn = 2;
}
message QueryResponse{
string content = 1;
string code = 2;
}
message AnyID {
Buckets bucket = 1;
string docID = 2;
repeated google.protobuf.Any details = 3;
}
and
syntax = "proto3";
package org.querc.cb_grpc.msg.database;
option java_package = "org.querc.cb_grpc.msg";
option java_outer_classname = "database";
message TxnLog {
string doc_id = 1;
repeated string changes = 2;
}