I am trying to create a grpc server streaming endpoint.
Here is my protobuf file
syntax = "proto3";
option go_package = "mirror_streampb";
option java_package = "com.mirror_stream";
option java_outer_classname = "StreamIdsProto";
option java_multiple_files = true;
package mirror_stream;
service StreamIDs {
rpc ListIDs(ListIDsRequest) returns (stream ListIDsResponse) {}
}
message ListIDsRequest {
int32 num = 1;
}
message ListIDsResponse {
int32 num = 1;
int32 id = 2;
}
And here is my golang implementation for that method. Which only returns some random numbers.
func (s *Server) ListIDs(req *streampb.ListIDsRequest, stream streampb.StreamIDs_ListIDsServer) error {
for i := int32(0); i < req.Num; i++ {
resp := &streampb.ListIDsResponse{
Num: i,
Id: int32(rand.Intn(10000000)),
}
if err := stream.Send(resp); err != nil {
return err
}
}
return nil
}
So when I try to call that method, I get this error Failed while making call: code:unknown message:grpc: client streaming protocol violation: get <nil>, want <EOF>
I am not sure why it is coming from and where it is coming from.
Can anyone help me figure this out?
Related
I have a Go server defined like this in proto file:
syntax = "proto3";
package go-package;
option go_package = "github.com/path/to/go-package";
import "vibe.proto";
service MyService {
rpc Function (stream TheRequest) returns (stream TheResponse) {}
}
message TheRequest {
oneof Payload {
Config config = 1;
Messages messages = 2;
}
}
message Config {
// config some fields here
}
message Messages {
repeated string messages = 1;
}
I'm using https://github.com/grpc-ecosystem/go-grpc-middleware as an interceptor engine. I need to implement logging of the request or Messages when I have a panic, because right now the code is panicked, but i don't have any information about the request.
Just add interceptor with recovery, simple example:
func RecoveryUnaryInterceptor(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (_ interface{}, err error) {
defer func() {
if r := recover(); r != nil {
err = status.Error(codes.Internal, fmt.Sprintf("Panic: `%s` %s", info.FullMethod, string(debug.Stack())))
}
}()
return handler(ctx, req)
}
syntax = "proto3";
package TestServer;
service RelaySrv{
rpc UpdateGroupDetails (Group) returns (Response);
}
message Person
{
int64 id = 1;
string name = 2;
}
message Group{
repeated Person persons = 1;
}
Go code:
var buf bytes.Buffer
m := jsonpb.Marshaler{}
err := m.Marshal(&buf, Group)
After doing a marshal on group protobuf message buf variable will have:
{ "persons" : [{"id":"1","name":"sun"},
{"id":"2","name":"sam"}] }
How do I extract just
[{"id":"1","name":"sun"},
{"id":"2","name":"sam"}]
from buf without emptying it??
Not sure if there is a better way but instead of marshal into json struct, it worked with encode/decode to json. The ingest.stream doesn't complain anymore and I can see the data in azure-data-explorer
//have a json struct for Group and Person to match the protbuf message posted in the question
var g Group
json.NewDecoder(&buf).Decode(&g)
var b bytes.Buffer
for i := 0; i < len(g.Persons); i++ {
e := json.NewEncoder(&b).Encode(&g.Persons[i])
if e != nil {
panic("issue marshalling protobuf")
}
}
If I understand u correctly, this is what you want.
const b = `[{"id":1,"Name":"sun"}, {"id":2,"Name":"sam"}]`
persons := []*pb.Person{}
err := json.Unmarshal([]byte(b), &persons)
if err != nil {
panic(err.Error())
}
log.Println(persons)
// 2021/03/06 22:34:15 [id:1 name:"sun" id:2 name:"sam" ]
In the GetList function the client sends a message and the StreamReply returns RowValue and boolean value from a loop one by one. How can I implement it with gRPC stream?
test.proto
service GetList {
rpc Listdata (StreamRequest) returns (stream StreamReply){}
}
message StreamRequest{
string StreamName= 1;
bool boolean= 3;
}
message StreamReply{
string RowValue=1;
bool boolean = 2;
}
The problem lies in my go function. I don't understand how to simply stream values using stream.Send()
server.go
func (s *server) GetList(in *pb.StreamRequest, stream pb.GetList_ListdataServer) error {
strings:= "random strings"
count := 5
for i := 0; i < count; i++ {
if err := stream.Send(strings, true); err != nil {
return err
}
}
return nil
}
I am implementing a simple grpc service where the summary of a task is to be sent to the grpc server. Everything works fine if I send less number of messages but when I begin to send like 5000 messages the server stops and gets deadline exceeded message in client side. I also tried to reconnect again but found the error message as.
rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: timed out waiting for server handshake
The server shows no error and is alive.
I tried setting GRPC_GO_REQUIRE_HANDSHAKE=off as well but the error still prevails. I also implemented sending summary in batch but same scenerio repeated.
Is there any limitations to number of messages to be sent in grpc?
Here is my service proto
// The Result service definition.
service Result {
rpc ConntectMaster(ConnectionRequest) returns (stream ExecutionCommand) {}
rpc postSummary(Summary) returns(ExecutionCommand) {}
}
message Summary{
int32 successCount = 1;
int32 failedCount = 2;
int32 startTime = 3;
repeated TaskResult results = 4;
bool isLast = 5;
string id = 6;
}
postSummary implementation in sever
// PostSummary posts the summary to the master
func (server *Server) PostSummary(ctx context.Context, in *pb.Summary) (*pb.ExecutionCommand, error) {
for i := 0; i < len(in.Results); i++ {
res := in.Results[i]
log.Printf("%s --> %d Res :: %s, len : %d", in.Id, i, res.Id, len(in.Results))
}
return &pb.ExecutionCommand{Type: stopExec}, nil
}
func postSummaryInBatch(executor *Executor, index int) {
summary := pb.Summary{
SuccessCount: int32(executor.summary.successCount),
FailedCount: int32(executor.summary.failedCount),
Results: []*pb.TaskResult{},
IsLast: false,
}
if index >= len(executor.summary.TaskResults) {
summary.IsLast = true
return
}
var to int
batch := 500
if (index + batch) <= len(executor.summary.TaskResults) {
to = index + batch
} else {
to = len(executor.summary.TaskResults)
}
for i := index; i < to; i++ {
result := executor.summary.TaskResults[i]
taskResult := pb.TaskResult{
Id: result.id,
Msg: result.msg,
Time: result.time,
}
// log.Printf("adding res : %s ", taskResult.Id)
if result.err != nil {
taskResult.IsError = true
}
summary.Results = append(summary.Results, &taskResult)
}
summary.Id = fmt.Sprintf("%d-%d", index, to)
log.Printf("sent from %d to %d ", index, to)
postSummary(executor, &summary, 0)
postSummaryInBatch(executor, to)
}
func postSummary(executor *Executor, summary *pb.Summary, retryCount int) {
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
cmd, err := client.PostSummary(ctx, summary)
if err != nil {
if retryCount < 3 {
reconnect(executor)
postSummary(executor, summary, retryCount+1)
}
log.Printf(err.Error())
// log.Fatal("cannot send summary report")
} else {
processServerCommand(executor, cmd)
}
}
grpc default maxReceiveMessageSize is 4MB, your grpc client probably went over that limit.
grpc uses h2 in transport layer which opens only one tcp conn and multiplex "requests" over that, reduce significant overhead compare to h1, I wouldn't worry too much for batching and will just make individual calls to grpc server.
I have been trying to create a grpc client in Go and I have followed the correct instructions as shown in the official grpc site. When I start my grpc server written in node.js, the connection works well but upon compiling the protocol buffer in Go and creating a client interface with the correct grpc client configurations, I run into an error.
Here is my what I have in my identity.pb.go.
type IdentityServiceClient interface {
CreateUser(ctx context.Context, in *GoogleIdToken, opts ...grpc.CallOption) (error, *UserInfo)
}
type simpleServerClient struct {
connection *grpc.ClientConn
}
func NewSimpleServerClient(connection *grpc.ClientConn) IdentityServiceClient {
return &simpleServerClient{connection}
}
func (simpleClient *simpleServerClient) CreateUser(ctx context.Context, in *GoogleIdToken, opts ...grpc.CallOption) (error, *UserInfo) {
out := new(UserInfo)
err := simpleClient.connection.Invoke(ctx, "/protobuf.IdentityService/CreateUser", in, out, opts...)
if err != nil {
return err, nil
}
return nil, out
}
here is the identity.proto
syntax="proto3";
package protobuf;
service IdentityService {
rpc CreateUser (GoogleIdToken) returns (UserInfo) {}
}
message GoogleIdToken {
string token = 1;
}
message UserInfo {
string name = 1;
string email = 2;
message Profile {
string imageUrl = 1;
string lastUpdated = 2;
};
Profile profile = 3;
string token = 4;
}
here is my main.go
import pb "github.com/Duncanian/iam-gateway/server/protobuf"
func grpcConnection() {
conn, err := grpc.Dial("localhost:50051", grpc.WithInsecure())
if err != nil {
log.Fatalf("Failed to start gRPC connection: %v", err)
}
defer conn.Close()
client := pb.NewSimpleServerClient(conn)
err, _ = client.CreateUser(context.Background(), &pb.GoogleIdToken{Token: "tevgvybububvvg"})
if err != nil {
log.Fatalf("Failed to create user: %v", err)
}
log.Println("Created user!")
}
I expect the output of passing a correct google auth token to get me the correct user details which are
name: user,
email: user.email#user.com,
profile: {
imageUrl: myimageUrl,
lastUpdated: mylastUpdatedTime,
},
token,
but got
rpc error: code = Unimplemented desc = RPC method not implemented /protobuf.IdentityService/CreateUser
Here are my github repos:
Go grpc client &
Node JS grpc server
I had the same problem.
Here is my solution:
After compiling .proto I created 2 files: client.go and server.go.
In client.go I implemented my methods (rpc in terms of protobuffers) and had a main function.
In server.go I defined a server struct server with one field: Unimplemented*ServiceName*Server. After that I also implemented the mentioned above methods, but those had a receiver type: func (s *server) Foo(ctx context.Context, *other params*)
That worked for me, hopefully it will help you!
The error indicates that the /protobuf.IdentityService/CreateUser method is not registered at the server side. And I didn't see any service registration code in your linked server code. Please take a look at the node.js guide here.
Using GUI of grpc try to send your protos to your both servers and check for upcoming errors/correct endpoints.
In my case java proto had a package inside which was added to the endpoint.
Had package com.example.grpc;
instead of
option java_package = "com.example.grpc";