How to troubleshoot protobuf issues - protocol-buffers

Question

During some load tests, in some circumstances I have:
session with status Status{code=CANCELLED, description=Failed to read message., cause=io.grpc.StatusRuntimeException: INTERNAL: Invalid protobuf byte sequence
at io.grpc.Status.asRuntimeException(Status.java:526)
at io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller.parse(ProtoLiteUtils.java:218)
at io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller.parse(ProtoLiteUtils.java:118)
at io.grpc.MethodDescriptor.parseResponse(MethodDescriptor.java:284)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:661)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:646)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).
at com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:125)
at com.google.protobuf.CodedInputStream$ArrayDecoder.readTag(CodedInputStream.java:633)
at c.n.r.m.c.v1.CMM.<init>(CMM.java:45)
at c.n.r.m.c.v1.CMM$1.parsePartialFrom(CMM.java:974)
at c.n.r.m.c.v1.CMM$1.parsePartialFrom(CMM.java:968)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:86)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:48)
at io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller.parseFrom(ProtoLiteUtils.java:223)
at io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller.parse(ProtoLiteUtils.java:215)
... 9 more
}.
The client/server are some java aps. Both have the same mvn dependencies incuding protos.
Any idea how I can debug this ? The msg is printed on the client stream onError. So I assume that it is the server that cannot decode the grpc message.
A similar issue occurs if the server is using a python grpc implementation.
I cannot put a breakpoint in the c.n.r.m.c.v1.CMM$1.parsePartialFrom(CMM.java:974)_ - yet. Some challenges from the IDE.

Answer 1

Protocol message contained an invalid tag (zero).
This means the received message was invalid. This normally happens if the message is corrupted or it wasn't a protobuf message at all. Normal schema evolution does not trigger this.