ALPN negotiation error from gRPC Server on java gRPC client using envoy proxy in the middle - proxy

Question

I have a java (java-11) gRPC client streamer trying to talk with a golang gRPC server, with tls and envoy proxy in the middle.
And I'm getting
UNAVAILABLE: Failed ALPN negotiation: Unable to find compatible
protocol
error from the ServerResponseObserver.
Full stacktrace:
12/05/22 18:29:29 (798) grpc-default-executor-1 SEVERE *** ERROR *** server respond with onError:
io.grpc.StatusRuntimeException: UNAVAILABLE: Failed ALPN negotiation: Unable to find compatible protocol
Channel Pipeline: [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]
at io.grpc.Status.asRuntimeException(Status.java:535)
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:487)
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:468)
at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:432)
at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:465)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
The calls do not reach the gRPC server and seem like they won't pass the envoy proxy.
envoy trace logs show a few ssl errors such as "ssl error occurred while read: WANT_READ"
configuration of the java gRPC client streamer:
ChannelCredentials credentials;
credentials = TlsChannelCredentials.newBuilder()
.trustManager(new File("<path-to-file>/cacert.pem"))
.build();
originChannel = Grpc.newChannelBuilderForAddress(host, port, credentials).build();
ClientInterceptor interceptor = new HeaderClientInterceptor();
Channel channel = ClientInterceptors.intercept(originChannel, interceptor);
stub = LogStreamerGrpc.newStub(channel);

Answer 1

Apparently, when using a proxy one need to ensure the proxy is configured correctly with h2 enabled for ALPN
from envoy documantation alpn_protocols:
alpn_protocols:
(repeated string) Supplies the list of ALPN protocols that the listener should expose. In practice this is likely to be set to one of two values (see the codec_type parameter in the HTTP connection manager for more information):
“h2,http/1.1” If the listener is going to support both HTTP/2 and HTTP/1.1.
“http/1.1” If the listener is only going to support HTTP/1.1.
There is no default for this parameter. If empty, Envoy will not expose ALPN.
For me, supporting h2 was enough.
Here is the configuration I added to envoy.yaml running on go under in order to make it work:
transport_socket:
common_tls_context:
alpn_protocols: h2