SSL Exception Tag mismatch error, with spring webflux webclient requests - spring

Question

I just converted a service to the webflux/netty stack (formerly it was mvc/undertow)
This service makes https requests to downstream services using spring webclient(formerly okHttp client).
The service is built with JDK 17 (and runs on JVM 18)
On the production environment, we are experiencing SSL errors starting after some time (15 minutes or up to 120 minutes) which increases over time. If the service is restarted, the errors are gone, until they start again later.
The exception I get:
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Tag mismatch
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:425)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:391)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: javax.net.ssl.SSLException: Tag mismatch
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:371)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:314)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:309)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:123)
at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736)
at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482)
at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679)
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1342)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
... 18 common frames omitted
Caused by: javax.crypto.AEADBadTagException: Tag mismatch
at java.base/com.sun.crypto.provider.GaloisCounterMode$GCMDecrypt.doFinal(GaloisCounterMode.java:1591)
at java.base/com.sun.crypto.provider.GaloisCounterMode.engineDoFinal(GaloisCounterMode.java:454)
at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2501)
at java.base/sun.security.ssl.SSLCipher$T12GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1659)
at java.base/sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:239)
at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:196)
at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:159)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
... 29 common frames omitted
The webclient config code:
#Bean
public WebClient webClient(MetricsWebClientCustomizer metricsCustomizer,
WebClient.Builder builder,
#Value("${s2s.client.read-timeout-ms}") int readTimeoutMs) throws SSLException {
metricsCustomizer.customize(builder);
HttpClient httpClient = createNettyHttpClient(readTimeoutMs);
WebClient webClient = builder
.clientConnector(new ReactorClientHttpConnector(httpClient))
.build();
// warming up web client to allow smooth first requests
httpClient.warmup().subscribe((ignored) -> log.info("web client initialized"));
return webClient;
}
private HttpClient createNettyHttpClient(int readTimeoutMs) throws SSLException {
final ConnectionProvider connectionProvider = ConnectionProvider
.builder("webclient-connection-provider")
.maxIdleTime(Duration.ofSeconds(poolConnectionMaxIdleTimeSeconds))
.maxConnections(maxConnections)
.pendingAcquireMaxCount(pendingAcquireMaxCount)
.build();
return HttpClient.create(connectionProvider)
.doOnConnected(
c -> c.addHandlerLast(new ReadTimeoutHandler(readTimeoutMs))
.addHandlerLast(new WriteTimeoutHandler(readTimeoutMs)))
.option(CONNECT_TIMEOUT_MILLIS, readTimeoutMs)
.compress(true)
.runOn(LoopResources.create("eventloop-webclient", nettyWorkerThreadsCount, true));
}