Spring 5 WebClient Intermittent SSLHandshakeException - spring-boot

Question

I am seeing javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure errors with Spring WebClient using Netty. The same code has been working great connecting to various endpoints for dev work. Now we are connecting to a production environment(different target hostname) and this error is occurring. Sometimes we can go for hours or days with no issue. Or sometimes right after an app restart. Then once it starts happening all requests will fail with a SSLHandshakeException. A restart of the app fixes it usually.
I appreciate any assistance. Thank you.
Environment
Spring Boot 2.5.8
Spring WebFlux 5.3.14
io.netty 4.1.72.Final
OpenJDK 11
WebClient Configuration
WebClient is configured as a Spring Bean with the following configuration.
#Bean
protected WebClient myWebClient() {
final HttpClient httpClient = HttpClient.create()
.wiretap(HttpClient.class.getCanonicalName(), LogLevel.DEBUG, AdvancedByteBufFormat.TEXTUAL, StandardCharsets.UTF_8)
.responseTimeout(Duration.ofSeconds(120));
return WebClient.builder()
.defaultHeader("sourceSystem", "source system name")
.clientConnector(new ReactorClientHttpConnector(httpClient))
.filters(exchangeFilterFunctions -> {
exchangeFilterFunctions.add(errorHandlingFilter());
})
.build();
private ExchangeFilterFunction errorHandlingFilter() {
return ExchangeFilterFunction.ofResponseProcessor(clientResponse -> {
if(clientResponse.statusCode().isError()) {
return clientResponse.bodyToMono(String.class)
.flatMap(errorBody -> Mono.error(new MyException(clientResponse.statusCode(), extractErrorMessage(errorBody))));
} else {
return Mono.just(clientResponse);
}
});
}
HTTP POST
ResponseEntity<JsonNode> resp = webClient
.post()
.uri("https://externalurl")
.contentType(MediaType.APPLICATION_FORM_URLENCODED)
.body(BodyInserters.fromValue(bodyData))
.exchangeToMono(clientResponse -> clientResponse.toEntity(JsonNode.class))
.block();
Log Messages - SUCCESS
2022-06-09 14:03:11,645 DEBUG [reactor-http-nio-2] r.netty.resources.DefaultPooledConnectionProvider: [49be2ee8-18, L:/my-ip-address:53320 - R:external-hostname/external-ip-address:443] onStateChange(POST{uri=/connect/token, connection=PooledConnection{channel=[id: 0x49be2ee8, L:/my-ip-address:53320 - R:external-hostname/external-ip-address:443]}}, [response_received])
2022-06-09 14:03:11,646 DEBUG [reactor-http-nio-2] reactor.netty.channel.FluxReceive: [49be2ee8-18, L:/my-ip-address:53320 - R:external-hostname/external-ip-address:443] FluxReceive{pending=0, cancelled=false, inboundDone=false, inboundError=null}: subscribing inbound receiver
2022-06-09 14:03:11,648 DEBUG [reactor-http-nio-2] reactor.netty.http.client.HttpClient: [49be2ee8-18, L:/my-ip-address:53320 - R:external-hostname/external-ip-address:443] READ: 5B 0
2022-06-09 14:03:11,648 DEBUG [reactor-http-nio-2] reactor.netty.http.client.HttpClientOperations: [49be2ee8-18, L:/my-ip-address:53320 - R:external-hostname/external-ip-address:443] Received last HTTP packet
2022-06-09 14:03:11,650 DEBUG [reactor-http-nio-2] r.netty.resources.DefaultPooledConnectionProvider: [49be2ee8, L:/my-ip-address:53320 - R:external-hostname/external-ip-address:443] onStateChange(POST{uri=/connect/token, connection=PooledConnection{channel=[id: 0x49be2ee8, L:/my-ip-address:53320 - R:external-hostname/external-ip-address:443]}}, [response_completed])
2022-06-09 14:03:11,651 DEBUG [reactor-http-nio-2] reactor.netty.ReactorNetty: [49be2ee8, L:/my-ip-address:53320 - R:external-hostname/external-ip-address:443] Removed handler: reactor.left.responseTimeoutHandler, pipeline: DefaultChannelPipeline{(reactor.left.sslHandler = io.netty.handler.ssl.SslHandler), (reactor.left.loggingHandler = reactor.netty.transport.logging.ReactorNettyLoggingHandler), (reactor.left.httpCodec = io.netty.handler.codec.http.HttpClientCodec), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
2022-06-09 14:03:11,652 DEBUG [reactor-http-nio-2] r.netty.resources.DefaultPooledConnectionProvider: [49be2ee8, L:/my-ip-address:53320 - R:external-hostname/external-ip-address:443] onStateChange(POST{uri=/connect/token, connection=PooledConnection{channel=[id: 0x49be2ee8, L:/my-ip-address:53320 - R:external-hostname/external-ip-address:443]}}, [disconnecting])
2022-06-09 14:03:11,653 DEBUG [reactor-http-nio-2] r.netty.resources.DefaultPooledConnectionProvider: [49be2ee8, L:/my-ip-address:53320 - R:external-hostname/external-ip-address:443] Releasing channel
2022-06-09 14:03:11,654 DEBUG [reactor-http-nio-2] reactor.netty.resources.PooledConnectionProvider: [49be2ee8, L:/my-ip-address:53320 - R:external-hostname/external-ip-address:443] Channel cleaned, now: 0 active connections, 1 inactive connections and 0 pending acquire requests.
2022-06-09 14:03:11,655 DEBUG [reactor-http-nio-3] reactor.netty.resources.PooledConnectionProvider: [0b61c34f] Created a new pooled channel, now: 0 active connections, 1 inactive connections and 0 pending acquire requests.
Log Messages - FAILURE
2022-06-09 14:03:11,657 DEBUG reactor.util.Loggers$Slf4JLogger [reactor-http-nio-3] [0b61c34f] SSL enabled using engine sun.security.ssl.SSLEngineImpl#31d4d4e6 and SNI external-hostname:443
2022-06-09 14:03:11,657 DEBUG reactor.util.Loggers$Slf4JLogger [reactor-http-nio-3] [0b61c34f] Initialized pipeline DefaultChannelPipeline{(reactor.left.sslHandler = io.netty.handler.ssl.SslHandler), (reactor.left.loggingHandler = reactor.netty.transport.logging.ReactorNettyLoggingHandler), (reactor.left.sslReader = reactor.netty.tcp.SslProvider$SslReadHandler), (reactor.left.httpCodec = io.netty.handler.codec.http.HttpClientCodec), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
2022-06-09 14:03:11,657 DEBUG io.netty.util.internal.logging.AbstractInternalLogger [reactor-http-nio-3] [0b61c34f] REGISTERED
2022-06-09 14:03:11,711 DEBUG reactor.util.Loggers$Slf4JLogger [reactor-http-nio-3] [0b61c34f] Connecting to [external-hostname/external-ip-address:443].
2022-06-09 14:03:11,712 DEBUG io.netty.util.internal.logging.AbstractInternalLogger [reactor-http-nio-3] [0b61c34f] CONNECT: external-hostname/external-ip-address:443
2022-06-09 14:03:11,754 DEBUG reactor.util.Loggers$Slf4JLogger [reactor-http-nio-3] [0b61c34f, L:/my-ip-address:53603 - R:external-hostname/external-ip-address:443] Registering pool release on close event for channel
2022-06-09 14:03:11,754 DEBUG reactor.util.Loggers$Slf4JLogger [reactor-http-nio-3] [0b61c34f, L:/my-ip-address:53603 - R:external-hostname/external-ip-address:443] Channel connected, now: 1 active connections, 1 inactive connections and 0 pending acquire requests.
2022-06-09 14:03:11,773 DEBUG io.netty.util.internal.logging.AbstractInternalLogger [reactor-http-nio-3] [0b61c34f, L:/my-ip-address:53603 - R:external-hostname/external-ip-address:443] ACTIVE
2022-06-09 14:03:11,802 DEBUG io.netty.util.internal.logging.AbstractInternalLogger [reactor-http-nio-3] [0b61c34f, L:/my-ip-address:53603 - R:external-hostname/external-ip-address:443] USER_EVENT: SslHandshakeCompletionEvent(javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure)
2022-06-09 14:03:11,811 WARN reactor.util.Loggers$Slf4JLogger [reactor-http-nio-3] [0b61c34f, L:/my-ip-address:53603 - R:external-hostname/external-ip-address:443] The connection observed an error
javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:128)
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:308)
at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:279)
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:181)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:672)
at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:627)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:443)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:422)
at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:634)
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1341)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)