How to enabled http1.1 and http2.0 both together in a spring boot REST application in tomcat 9.0.14 - spring-boot

I have enabled http 2.0 with SSL in a spring boot 2.1.2 REST application with embedded tomcat using configuration "server.http2.enabled=true" and other SSL configurations in application.properties file. It is working fine. When a http2 client post https request, my server is processing and responding properly (java 9 http client is being used). But when the http client version chanced to 1.1, below exception is seen -
WARNING: Using incubator modules: jdk.incubator.httpclient
java.util.concurrent.ExecutionException: java.io.IOException: Engine is closed
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2022)
at KnHttp2Client.sendGet(KnHttp2Client.java:81)
at KnHttp2Client.main(KnHttp2Client.java:113)
Caused by: java.io.IOException: Engine is closed
I have tried to post https request using apache-http-client4.5.7 also still I am getting below Exception
Jan 30, 2019 4:45:12 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {s}->https://localhost:8443: The target server failed to respond
Jan 30, 2019 4:45:12 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {s}->https://localhost:8443
org.apache.http.NoHttpResponseException: localhost:8443 failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at apacheclient.KnConnPoolClient$1.run(KnConnPoolClient.java:48)
at java.base/java.lang.Thread.run(Thread.java:844)
After setting setRetryHandler(new DefaultHttpRequestRetryHandler(0, true)) got below exception <br>
org.apache.http.NoHttpResponseException: localhost:8443 failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at apacheclient.KnConnPoolClient$1.run(KnConnPoolClient.java:48)
at java.base/java.lang.Thread.run(Thread.java:844)
Working client for with http 2.0
{HttpClient httpClient = HttpClient.newBuilder()
.version(HttpClient.Version.HTTP_2)
.build();
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create("https://localhost:8443/....."))
.GET()
.build();
CompletableFuture<HttpResponse<String>> resp = httpClient.sendAsync(request, HttpResponse.BodyHandler.asString()); }
When the http client version is changed to 1.1 as below it is not working
{HttpClient httpClient = HttpClient.newBuilder()
.version(HttpClient.Version.HTTP_1_1)
.build();}
Below client code with Apache client is also not working
{
final CloseableHttpClient httpclient =
HttpClients.custom().setConnectionManager(pool).build();
final String url = "https://localhost:8443/....";
for (int i = 0; i < 1; i++) {
new Thread(new Runnable() {
public void run() {
try {
HttpGet httpGet = new HttpGet(url);
CloseableHttpResponse response =httpclient.execute(httpGet);
HttpEntity entity = response.getEntity();
System.out.println(EntityUtils.toString(entity));
EntityUtils.consume(entity);
response.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}).start();
}
I expect that the server which supports http 2.0 should support http 1.1 also.

This is a known issue in Spring Boot, see issue #15764.
Currently, only http/2 is supported when Tomcat is configured, but Jetty and Undertow are supporting both. Until this issue is addressed, you can switch to Jetty or Undertow as a workaround.

Related

SSL Exception Tag mismatch error, with spring webflux webclient requests

I just converted a service to the webflux/netty stack (formerly it was mvc/undertow)
This service makes https requests to downstream services using spring webclient(formerly okHttp client).
The service is built with JDK 17 (and runs on JVM 18)
On the production environment, we are experiencing SSL errors starting after some time (15 minutes or up to 120 minutes) which increases over time. If the service is restarted, the errors are gone, until they start again later.
The exception I get:
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Tag mismatch
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:425)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:391)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: javax.net.ssl.SSLException: Tag mismatch
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:371)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:314)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:309)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:123)
at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736)
at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482)
at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679)
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1342)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
... 18 common frames omitted
Caused by: javax.crypto.AEADBadTagException: Tag mismatch
at java.base/com.sun.crypto.provider.GaloisCounterMode$GCMDecrypt.doFinal(GaloisCounterMode.java:1591)
at java.base/com.sun.crypto.provider.GaloisCounterMode.engineDoFinal(GaloisCounterMode.java:454)
at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2501)
at java.base/sun.security.ssl.SSLCipher$T12GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1659)
at java.base/sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:239)
at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:196)
at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:159)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
... 29 common frames omitted
The webclient config code:
#Bean
public WebClient webClient(MetricsWebClientCustomizer metricsCustomizer,
WebClient.Builder builder,
#Value("${s2s.client.read-timeout-ms}") int readTimeoutMs) throws SSLException {
metricsCustomizer.customize(builder);
HttpClient httpClient = createNettyHttpClient(readTimeoutMs);
WebClient webClient = builder
.clientConnector(new ReactorClientHttpConnector(httpClient))
.build();
// warming up web client to allow smooth first requests
httpClient.warmup().subscribe((ignored) -> log.info("web client initialized"));
return webClient;
}
private HttpClient createNettyHttpClient(int readTimeoutMs) throws SSLException {
final ConnectionProvider connectionProvider = ConnectionProvider
.builder("webclient-connection-provider")
.maxIdleTime(Duration.ofSeconds(poolConnectionMaxIdleTimeSeconds))
.maxConnections(maxConnections)
.pendingAcquireMaxCount(pendingAcquireMaxCount)
.build();
return HttpClient.create(connectionProvider)
.doOnConnected(
c -> c.addHandlerLast(new ReadTimeoutHandler(readTimeoutMs))
.addHandlerLast(new WriteTimeoutHandler(readTimeoutMs)))
.option(CONNECT_TIMEOUT_MILLIS, readTimeoutMs)
.compress(true)
.runOn(LoopResources.create("eventloop-webclient", nettyWorkerThreadsCount, true));
}

okhttp 2.7.5 fixing without going to okhttp3

Got this vulnerability,
okhttp-2.7.5.jar | Reference: CVE-2021-0341 | CVSS Score: 7.5 | Category: CWE-295 | In verifyHostName of OkHostnameVerifier.java, there is a possible way to accept a certificate for the wrong domain due to improperly used crypto. This could lead to remote information disclosure with no additional execution privileges needed. User interaction is not needed for exploitation.Product: AndroidVersions: Android-8.1 Android-9 Android-10 Android-11Android ID: A-171980069
Can this be fix without going to okhttp3?
My code would be changed with all the okhttp library, all ok except this com.squareup.okhttp.MultipartBuilder to okhttp3.MultipartBody.Builder. Which is going to give me error with,
java.lang.NullPointerException: trustManager.acceptedIssuers must not be null
at okhttp3.internal.platform.Platform.buildTrustRootIndex(Platform.kt:163)
at okhttp3.internal.platform.Platform.buildCertificateChainCleaner(Platform.kt:160)
at okhttp3.OkHttpClient$Builder.sslSocketFactory(OkHttpClient.kt:754)
org.apache.camel.support.processor.DelegateSyncProcessor.process(DelegateSyncProcessor.java:65)
at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$SimpleTask.run(RedeliveryErrorHandler.java:471)
at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:193)
at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:64)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:184)
at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:398)
at org.apache.camel.component.jetty.CamelContinuationServlet.doService(CamelContinuationServlet.java:245)
at org.apache.camel.http.common.CamelServlet.service(CamelServlet.java:130)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.lang.Thread.run(Thread.java:748)
Also I'm limited to java 8.
Thanks.
It's a pretty bogus CVE in that you need to use the HostnameVerifier API directly with untrusted input to exploit. You're probably not doing that; that interface is designed for end users to plug in a custom implementation and will rarely be used by end users directly. (OkHttp sanitizes input before passing it to this API.)
If you confirm you're not using HostnameVerifier directly your vulnerability is mitigated.
But you should upgrade anyway cause OkHttp 2.x is very obsolete.
Vulnerability got fixed with this and I'm using okhttp3,
TrustManager[] certs = new TrustManager[] { new X509TrustManager() {
private X509TrustManager standardTrustManager = null;
#Override
public X509Certificate[] getAcceptedIssuers() {
return new X509Certificate[]{};
}
#Override
public void checkServerTrusted(X509Certificate[] certificates, String authType)
throws CertificateException {
if ((certificates != null) && (certificates.length == 1)) {
certificates[0].checkValidity();
} else {
standardTrustManager.checkServerTrusted(certificates, authType);
}
}
#Override
public void checkClientTrusted(X509Certificate[] certificates, String authType)
throws CertificateException {
standardTrustManager.checkClientTrusted(certificates, authType);
}
} };
But got new error,
com.example.HTTPS$HTTPSError: javax.net.ssl.SSLException: java.lang.NullPointerException
at org.apache.camel.support.processor.DelegateSyncProcessor.process(DelegateSyncProcessor.java:65)
at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$SimpleTask.run(RedeliveryErrorHandler.java:471)
at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:193)
at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:64)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:184)
at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:398)
at org.apache.camel.component.jetty.CamelContinuationServlet.doService(CamelContinuationServlet.java:245)
at org.apache.camel.http.common.CamelServlet.service(CamelServlet.java:130)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
Caused by: javax.net.ssl.SSLException: java.lang.NullPointerException
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1903)
at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1886)
Caused by: java.lang.NullPointerException
at sun.security.ssl.AbstractTrustManagerWrapper.checkServerTrusted(SSLContextImpl.java:1091)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)

Workaround for the slowness of the WebClient first request

I am using WebClient in a Spring Boot MVC 2.1 project and found that the first request made by the client takes up to 6 seconds. Subsequent requests are way faster (~30ms).
There's a closed issue in Spring's JIRA that advices using Jetty as the WebClient Http connector. I have tried that approach, improving the figures, with a ~800ms first request. This time is an improvement but it's still far from RestTemplate which usally takes <200ms.
Netty approach (5s first request):
Conf:
#Bean
public WebClient webClient() {
return WebClient.create();
}
Usage:
private final WebClient webClient;
#GetMapping(value="/wc", produces = APPLICATION_JSON_UTF8_VALUE)
public Mono<String> findWc() throws URISyntaxException {
URI uri = new URI("http://xxx");
final Mono<String> response = webClient.get().uri(uri).retrieve().bodyToMono(String.class);
return response;
}
Jetty approach (800ms first request):
Conf:
#Bean
public JettyResourceFactory resourceFactory() {
return new JettyResourceFactory();
}
#Bean
public WebClient webClient() {
ClientHttpConnector connector = new JettyClientHttpConnector(resourceFactory(), null);
return WebClient.builder().clientConnector(connector).build();
}
Usage: same as before.
There's another "problem" with the Jetty approach. On server shutdown it always produces the following exception:
27-Dec-2018 11:24:20.463 INFO [jetty-http#74305db9-65] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [org.eclipse.jetty.io.ManagedSelector$StopSelector]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.eclipse.jetty.io.ManagedSelector$StopSelector]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1348)
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1336)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1195)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1157)
at java.lang.Class.getDeclaringClass0(Native Method)
at java.lang.Class.getDeclaringClass(Class.java:1235)
at java.lang.Class.getEnclosingClass(Class.java:1277)
at java.lang.Class.getSimpleBinaryName(Class.java:1443)
at java.lang.Class.getSimpleName(Class.java:1309)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.toString(ManagedSelector.java:534)
at java.lang.String.valueOf(String.java:2994)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.getString(EatWhatYouKill.java:458)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.toStringLocked(EatWhatYouKill.java:447)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.toString(EatWhatYouKill.java:440)
at org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:299)
at org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271)
at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233)
at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173)
at org.eclipse.jetty.util.log.JettyAwareLogger.log(JettyAwareLogger.java:680)
at org.eclipse.jetty.util.log.JettyAwareLogger.debug(JettyAwareLogger.java:224)
at org.eclipse.jetty.util.log.Slf4jLog.debug(Slf4jLog.java:97)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:288)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.lang.Thread.run(Thread.java:748)
SLF4J: Failed toString() invocation on an object of type [org.eclipse.jetty.util.thread.strategy.EatWhatYouKill]
Reported exception:
java.lang.NoClassDefFoundError: org/eclipse/jetty/io/ManagedSelector$StopSelector
at java.lang.Class.getDeclaringClass0(Native Method)
at java.lang.Class.getDeclaringClass(Class.java:1235)
at java.lang.Class.getEnclosingClass(Class.java:1277)
at java.lang.Class.getSimpleBinaryName(Class.java:1443)
at java.lang.Class.getSimpleName(Class.java:1309)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.toString(ManagedSelector.java:534)
at java.lang.String.valueOf(String.java:2994)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.getString(EatWhatYouKill.java:458)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.toStringLocked(EatWhatYouKill.java:447)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.toString(EatWhatYouKill.java:440)
at org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:299)
at org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271)
at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233)
at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173)
at org.eclipse.jetty.util.log.JettyAwareLogger.log(JettyAwareLogger.java:680)
at org.eclipse.jetty.util.log.JettyAwareLogger.debug(JettyAwareLogger.java:224)
at org.eclipse.jetty.util.log.Slf4jLog.debug(Slf4jLog.java:97)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:288)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: Illegal access: this web application instance has been stopped already. Could not load [org.eclipse.jetty.io.ManagedSelector$StopSelector]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1338)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1195)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1157)
... 25 more
Caused by: java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.eclipse.jetty.io.ManagedSelector$StopSelector]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1348)
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1336)
... 27 more
27-Dec-2018 11:24:20.467 INFO [jetty-http#74305db9-65] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [ch.qos.logback.classic.spi.ThrowableProxy]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [ch.qos.logback.classic.spi.ThrowableProxy]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1348)
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1336)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1195)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1157)
at ch.qos.logback.classic.spi.LoggingEvent.<init>(LoggingEvent.java:119)
at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:419)
at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
at ch.qos.logback.classic.Logger.log(Logger.java:765)
at org.eclipse.jetty.util.log.JettyAwareLogger.log(JettyAwareLogger.java:668)
at org.eclipse.jetty.util.log.JettyAwareLogger.warn(JettyAwareLogger.java:474)
at org.eclipse.jetty.util.log.Slf4jLog.warn(Slf4jLog.java:73)
at org.eclipse.jetty.util.log.Slf4jLog.warn(Slf4jLog.java:67)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.execute(EatWhatYouKill.java:375)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:305)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.lang.Thread.run(Thread.java:748)
Exception in thread "jetty-http#74305db9-65" java.lang.NoClassDefFoundError: ch/qos/logback/classic/spi/ThrowableProxy
at ch.qos.logback.classic.spi.LoggingEvent.<init>(LoggingEvent.java:119)
at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:419)
at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
at ch.qos.logback.classic.Logger.log(Logger.java:765)
at org.eclipse.jetty.util.log.JettyAwareLogger.log(JettyAwareLogger.java:668)
at org.eclipse.jetty.util.log.JettyAwareLogger.warn(JettyAwareLogger.java:474)
at org.eclipse.jetty.util.log.Slf4jLog.warn(Slf4jLog.java:73)
at org.eclipse.jetty.util.log.Slf4jLog.warn(Slf4jLog.java:67)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:740)
at java.lang.Thread.run(Thread.java:748)
How can I avoid this exception?
Is there any other way we can use to improve the WebClient first request slowness?
I went through the same problem and managed to solve it by changing the connector used by the WebClient.
Below the configuration file with the appropriate imports
import org.eclipse.jetty.client.HttpClient;
import org.eclipse.jetty.util.ssl.SslContextFactory;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.http.client.reactive.ClientHttpConnector;
import org.springframework.http.client.reactive.JettyClientHttpConnector;
import org.springframework.web.reactive.function.client.WebClient;
The class
SslContextFactory.Client sslContextFactory = new SslContextFactory.Client();
HttpClient httpClient = new HttpClient(sslContextFactory);
httpClient.setFollowRedirects(false);
httpClient.setConnectTimeout(TIMEOUT);
httpClient.start();
final ClientHttpConnector connector = new JettyClientHttpConnector(httpClient);
final WebClient webClient = WebClient.builder()
.clientConnector(connector)
.baseUrl(BASE_URL)
.build();
It is important to know how to add the right libs to your project, so below is how I managed to import using gradle:
implementation 'org.eclipse.jetty:jetty-client'
implementation 'org.eclipse.jetty:jetty-reactive-httpclient'
We upgraded Spring Boot to 2.4.2 with reactor-netty 1.0.3 but still encounter this issue.
Here is our upgraded configuration:
#Bean
public WebClient createWebClient(WebClient.Builder webClientBuilder) {
log.info("Initializing WebClient Bean");
final int timeoutInMillis = Long.valueOf(TimeUnit.SECONDS.toMillis(timeout)).intValue();
final HttpClient httpClient = HttpClient.create()
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, timeoutInMillis)
.responseTimeout(Duration.ofMillis(timeoutInMillis))
.doOnConnected(conn ->
conn.addHandlerLast(new ReadTimeoutHandler(timeoutInMillis, TimeUnit.MILLISECONDS))
.addHandlerLast(new WriteTimeoutHandler(timeoutInMillis, TimeUnit.MILLISECONDS)));
final ClientHttpConnector connector = new ReactorClientHttpConnector(httpClient);
final WebClient webClient = webClientBuilder
.clientConnector(connector)
.defaultHeader("x-clientname", clientname)
.build();
httpClient.warmup().block();
log.info("WebClient initialized");
return webClient;
}
Our call with WebClient:
ResoponseObject doCall() {
return this.webClient
.get()
.uri("http://***.de/api/rest/***")
.accept(MediaType.APPLICATION_JSON)
.retrieve()
.bodyToMono(ResponseObject.class)
.block();
}
With debugging enabled via application.yaml:
logging.level.reactor.netty: debug
We now see this during application startup:
2021-02-10 17:02:31,922 INFO d.t.e.b.c.c.WebClientAutoConfiguration - Initializing WebClient Bean
2021-02-10 17:02:31,959 DEBUG r.n.r.DefaultLoopIOUring - Default io_uring support : false
2021-02-10 17:02:31,967 DEBUG r.n.r.DefaultLoopEpoll - Default Epoll support : true
2021-02-10 17:02:31,997 INFO d.t.e.b.c.c.WebClientAutoConfiguration - WebClient initialized
This should be an indication that the warmup works as expected?
But on first request this happens:
2021-02-10 17:05:16,045 DEBUG o.s.w.r.f.c.ExchangeFunctions - [73d400c8] HTTP GET http://***.de/api/rest/***
2021-02-10 17:05:16,050 DEBUG r.n.r.PooledConnectionProvider - Creating a new [http] client pool [PoolFactory{evictionInterval=PT0S, leasingStrategy=fifo, maxConnections=500, maxIdleTime=-1, maxLifeTime=-1, metricsEnabled=false, pendingAcquireMaxCount=1000, pendingAcquireTimeout=45000}] for [***.de/<unresolved>:80]
2021-02-10 17:05:29,619 DEBUG r.n.r.DefaultPooledConnectionProvider - [id: 0x71b840f4] Created a new pooled channel, now 1 active connections and 0 inactive connections
2021-02-10 17:05:29,635 DEBUG r.n.t.TransportConfig - [id: 0x71b840f4] Initialized pipeline DefaultChannelPipeline{(reactor.left.httpCodec = io.netty.handler.codec.http.HttpClientCodec), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
...
In our case, creating the client pool is the problem. On a decent machine, it takes about 13 seconds.
Can you give us any comment on that? This is very this is very frustrating for us.
Thanks a lot!

Getting connection timed out error in microservice

I have built a microservice using Java 8 and SpringBoot 2. From this microservice, I'm trying to consume another REST API service. However, I'm getting the following error on Chrome. I have already disabled the windows firewall and McAfee antivirus firewall but still getting the same error. I can call the REST API directly using the Postman tool but not through my microservice.
Error:-
java.lang.IllegalStateException: The underlying HTTP client completed
without emitting a response.
2018-06-12 15:21:29.300 ERROR 17996 --- [ctor-http-nio-3]
.a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request
[GET http://localhost:8080/category/search]
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection
timed out: no further information: test.usdemo.xyz.com/92.54.41.24:443
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
~[na:1.8.0_171] at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
~[na:1.8.0_171] at
io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:325)
~[netty-transport-4.1.24.Final.jar:4.1.24.Final] at
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
~[netty-transport-4.1.24.Final.jar:4.1.24.Final] at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633)
~[netty-transport-4.1.24.Final.jar:4.1.24.Final] at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
~[netty-transport-4.1.24.Final.jar:4.1.24.Final] at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
~[netty-transport-4.1.24.Final.jar:4.1.24.Final] at
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
~[netty-transport-4.1.24.Final.jar:4.1.24.Final] at
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
~[netty-common-4.1.24.Final.jar:4.1.24.Final] at
java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_171] Caused by:
java.net.ConnectException: Connection timed out: no further
information ... 10 common frames omitted
Controller class:-
#RestController
public class CategorySearchController {
private final CategorySearchService categorySearchService;
#Autowired
public CategorySearchController(CategorySearchService categorySearchService) {
this.categorySearchService = categorySearchService;
}
#GetMapping(path = "/search-category")
public Mono<CategoryResponse> searchCategories(SearchRequest categorySearchRequest){
return categorySearchService
.searchCategories()
.switchIfEmpty(Mono.error(
new EntityNotFoundException("No category matching " + categorySearchRequest.getSearchTerm() + " was found")));
}
}
Service class:-
#Service
public class CategorySearchServiceImpl implements CategorySearchService {
private String baseUrl = "https://demo0954903.mockable.io";
#Override
public Mono<CategoryResponse> searchCategories() {
WebClient webClient = WebClient.create(baseUrl);
return webClient.
get()
.uri("/category-search")
.retrieve()
.bodyToMono(CategoryResponse.class);
}
}
I found the solution for this issue. I need to add the proxy in the webclient as follows:-
private final WebClient webClient;
ReactorClientHttpConnector connector = new ReactorClientHttpConnector(
options -> options.httpProxy(addressSpec -> {
return addressSpec.host(PROXY_HOST).port(PROXY_PORT);
}));
this.webClient = webClientBuilder.baseUrl(BASE_URL).clientConnector(connector).build();

Getting connection timed out error in Java microservice

I have built a microservice using Java 8 and SpringBoot 2. From this microservice, I'm trying to consume another REST API service. However, I'm getting the following error on Chrome
java.lang.IllegalStateException: The underlying HTTP client completed
without emitting a response.
2018-06-12 15:21:29.300 ERROR 17996 --- [ctor-http-nio-3]
.a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request
[GET http://localhost:8080/category/search]
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection
timed out: no further information: test.usdemo.xyz.com/92.54.41.24:443
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_171]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
~[na:1.8.0_171]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:325)
~[netty-transport-4.1.24.Final.jar:4.1.24.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
~[netty-transport-4.1.24.Final.jar:4.1.24.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633)
~[netty-transport-4.1.24.Final.jar:4.1.24.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
~[netty-transport-4.1.24.Final.jar:4.1.24.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
~[netty-transport-4.1.24.Final.jar:4.1.24.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
~[netty-transport-4.1.24.Final.jar:4.1.24.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
~[netty-common-4.1.24.Final.jar:4.1.24.Final]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_171] Caused by: java.net.ConnectException: Connection timed
out: no further information
... 10 common frames omitted
I'm able to consume the same service successfully using PostMan but not through my microservice.
Please assist to advise on this.
I do not know how to edit my own question above as I do not see the edit option hence I'm adding the additional details here to get the resolution of my query.
Controller class:-
#RestController
public class CategorySearchController {
private final CategorySearchService categorySearchService;
#Autowired
public CategorySearchController(CategorySearchService categorySearchService) {
this.categorySearchService = categorySearchService;
}
#GetMapping(path = "/search-category")
public Mono<CategoryResponse> searchCategories(SearchRequest categorySearchRequest){
return categorySearchService
.searchCategories()
.switchIfEmpty(Mono.error(
new EntityNotFoundException("No category matching " + categorySearchRequest.getSearchTerm() + " was found")));
}
}
Service class:-
#Service
public class CategorySearchServiceImpl implements CategorySearchService {
private String baseUrl = "https://demo0954903.mockable.io";
#Override
public Mono<CategoryResponse> searchCategories() {
WebClient webClient = WebClient.create(baseUrl);
return webClient.
get()
.uri("/category-search")
.retrieve()
.bodyToMono(CategoryResponse.class);
}
}
I found the solution for this issue. I need to add the proxy in the webclient as follows:-
private final WebClient webClient;
ReactorClientHttpConnector connector = new ReactorClientHttpConnector(
options -> options.httpProxy(addressSpec -> {
return addressSpec.host(PROXY_HOST).port(PROXY_PORT);
}));
this.webClient = webClientBuilder.baseUrl(BASE_URL).clientConnector(connector).build();
Check your proxy settings. In Spring applications, you can add the following to the VM arguments to set your proxy: -Dhttp.proxyHost=your.proxy.net -Dhttp.proxyPort=8080
I met a similar issue on a scala project. According to the error message, I found IntelliJ took Ant as the compiler which I was expected sbt as the compiler.
Solve the issue by turn on sbt shell use for builds option (Preferences > Build, Execution, Deployment > Build Tools > sbt > sbt projects > sbt shell > check builds )
Executing pre-compile tasks...
Loading Ant configuration...
Running Ant tasks...
Abnormal build process termination:
{remove some sensitive logs here}
io.netty.channel.AbstractChannel$AnnotatedConnectException: Operation timed out: /127.0.0.1:50154
Caused by: java.net.ConnectException: Operation timed out
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:707)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.lang.Thread.run(Thread.java:748)
Exiting.
Executing post-compile tasks...
Loading Ant configuration...
Running Ant tasks...
Synchronizing output directories...
2021/9/15, 8:52 AM - Build completed with 1 error and 0 warnings in 27 sec, 726 ms

Resources