I have a spring boot application (2.2.6.RELEASE) which makes call to other Spring boot service through
org.springframework.web.reactive.function.client.WebClient which I build in the following manner :
final SslContext sslContext = buildSslContext(keyStorePassword, trustStore, keyStore);
final HttpClient httpClient = HttpClient.create().secure(ssl -> ssl.sslContext(sslContext));
return WebClient.builder().clientConnector(new ReactorClientHttpConnector(httpClient.wiretap(true)))
.exchangeStrategies(loggingExchangeStrategy()).build();
Normally this works fine. However I when I started performance testing the application using jmeter and bombarding it with requests, I randomly started getting failed responses with the following exception :
org.apache.http.MalformedChunkCodingException: Bad chunk header: {"order":{"status" {"status":"Success"}}}
at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:274)
at org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:222)
at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:183)
at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135)
at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:148)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.readResponse(HTTPSamplerBase.java:1905)
at org.apache.jmeter.protocol.http.sampler.HTTPAbstractImpl.readResponse(HTTPAbstractImpl.java:476)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:668)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:66)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1281)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1270)
at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:630)
at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:558)
at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:489)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256)
at java.lang.Thread.run(Thread.java:748)
I also then got a similar exception while normally testing the application with the following exception :
"org.apache.http.ConnectionClosedException: Premature end of chunk coded message body: closing chunk expected\r\n\tat org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:268)\r\n\tat org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:227)\r\n\tat org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:186)\r\n\tat org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137)\r\n\tat sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)\r\n\tat sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)\r\n\tat sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)\r\n\tat
Since the exception dont seem to be from my code it seems that the problem is with WebClient. Is it?
It seems that it has something to do with Java, MalformedChunkCodingException
If this is the case then the issue seems to be the "transfer-encoding" header key with the value "chunked" and the HttpClient. Any idea anyone?
Also just for experimenting how can I override and downgrade or upgrade the HttpClient used by WebClient in the pom?
Related
I have one issue in PRD. we recently released a springboot application and it has REST API exposed. Mobile/web APP call a legacy spring application which is in spring [not sprintboot] and it is a web applicationwhich then routes and makes a call to the these failing apis in new springboot. We are seeing timeout exception for these apis only .
there are lots of other OUTBOUND api calls made from spring legacy web application to other applications eg : login API [which has apis heavy traffic but these legacy apis work well and call other legacy applications.
There are no exception/error in logs in springboot application which has these REST API exposed. Infact we only see timeout in spring web application -meaning connection is exhausted but that does not explain why other apis OUTBOUND call are not failing which use same wrapper HTTPClient. Those which fail with timeout dont have request logs in springboot [ obviously because they dont leave spring web application tomcat JVM and die there due to timeout ]
So if we say connection pool is exhausted, the other heavey traffic OUTBOUnd calls should also face same issue but we dont see that.
All API call OUTWARD use HTTPCLient [apache.]
Not clear what is causing issue. I also explicitly defined below in new springboot for server side [I just did it to see if that makes difference but in vain]:
server:
tomcat:
connection-timeout: 10s
max-connections: 20000
max-threads: 500
min-spare-threads: 10
tomcat Log at spring web applicaiton [caller]:
org.apache.http.conn.ConnectionPoolTimeoutException
org.apache.http.conn.ConnectionPoolTimeoutException.Timeout waiting for connection from pool
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:313)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:279)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:191)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:221)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:165)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:140)
at
Any inputs?
Code snippet of Wrapper HTTPClient :
SSLContext sslContext = SSLContexts.createDefault();
HostnameVerifier hostnameVerifier = new DefaultHostnameVerifier();
SSLConnectionSocketFactory secureSSLConnectionSocketFactory = new SSLConnectionSocketFactory(
sslContext,
sslProtocolsArray,
ciphersArray,
hostnameVerifier);
ConnectionSocketFactory nonSecureConnectionSocketFactory = PlainConnectionSocketFactory.getSocketFactory();
Registry<ConnectionSocketFactory> socketFactoryRegistry = RegistryBuilder
.<ConnectionSocketFactory>create()
.register("https", secureSSLConnectionSocketFactory)
.register("http", nonSecureConnectionSocketFactory)
.build();
securePoolingConnectionManager = new PoolingHttpClientConnectionManager(socketFactoryRegistry);
securePoolingConnectionManager.setMaxTotal(this.connectionMgrMaxTotalSecure);
securePoolingConnectionManager.setDefaultMaxPerRoute(this.connectionMgrMaxPerRouteSecure);
SocketConfig secureSocketConfig = SocketConfig
.custom()
.setSoKeepAlive(true)
.setTcpNoDelay(true)
.build();
secureHttpsClient = HttpClients
.custom()
.setSSLSocketFactory(secureSSLConnectionSocketFactory)
.setConnectionManager(securePoolingConnectionManager)
.setDefaultRequestConfig(secureRequestConfig)
.setDefaultSocketConfig(secureSocketConfig)
.disableAutomaticRetries()
.build();
Stacktrace after above is just failing at wrapper HTTPClient method where call is invoked :
protected String execute(HttpClient httpclient, HttpRequestBase http) throws IOException {
String result;
ResponseHandler<String> responseHandler = new BasicResponseHandler();
result = httpclient.execute(http, responseHandler);
return result;
}
So I have to dig in another wrapper which was also using this HTTP pool and was being used in our legacy which was leaking. Closing this. Fortunately there was pool statistics api exposed so that I can see leased connection count which confirmed leaking. Since this second wrapper was used rerely and we had used in this release this was suspect and removing it solved the issue. It is another matter to dig that wrapper and find out how the pool was handled but the cause was caught.
Fault tolerance #Asynchronous-annotation on a REST client cause a core dump on Windows and the following error on MacOS:
*** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message transform method call failed at ./src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 873
Affected version: (Open Liberty 20.0.0.6/wlp-1.0.41.cl200620200528-0414) on OpenJDK 64-Bit Server VM, version 11.0.8+10-LTS
Used features:
[appSecurity-3.0, beanValidation-2.0, cdi-2.0, concurrent-1.0, distributedMap-1.0, ejbLite-3.2, el-3.0, jaxb-2.2, jaxrs-2.1, jaxrsClient-2.1, jdbc-4.2, jndi-1.0, jpa-2.2, jpaContainer-2.2, json-1.0, jsonb-1.0, jsonp-1.1, monitor-1.0, mpConfig-1.4, mpFaultTolerance-2.1, mpHealth-2.2, mpMetrics-2.3, mpOpenAPI-1.1, mpRestClient-1.4, requestTiming-1.0, servlet-4.0, ssl-1.0, transportSecurity-1.0].
To re-produce:
#ApplicationScoped
#RegisterRestClient(baseUri = "https://postman-echo.com")
public interface TestingClient {
#GET
#Asynchronous
#Path("delay/4")
#Consumes(value = MediaType.APPLICATION_JSON)
CompletionStage<Response> doesItCrashWithDelay(String dummyData);
Inject it:
#Inject
#RestClient
private TestingClient testingClient;
Use it:
#GET
#Path("doesitcrash")
public void doesItCrash() {
final Response response = testingClient.doesItCrashWithDelay("dummydata").toCompletableFuture().join();
logger.info(response.readEntity(String.class));
}
Workaround is to have another CDI bean invoking the rest client which have the fault tolerance annotations. But according to this blog post the REST client interface should be able to have fault tolerance annotations:
https://openliberty.io/blog/2020/06/04/asynchronous-programming-microprofile-fault-tolerance.html
Are #Asynchronous allowed on REST client that is already async due to CompletionStage? As mentioned, all other annotations like #Timeout and #Retry seems to work.
First, you are 100% correct that you don't need the #Asynchronous annotation on the MP Rest Client interface method - per the MP Rest Client specification, a return type of CompletionStage makes it asynchronous. If you remove the #Asynchronous annotation, it should work.
While investigating the JVM error message, I came across this helpful post that indicates that this message means the JVM encountered a super large exception - probably a StackOverflowError. My guess is that the error occurred because there are now two different asynchronous mechanisms (MP Rest Client and MP Fault Tolerance) playing together - and probably not playing nicely. Without seeing the exception stack trace, we won't know for sure.
I would first suggest removing the annotation and verifying that that works - that is probably a better workaround than using a separate CDI bean. Next, I would suggest opening an issue at https://github.com/OpenLiberty/open-liberty/issues to investigate a better overall solution.
I've implemented a service which makes ReST calls out to other services to implement part of its functionality. I'm using the reactive WebClient for this, something like:
webClient.post()
.uri(....)
.contentType(....)
.accept(....)
.header(....)
.syncBody(someRequestObject)
.exchange()
.flatMap(someResponseHandler::handleResponse)
.doOnError(throwable -> {
// do interesting things depending on throwable
})
.retry(1, this::somePredicateDependingOnThrowable);
Now... I handle HTTP statuses in someResponseHandler::handleResponse, but what I really want to know is, what other kinds of exception/error to expect from the exchange() - i.e.
what exceptions/errors do I get if I can't connect to the downstream service at all?
What exceptions/errors do I get if the connection attempt times out?
What exceptions/errors do I get if I can connect but then the request times out before I get a response?
None of these are HTTP status codes, obviously - but I can't find any documentation to tell me what I can look for. Am I just not looking in the right places? I've had a look through the documentation for the reactive WebClient, and I've had a look through the Reactor Netty Reference Guide, but no luck.
For background, this is important because we do HATEOAS-based service discovery - for some of these exceptions, I want to trigger rediscovery, for some of them, I don't.
I recommend testing your code that uses the WebClient to see how it handles the various scenarios you mentioned. You can test your code against something like MockWebServer easily from unit tests. MockWebServer can simulate most of the errors mentioned here.
Having said that, here's what I have seen in my testing when using WebClient with the ReactorClientHttpConnector. Other connectors may throw slightly different exceptions, but will likely share a super class in the exception class hierarchy as those mentioned below.
Unknown host
java.net.UnknownHostException
Connection refused (port not open on server)
java.net.ConnectException (or subclass)
reactor-netty throws io.netty.channel.AbstractChannel$AnnotatedConnectException
Connect timeout
If you have configured a connect timeout, then you will receive java.net.ConnectException (or subclass)
reactor-netty throws io.netty.channel.ConnectTimeoutException
SSL handshake errors
javax.net.ssl.SSLHandshakeException (or subclass)
Request body encoding error
This varies by the encoder being used, but generally will be org.springframework.core.codec.EncodingException (or subclass)
Some encoders also throw java.lang.IllegalStateException if encoding is configured incorrectly
Response body decoding error
This varies by the decoder being used, but generally will be org.springframework.core.codec.DecodingException (or subclass)
Some decoders also throw java.lang.IllegalStateException if decoding is configured incorrectly
Read Timeout
If using reactor-netty, and you configured a io.netty.handler.timeout.ReadTimeoutHandler, then io.netty.handler.timeout.ReadTimeoutException
If you use the .timeout operator somewhere in the reactive stream call chain, then java.util.concurrent.TimeoutException
Write Timeout
If using reactor-netty, and you configured a io.netty.handler.timeout.WriteTimeoutHandler, then io.netty.handler.timeout.WriteTimeoutException
Connection closed by server prematurely (before response completes)
java.io.IOException (or subclass)
reactor-netty throws reactor.netty.http.client.PrematureCloseException
Others
Any exceptions that occur during your someResponseHandler::handleResponse
I have one question regarding Spring WebClient
In my application I need to do many similar API calls, sometimes I need change headers in the calls (Authentication token). So the question arises, what would be better of the two options:
To create one WebClient for all incoming requests to MyService.class, by making it private final field, like code below:
private final WebClient webClient = WebClient.builder()
.baseUrl("https://another_host.com/api/get_inf")
.defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE)
.defaultHeader(HttpHeaders.ACCEPT, MediaType.APPLICATION_JSON_VALUE)
.build();
Here arises another question: is WebClient thread-safe? (because service is used by many threads)
To create new WebClient for each new request incoming to service class.
I want to provide maximum performance, and to use it in right way, but I don't know how WebClient works inside it, and how it expects to be used.
Thank you.
Two key things here about WebClient:
Its HTTP resources (connections, caches, etc) are managed by the underlying library, referenced by the ClientHttpConnector that you can configure on the WebClient
WebClient is immutable
With that in mind, you should try to reuse the same ClientHttpConnector across your application, because this will share the connection pool - this is arguably the most important thing for performance. This means you should try to derive all WebClient instances from the same WebClient.create() call. Spring Boot helps you with that by creating and configuring for you a WebClient.Builder bean that you can inject anywhere in your app.
Because WebClient is immutable it is thread-safe. WebClient is meant to be used in a reactive environment, where nothing is tied to a particular thread (this doesn't mean you cannot use in a traditional Servlet application).
If you'd like to change the way requests are made, there are several ways to achieve that:
configure things in the builder phase
WebClient baseClient = WebClient.create().baseUrl("https://example.org");
configure things on a per-request basis
Mono<ClientResponse> response = baseClient.get().uri("/resource")
.header("token", "secret").exchange();
create a new client instance out of an existing one
// mutate() will *copy* the builder state and create a new one out of it
WebClient authClient = baseClient.mutate()
.defaultHeaders(headers -> {headers.add("token", "secret");})
.build();
From my experience, if you are calling an external API on a server you have no control over, don't use WebClient at all, or use it with the pooling mechanism turned off. Any performance gains from connection pooling are greatly overweighed by the assumptions built into the (default reactor-netty) library that will cause random errors on one API call when another was abruptly terminated by the remote host, etc. In some cases, you don't even know where the error occurred because the calls are all made from a shared worker thread.
I made the mistake of using WebClient because the doc for RestTemplate said it would be deprecated in the future. In hindsight, I would go with regular HttpClient or Apache Commons HttpClient, but if you are like me and already implemented with WebClient, you can turn off the pooling by creating your WebClient as follows:
private WebClient createWebClient(int timeout) {
TcpClient tcpClient = TcpClient.newConnection();
HttpClient httpClient = HttpClient.from(tcpClient)
.tcpConfiguration(client -> client.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, timeout * 1000)
.doOnConnected(conn -> conn.addHandlerLast(new ReadTimeoutHandler(timeout))));
return WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(httpClient))
.build();
}
*** Creating a separate WebClient does not mean that WebClient will have a separate connection pool. Just look at the code for HttpClient.create - it calls HttpResources.get() to get the global resources. You could provide the pool settings manually but considering the errors that occur even with the default setup, I don't consider it worth the risk.
I have registered handler for the SockJS service as
#Bean
public SimpleUrlHandlerMapping handlerMapping() {
SockJsService sockJsService = new DefaultSockJsService(sockJsTaskScheduler());
Map<String, Object> urlMap = new HashMap<String, Object>();
urlMap.put("/sockjs/cobrowse/agent/**", new SockJsHttpRequestHandler(sockJsService, coBrowseSockJsAgentHandler()));
urlMap.put("/sockjs/cobrowse/customer/**", new SockJsHttpRequestHandler(sockJsService, coBrowseSockJsCustomerHandler()));
urlMap.put("/sockjs/cobrowse/admin/**", new SockJsHttpRequestHandler(sockJsService, coBrowseSockJsAdminHandler()));
SimpleUrlHandlerMapping hm = new SimpleUrlHandlerMapping();
hm.setOrder(-1);
hm.setUrlMap(urlMap);
return hm;
}
But when I try calling from client using one of registered URLs, it throws an exception!
org.springframework.web.socket.support.ExceptionWebSocketHandlerDecorator - ERROR - Closing due to exception for SockJS session id=8cf_1l_6
java.lang.IllegalArgumentException: WebSocketHandler not found for SockJS session id=8cf_1l_6
at org.springframework.util.Assert.isTrue(Assert.java:65)
at org.springframework.web.socket.support.PerConnectionWebSocketHandler.getHandler(PerConnectionWebSocketHandler.java:91)
at org.springframework.web.socket.support.PerConnectionWebSocketHandler.handleTransportError(PerConnectionWebSocketHandler.java:97)
at org.springframework.web.socket.support.ExceptionWebSocketHandlerDecorator.handleTransportError(ExceptionWebSocketHandlerDecorator.java:79)
at org.springframework.web.socket.support.WebSocketHandlerDecorator.handleTransportError(WebSocketHandlerDecorator.java:64)
at org.springframework.web.socket.support.LoggingWebSocketHandlerDecorator.handleTransportError(LoggingWebSocketHandlerDecorator.java:66)
at org.springframework.web.socket.sockjs.transport.session.AbstractSockJsSession.delegateError(AbstractSockJsSession.java:188)
at org.springframework.web.socket.sockjs.transport.session.AbstractSockJsSession.tryCloseWithSockJsTransportError(AbstractSockJsSession.java:255)
at org.springframework.web.socket.sockjs.transport.session.WebSocketServerSockJsSession.initializeDelegateSession(WebSocketServerSockJsSession.java:122)
at org.springframework.web.socket.sockjs.transport.handler.SockJsWebSocketHandler.afterConnectionEstablished(SockJsWebSocketHandler.java:72)
at org.springframework.web.socket.adapter.StandardWebSocketHandlerAdapter.onOpen(StandardWebSocketHandlerAdapter.java:93)
at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.init(WsHttpUpgradeHandler.java:130)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:653)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:223)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1584)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1543)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
This is even more surprising because same code was running till yesterday -No major changes on my side. My guess is, it's something to do with spring jars, may be nightly update -I am using "4.0.0.BUILD-SNAPSHOT", but then I do not have previous libs!
Posting it here, if someone have came across this, meanwhile I am trying to get hold of old libs to try out - i do not know how to get it from official repo (Older version of same release!).
I spend couple of hours comparing my code against repository and tried to figure out what went wrong -when I was confident enough that I have not broken anything, I guessed that it might be that its due to the latest JAR. I use maven so my repo got updated today morning. Since libs are from Spring -this guess has to com last!
My guess turned out to be correct -when I replaced current version "4.0.0.BUILD-20130823.020236" with an earlier one which is "4.0.0.BUILD-20130814.154539-172" -Issue got resolved. Thanks a lot microsoft recycle bin!