Since we switched to http/2 we see a lot of ClientAbortException. Either Broken Pipe or Connection reset by peer.
The setup is Client <-> HAProxy <-> Spring Boot 2 application with embedded Tomcat 8
Is this normal behavior? If not, what would be the best way to find out why the connection is broken?
Related
The issue
I've stumbled upon the issue:
Error message: org.springframework.web.reactive.function.client.WebClientRequestException: Connection prematurely closed BEFORE response; nested exception is reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response
General info about the issue
It's a Spring Boot app (2.4.5) running on Reactive (WebFlux) stack.
The app also uses Playtika OSS reactive Feign Client (starter 3.0.3) for synchronious REST API communication.
Underlying web client is Netty.
There are no any special Feign or WebClient configs in the app.
All the other microservice parties are running on embedded Tomcat with default Spring Boot autoconfigurations.
All apps are running in Kubernetes cluster.
The error log observed from time to time (not every day).
Guesses
After some investigation, my best guess would be that some long-lived connections are being dropped from the pool on certain conditions. This causing the error log.
This thought is based on Instana that connects the error log to the span that spans accross a lot of subcalls.
Also no data losses/ other inconsistencies were noticed so far xD
Questions
Does Feign has a connection pool by default?
How to know if those are live or idle connections from the pool being closed?
How the connection pool can be configured or dropped to avoid long-running connections ?
Is it possible that Kubernetes can somehow close these connections?
What else can close connections?
We are using Websphere Applicaion Server with Spring JMS, Around 25 applications connect to a IBM MQ. Off-late we are seeing lots of stale connection on the MQ channel to which all these applications connect.
By stale connections I mean the connection are not being used for many days and the application keeps creating new connections. We are not able to identify which application creates these connections that are not being used but they all use the same framework code
WAS version = 8.5.5
Spring = 4.1.2
The Spring jms:listener-container has the following configuration
connection-factory = org.springframework.jms.connection.DelegatingConnectionFactory
acknowledge=auto
concurrency=2-10
Any pointers on any configurations that can be done on QueueConnectionFactory (JMS Resource) on WAS or on the spring side would be helpful.
I know I have not given much information, but the problem is that there are no errors / exceptions, the application creates these connections to the MQ channel and all connections gets cleared when the server is restarted
Adding one more question
We use org.springframework.jms.connection.DelegatingConnectionFactory , for replying back, Does it make sense for us to close the session once we have send back the message?
Thanks in Advance
Charlie
I haven't found any solution about this. It seems Jetty doesn't support this feature yet. I might be wrong so please, enlighten me.
I've got a very simple Java client which connects to a Java server at localhost:8080. I would like to add a transparent proxy between them in order to simulate what we could find in a company's private network.
Update: May, 2017
Starting in Jetty 9.4.0 and onwards, the native Jetty WebSocketClient supports Proxies via the Jetty HttpClient.
This works by declaring an HttpClient, along with its proxy configurations, and then handing that off to the WebSocketClient constructor to use.
This only works with the following:
HTTP/1.1 upgrade to WebSocket
Native Jetty WebSocket APIs
This does not work with the following:
HTTP/2 (there is no spec for WebSocket over HTTP/2 as of yet)
JSR356 javax.websocket (there are ideas for API breaking changes to the JSR356 ClientContainer to allow passing in a Jetty HttpClient via a constructor, let us know if this is viable option for you by filing a new issue on github saying so)
Original Answer
With Jetty 9, there's no Proxy support for either the Jetty Native WebSocket client, or the JSR-356 (javax.websocket) client implementation.
This support is scheduled for Jetty 10 (which is tracking Servlet 4), and will result in a complete reworking of the entire client library suite in Jetty to have equal support for :
HTTP/1.1
HTTP/2 (native/direct)
HTTP/1.1 upgrade to HTTP/2 (h2c)
HTTP/1.1 upgrade to WebSocket
HTTP/2 websocket channel (currently in draft specs)
Proxy support
Cookie support
etc ...
The existing WebSocket client implementations on Jetty are standalone, due to the JSR-356 support requirements.
The existing WebSocket client does not leverage the existing Jetty HttpClient in Jetty 9.x. If it did then proxy support could, maybe, under a very limited set of scenarios, work.
This is a low priority feature request, as there are few existing proxies out there that support WebSocket so far (actually, they have generally bad support for HTTP/1.1 upgrade). Even Jetty's own Server side Proxy does not currently support HTTP/1.1 upgraded connections.
According to Figure 2 in How HTML5 Web Sockets Interact With Proxy Servers, if you are trying to use a transparent proxy, you don't have to require a proxy support on the client side. On the other hand, an explicit proxy requires client libraries to support proxy.
Jetty WebSocket client won't have any problem if your proxy is transparent.
Can somebody please let me know how i can add long polling and other fallback support for websocket using Jetty 9?
I generally recommend using something like CometD (http://www.cometd.org) which will be releasing version 3 fairly soon which will support Jetty 9. Another option would be to look at the Atmosphere project.
Using a messaging framework like these let you have the framework handle finding the best protocol to use, be it SPDY, Websocket, HTTP/1.1 or even polling HTTP/1.0....and they isolate you from future protocols like HTTP/2 which is coming (slowly) which is based on spdy. Using CometD once that protocol lands and becomes available get it for free, no changes needed in your app.
Hi there i am wondering if i create a webapp that uses Spring AMQP. Is that single webapp 1 AMQP client? Or is every request made by a user that results into an AMQP call a client, so potentially x numbers of clients?
I don't know AMQP much, but I suspect it has the same terminology as jms. In that sense your application is probably pooling connections to AMQP broker for better performance. Each connection in a pool is treated as a separate client (competing consumer).
Thus each request is not really creating a new connection (client), but your application isn't a single client as well. In fact, when your application tries to access AMQP broker, it picks any connection from the pool and puts it back once it's done. Another request can reuse the same connection (client) or use a different, idle one.