I am using Spring boot application with embedded Jetty.
Spring boot version - 2.2.1.RELEASE
Jetty Version - 9.4.25.v20191220
Applications are running on multiple hosts(Say 5 hosts) and with a load balancer in front of it.
Load balancer setting:-
Keep Alive conections - 20
Keepalive Timeout - 60s
Jetty Settings
server.jetty.connection-idle-timeout=62000 (62 Secs)
Load balancer is throwing 502 sometime with the logs below.
upstream prematurely closed connection while reading response header from upstream, client: ******, server: xyz.test.com, request: "GET /health/check HTTP/1.1", upstream: "https://XXXXX:8443/health/check", host: "xyz.test.com"
upstream prematurely closed connection while reading response header from upstream, client: ******, server: xyz.test.com, request: "GET /api/v1/weather/2485 HTTP/1.1", upstream: "https://XXXXXX:8443/api/v1/weather/2485", host: "xyz.test.com"
We are using all default setting for Spring boot embedded Jetty. Is there anything config to be changed?
We have not enabled access log on the server side and no logs related to 502 in our logs.
Related
Recently I was debugging a problem that a client reported when they were stress-testing a spring application running in tomcat. They ran the tests, but most of the requests immediately resulted in an exception, causing a 500 response code.
At some point the client was no longer able to open new connections because the ephemeral port range was exhausted - all ports were in a TIME_WAIT state. The reason appears to be that when an exception occurs the connection is closed by either spring or tomcat.
I created a simple spring boot 2.7.6 application with a single controller, with two endpoints - /5xx which throws ResponseStatusException(HttpStatus.INTERNAL_SERVER_ERROR), and /4xx
to throw ResponseStatusException(HttpStatus.BAD_REQUEST).
After requesting these endpoints, the response contains a Connection: close header, and looking at the local ports with ss the ports in TIME_WAIT get increased.
Further, requesting an endpoint that doesn't exist - doesn't close the connection and simply returns 404.
When I switched to undertow, instead of tomcat - the connection wasn't closed for all endpoints.
When I switched to spring boot 3 and tomcat - the connection isn't closed as well for all three endpoints.
So, my questions are:
What is the correct behavior? I'm thinking that the connection shouldn't be closed.
Does anyone know if this is a spring or tomcat feature and is there a configuration for this?
EDIT #1:
OS details:
Ubuntu 22.04.1 LTS
Linux 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
net.ipv4.ip_local_port_range = 1024 60999
Tomcat:
embedded tomcat v9.0.69 with default configuration provided by spring boot
the situation improves by increasing maxConnections to 30000 (default is 10000)
Most of the connections to the springboot application are in CLOSE_WAIT state , and are note being closed.
Tried configuration:
server:
servlet:
session:
timeout: 60s
Could anyone suggest a workaround
We have to connect out java camel application with an external system over https. In the middle we have a proxy, but this proxy only accepts http connections.
I have configured http and https proxies in the camel context but it seems that this does not help. The http4s component runs into connection closed exception. So I configured the proxy directly at the https4 endpoint. This configuration works but it seems that the component wants to communicate over https with our proxy and I receive this exception.
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
I inspect the debug log and I can see this log entry which indicates that the connection to the proxy is done over https
[DEBUG]: org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection request: [route: {tls}->https://<proxy>:<port>->https://<3rdPartySystem>:443][total kept alive: 0; route allocated: 0 of 20; total allocated: 0 of 200]
here the camel component configuration
to("https4:<3rdPartySystem>/services/oauth2/token?proxyAuthHost=...&proxyAuthPort=...")
So my question is: How can I configure a proxy in java in a way that https traffic is done over http between the java app and the proxy. From proxy to the 3rd party system communication should be done over https.
By the way the "old" http-camel component works perfect with the same proxy.
use proxyAuthScheme=http to avoid the SSLException
I'm trying to enable retry capability within a Zuul gateway, and am able to get things working locally, but when I deploy the gateway to PCF, I get the following error when zuul.retryable=true:
{
"timestamp": 1524669167094,
"status": 500,
"error": "Internal Server Error",
"exception": "com.netflix.zuul.exception.ZuulException",
"message": "COMMAND_EXCEPTION"
}
The related logs give me the following exception details:
com.netflix.zuul.exception.ZuulException: Forwarding error
Caused by: com.netflix.hystrix.exception.HystrixRuntimeException: spring-demo failed and no fallback available.
Caused by: org.apache.http.NoHttpResponseException: spring-demo.example.com:443 failed to respond
I've tested spring-demo.example.com and it responds correctly (200) within 200 ms and Zuul is also able to get a valid response when I remove zuul.retryable property (although then it doesn't retry any error status codes or timeouts).
When I run locally, I can see the RibbonLoadBalancedRetryPolicy try the different instances on timeout or when getting a 500 so it's only in PCF that I'm getting the error. I've verified that the instances show up in the PCF Eureka and also tried increasing the connect/read/hystrix timeouts.
Here's the service layout:
2 instances of "working" app connected to Eureka as "spring-demo"
2 instances of "broken" app connected to Eureka as "spring-demo" (times out or returns 500)
Zuul connected to Eureka
Zuul application.yml:
zuul:
ignoredServices: '*'
ignoredPatterns: '/**/actuator/**'
retryable: true
routes:
spring-demo: '/spring-demo/**'
ribbon:
retryableStatusCodes: 404, 500
MaxAutoRetries: 1
MaxAutoRetriesNextServer: 5
OkRetryOnConnectionErrors: true
Gradle dependency versions:
Spring Boot 1.5.12.RELEASE
Spring Cloud Edgware.SR3
Pivotal Services 1.6.3.RELEASE
spring-boot-starter-web
spring-boot-starter-actuator
spring-cloud-starter-netflix-zuul
spring-retry
spring-cloud-services-starter-service-registry
spring-cloud-services-starter-circuit-breaker
We are running consul in OpenShift cluster. All services have been developed by Spring Boot/Cloud APIs and they have been registered successfully in consul. There is a health point exposed using SpringBoot actuator. The health point itself works just fine when try to hit using curl.. sometimes we are just getting HTTP 200 status code and do not see any response. So which is causing Consul to throw below errors frequently which causes issues in discovering the service.
Any suggestions would be great help..
2016/08/05 05:57:15 [WARN] agent: http request failed 'http://10.1.0.18:9080/health': Get http://10.1.0.18:9080/health: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Discovered this after a long time, my solution was increasing the timeouts for the probes, not sure if this helps after 2 years but worth a shot