I've developed and deployed a Spring Boot REST application in an Amazon EC2 instance. Then I created a CloudFront distribution with behavior of "Redirect HTTP to HTTPS" and origin configuration as "HTTP Only" to be able to access the Rest API via HTTPS.
Normally, everything works fine while the app is accessed through http with EC2's self domain. But when I started to access through https CloudFront link, the application starts dying silently after a while.
What actions should I take to prevent app being terminated?
Here is what I saw in log files. But this is not the crash log, app keeps running after this log:
2020-09-09 00:09:15.877 INFO 27720 --- [nio-8080-exec-3] o.apache.coyote.http11.Http11Processor : Error parsing HTTP request header
Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in the HTTP protocol [HTTP/1.10x0aHost:]
at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:560) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:260) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.36.jar!/:9.0.36]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat-embed-core-9.0.36.jar!/:9.0.36]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590) [tomcat-embed-core-9.0.36.jar!/:9.0.36]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.36.jar!/:9.0.36]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_252]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_252]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.36.jar!/:9.0.36]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_252]
2020-09-09 11:24:35.544 INFO 27720 --- [nio-8080-exec-5] o.apache.coyote.http11.Http11Processor : Error parsing HTTP request header
Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in the HTTP protocol [HTTP/1.10x0aHost:]
at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:560) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:260) ~[tomcat-embed-core-9.0.36.jar!/:9.0.36]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.36.jar!/:9.0.36]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat-embed-core-9.0.36.jar!/:9.0.36]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590) [tomcat-embed-core-9.0.36.jar!/:9.0.36]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.36.jar!/:9.0.36]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_252]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_252]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.36.jar!/:9.0.36]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_252]
LOGBACK: No context given for c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy#532445947
That's all, no other exception, no shutdown logs, no other clue unfortunately...
I'm using following command to run the application:
sudo nohup java -Dspring.profiles.active=stage -jar myapplication.jar &
Spring Boot version is 2 and EC2 instance is Amazon Linux 2018.3
Edit: I'm adding some images about CloudFront configurations.
Origin Configurations:
Behavior Configurations:
Configuring Supported HTTP Versions of CloudFront distribution
as
HTTP/1.1, HTTP/1.0
instead of
HTTP/2, HTTP/1.1, HTTP/1.0
solved the problem.
The exception in the original post is still present, but it's no longer terminating.
I think the application serving the Rest API doesn't support HTTP 2, and when redirected from CloudFront some HTTP 2 request arrives. Then handling not supported version of a http request somehow causes some fatal errors resulting termination. Please correct me if I'm wrong.
As Steffen mentioned, HTTPS traffic is being forwarded to your application. Your Cloud Front configuration should do SSL termination and then forward the request to your application.
Related
In my current project, one of our microservices calls a REST Api of an another microservice and in an error scenario it returns a custom(469) error code.
I'm using a load balanced rest template in the calling service and its throwing an IllegalArgumentException error since 469 error code is not present in HttpStatus enum.
Is there any workaround to resolve this like using raw status codes instead of converting it into an enum?
java.lang.IllegalArgumentException: No matching constant for [469]
at org.springframework.http.HttpStatus.valueOf(HttpStatus.java:547)
at org.springframework.http.client.AbstractClientHttpResponse.getStatusCode(AbstractClientHttpResponse.java:33)
at org.springframework.cloud.client.loadbalancer.ResponseData.<init>(ResponseData.java:68)
at org.springframework.cloud.loadbalancer.blocking.client.BlockingLoadBalancerClient.getClientResponse(BlockingLoadBalancerClient.java:120)
at org.springframework.cloud.loadbalancer.blocking.client.BlockingLoadBalancerClient.execute(BlockingLoadBalancerClient.java:94)
at org.springframework.cloud.client.loadbalancer.RetryLoadBalancerInterceptor.lambda$intercept$2(RetryLoadBalancerInterceptor.java:129)
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:329)
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:225)
at org.springframework.cloud.client.loadbalancer.RetryLoadBalancerInterceptor.intercept(RetryLoadBalancerInterceptor.java:79)
at org.springframework.http.client.InterceptingClientHttpRequest$InterceptingRequestExecution.execute(InterceptingClientHttpRequest.java:93)
at org.springframework.boot.actuate.metrics.web.client.MetricsClientHttpRequestInterceptor.intercept(MetricsClientHttpRequestInterceptor.java:86)
at org.springframework.http.client.InterceptingClientHttpRequest$InterceptingRequestExecution.execute(InterceptingClientHttpRequest.java:93)
at org.springframework.cloud.sleuth.instrument.web.mvc.TracingClientHttpRequestInterceptor.intercept(TracingClientHttpRequestInterceptor.java:69)
at org.springframework.cloud.sleuth.instrument.web.client.LazyTraceClientHttpRequestInterceptor.intercept(LazyTraceClientHttpRequestInterceptor.java:51)
at org.springframework.http.client.InterceptingClientHttpRequest$InterceptingRequestExecution.execute(InterceptingClientHttpRequest.java:93)
at org.springframework.http.client.InterceptingClientHttpRequest.executeInternal(InterceptingClientHttpRequest.java:77)
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:66)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:776)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:670)
I think we can manipulate the status coming from downstream service using interceptors.
Have you tried ClientHttpRequestInterceptor? Here is an example
Fixed in Spring cloud 2021.0.2 and above
https://github.com/spring-cloud/spring-cloud-commons/issues/1066
To use raw status codes instead of converting into an enum add the following property
spring.cloud.loadbalancer.use-raw-status-code-in-response-data: true
I have a vanilla Spring Boot web application, with some controllers, REST services etc. There is no notion of thread pools, shutdown hooks etc. in the webapp's sources. The app itself does not use any other services, like external databases.
The app worked well until it started to die within two or three days on average after being started. In the logs everything looks normal until this lonely message:
--- [extShutdownHook] o.s.s.concurrent.ThreadPoolTaskExecutor : Shutting down ExecutorService 'taskExecutor'
that has the last timestamp, even if shown in the logs a bit out of chronological order. The message is preceded by this:
java.lang.IllegalArgumentException: Invalid character found in the HTTP protocol [HTTP/1.10x0aHost:]
at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:559) ~[tomcat-embed-core-9.0.43.jar!/:9.0.43]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:261) ~[tomcat-embed-core-9.0.43.jar!/:9.0.43]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.43.jar!/:9.0.43]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:887) ~[tomcat-embed-core-9.0.43.jar!/:9.0.43]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1684) ~[tomcat-embed-core-9.0.43.jar!/:9.0.43]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.43.jar!/:9.0.43]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[na:na]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.43.jar!/:9.0.43]
at java.base/java.lang.Thread.run(Thread.java:832) ~[na:na]
which in turn has no time stamp, so it is difficult to say how these two are related or if they are related at all.
After that, the app no longer responds to requests. Where can I look for causes?
The log fragment you have quoted is not a cause but it is an effect of application beeing gracefully shut down
--- [extShutdownHook]
says this is a hook set on a JVM exit routine. So when, for example, you send a kill singal to the JVM, hooks set via setShutdownHook() will be called - and this is exactly what you see - thread pool is shutdown because JVM is requested to be terminated.
Now, I cannot state why it is terminated, but it is possible that you run out of resoureces (for whatever reasons) and OS (linux?) is killing the most resource consuming process. It is also possible that OS is killing long running processes due to some settings (happenned to me in the past that hoster was killing long runing process)
If you are using scope as provided for tomcat dependency in pom.xml. Try removing that.
It might be the reason of graceful shutdown of tomcat when it finds ideal threads.
--- [extShutdownHook] o.s.s.concurrent.ThreadPoolTaskExecutor : Shutting down ExecutorService 'taskExecutor'
I'm facing below exception when I try to establish a secure connection between Zuul gateway and microservice. Both Zuul and microservice are registered to secure Service Registry Server(Eureka). Able to access the REST endpoints exposed in the microservicevia Zuul gateway. But,whenever any REST endpoint call is invoked, getting SSLHandshakeException in Zuul gateway.
NOTE: Generated wildcard certificate and the certificate is imported into "$JAVA_HOME/jre/lib/security/cacerts" of all microservices. Using embedded Jetty server.
javax.net.ssl.SSLHandshakeException: Unrecognized SSL message, plaintext connection?
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.fill(SslConnection.java:793)
at org.eclipse.jetty.server.HttpConnection.fillRequestBuffer(HttpConnection.java:322)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:231)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:278)
at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:148)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at sun.security.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:156)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:868)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.fill(SslConnection.java:642)
... 15 common frames omitted
Can anyone suggest a solution how to resolve this?
I have mircroservice environment based on spring-boot, where i am using zipkin server and discovery-server(eureka) and config-server. Now i have a rest-microservice which sends logs to zipkin server and this microservice is required to resolve where is zipkin server using discovery-server.
following is zipkin configuration i have in my rest-microservice's application.properties(pulled from config-server).
spring.zipkin.baseUrl=http://MTD-ZIPKIN-SERVER/
spring.zipkin.locator.discovery.enabled=true
spring.zipkin.enabled=true
...
here MTD-ZIPKIN-SERVER is zipkin-server name in discovery-server.
discovery-server dashboard.
but it does not try to resolve zipkin from discovery-server, instead it tries to connect directly using spring.zipkin.baseUrl, and i get below exception.
Dropped 1 spans due to ResourceAccessException(I/O error on POST request for "http://MTD-ZIPKIN-SERVER/api/v1/spans":
MTD-ZIPKIN-SERVER; nested exception is java.net.UnknownHostException:
MTD-ZIPKIN-SERVER)
org.springframework.web.client.ResourceAccessException: I/O error on
POST request for "http://MTD-ZIPKIN-SERVER/api/v1/spans":
MTD-ZIPKIN-SERVER; nested exception is java.net.UnknownHostException:
MTD-ZIPKIN-SERVER at
org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:666)
at
org.springframework.web.client.RestTemplate.execute(RestTemplate.java:628)
at
org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:590)
at
org.springframework.cloud.sleuth.zipkin.RestTemplateSender.post(RestTemplateSender.java:73)
at
org.springframework.cloud.sleuth.zipkin.RestTemplateSender.sendSpans(RestTemplateSender.java:46)
at
zipkin.reporter.AsyncReporter$BoundedAsyncReporter.flush(AsyncReporter.java:245)
at
zipkin.reporter.AsyncReporter$Builder.lambda$build$0(AsyncReporter.java:166)
at zipkin.reporter.AsyncReporter$Builder$$Lambda$1.run(Unknown
Source) at java.lang.Thread.run(Thread.java:745) Caused by:
java.net.UnknownHostException: MTD-ZIPKIN-SERVER at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
if i provide exact zipkin url in property spring.zipkin.baseUrl like below
spring.zipkin.baseUrl=http://localhost:5555/
then my rest-microservice is able to connect to zipkin-server.
My goal here is to read zipkin-server location from discovery-srever. What am i doing wrong? Do i need to add some zipkin enabling annotation on my spring-boot rest-microservice?
This feature is available in edgware release train. That corresponds to version 1.3.x of sleuth
The Grails Project that I am working on has the war deployed to mulitple Tomcats with Session Replication enabled.
We have made little changes recently related to sessions except for starting to use useToken in all g:form entities. (There may be others, but an initial look at all the recent code submissions could not find anything related.)
I understand that the error below is related to session objects not being serializable, but I am not sure where to begin looking.
Does anyone have experience with the error below in Grails?
Could it be related to the useToken in the forms?
Is this a known limitation of useToken in g:forms - that they do not support serialization?
Alternately, how can I find all object in the session to see which one is causing this error?
org.apache.catalina.ha.session.DeltaManager.requestCompleted Unable to serialize delta request for sessionid [6F6A26B3FF57A901F5D868FB68CA4A6F]
java.io.NotSerializableException: groovy.lang.MapWithDefault
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
at org.apache.catalina.ha.session.DeltaRequest$AttributeInfo.writeExternal(DeltaRequest.java:384)
at org.apache.catalina.ha.session.DeltaRequest.writeExternal(DeltaRequest.java:277)
at org.apache.catalina.ha.session.DeltaRequest.serialize(DeltaRequest.java:291)
at org.apache.catalina.ha.session.DeltaManager.serializeDeltaRequest(DeltaManager.java:617)
at org.apache.catalina.ha.session.DeltaManager.requestCompleted(DeltaManager.java:1000)
at org.apache.catalina.ha.session.DeltaManager.requestCompleted(DeltaManager.java:965)
at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:525)
at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:513)
at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:495)
at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:406)
at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:329)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:537)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1085)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:658)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:222)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1556)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1513)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Found the answer: This is a known issue with Grails 2.2.1.
See https://jira.grails.org/browse/GRAILS-9923
So essentially, if you are using Grails 2.2.1 with session replication enabled across the cluster, and are using Tokens, it will break session replication.