Noticed that my k8s pod was unexpectedly restarting while under load after a few minutes of REST traffic. k8s showed that my probes were not responding to my HTTP health checks (q/health/live and q/health/ready). Application logs started to show the following:
2021-10-12 19:49:38,542 WARN [io.net.boo.ServerBootstrap] (vert.x-acceptor-thread-0) Failed to register an accepted channel: [id: 0xc6f17976, L:0.0.0.0/0.0.0.0:9090]: java.lang.IllegalStateException
at io.vertx.core.net.impl.VertxEventLoopGroup.next(VertxEventLoopGroup.java:37)
at io.vertx.core.net.impl.VertxEventLoopGroup.register(VertxEventLoopGroup.java:53)
at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:215)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:97)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:484)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:829)
at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:567)
at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
These are repeated over and over again until this message is logged:
2021-10-12 19:49:38,913 ERROR [io.qua.ver.cor.run.VertxCoreRecorder] (vert.x-eventloop-thread-1) Uncaught exception received by Vert.x: java.lang.IllegalStateException: Client is closed
at io.vertx.core.net.impl.NetClientImpl.checkClosed(NetClientImpl.java:175)
at io.vertx.core.net.impl.NetClientImpl.connectInternal(NetClientImpl.java:219)
at io.vertx.core.net.impl.NetClientImpl.connect(NetClientImpl.java:206)
at io.vertx.core.net.impl.NetClientImpl.connect(NetClientImpl.java:185)
at io.vertx.core.net.impl.NetClientImpl.connect(NetClientImpl.java:191)
at io.vertx.redis.client.impl.RedisConnectionManager$RedisConnectionProvider.connect(RedisConnectionManager.java:141)
at io.vertx.core.net.impl.pool.SimpleConnectionPool.connect(SimpleConnectionPool.java:240)
at io.vertx.core.net.impl.pool.SimpleConnectionPool$Remove.lambda$execute$0(SimpleConnectionPool.java:390)
at io.vertx.core.net.impl.pool.CombinerExecutor.pollAndExecute(CombinerExecutor.java:55)
at io.vertx.core.net.impl.pool.CombinerExecutor.submit(CombinerExecutor.java:43)
at io.vertx.core.net.impl.pool.SimpleConnectionPool.execute(SimpleConnectionPool.java:231)
at io.vertx.core.net.impl.pool.SimpleConnectionPool.remove(SimpleConnectionPool.java:456)
at io.vertx.core.net.impl.pool.SimpleConnectionPool.access$000(SimpleConnectionPool.java:79)
at io.vertx.core.net.impl.pool.SimpleConnectionPool$Slot.onRemove(SimpleConnectionPool.java:138)
at io.vertx.redis.client.impl.RedisStandaloneConnection.evict(RedisStandaloneConnection.java:360)
at io.vertx.redis.client.impl.RedisStandaloneConnection.end(RedisStandaloneConnection.java:323)
at io.vertx.core.net.impl.ConnectionBase.checkCloseHandler(ConnectionBase.java:389)
at io.vertx.core.impl.future.FutureImpl$3.onSuccess(FutureImpl.java:141)
at io.vertx.core.impl.future.FutureBase.lambda$emitSuccess$0(FutureBase.java:54)
at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:81)
at io.vertx.core.impl.DuplicatedContext.execute(DuplicatedContext.java:173)
at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:51)
at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211)
at io.vertx.core.impl.future.PromiseImpl.tryComplete(PromiseImpl.java:23)
at io.vertx.core.Promise.complete(Promise.java:66)
at io.vertx.core.impl.future.PromiseImpl.operationComplete(PromiseImpl.java:65)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605)
at io.netty.util.concurrent.DefaultPromise.setSuccess(DefaultPromise.java:96)
at io.netty.channel.DefaultChannelPromise.setSuccess(DefaultChannelPromise.java:78)
at io.netty.channel.DefaultChannelPromise.setSuccess(DefaultChannelPromise.java:73)
at io.vertx.core.net.impl.ConnectionBase.handleClosed(ConnectionBase.java:380)
at io.vertx.core.net.impl.NetSocketImpl.handleClosed(NetSocketImpl.java:382)
at io.vertx.core.net.impl.VertxHandler.channelInactive(VertxHandler.java:144)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:829)
at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:567)
at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
Running the same load mix against a non-native image build did NOT have the same problem.
I am using the quarkus redis client created programmatically -- i.e. not #Inject into my app, but doing the following:
RedisClient defaultRedisClient = RedisClient.createClient();
Not sure if redis has anything to do with it, but seeing vert.x in the call-stack made me think of it. Also not sure what "Failed to register an accepted channel" might mean.
If I scale-back the traffic, I don't see this problem which leads me to think the app cannot handle the load. If that is the case, seems like there should be a way to detect this condition rather than having a slew of exceptions followed by k8s restarting the pod.
My worry is that there might be some underlying issue that I'm missing "going native" as I don't see this with a non-native build.
Thanks.
Related
I am doing performance testing (HTTPS request) on the custom server using Jmeter.
But I am facing these errors,
javax.net.ssl.SSLException: Connection reset
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:127)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:325)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:268)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:263)
at java.base/sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1667)
at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1049)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.executeRequest(HTTPHC4Impl.java:939)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:650)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:66)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1301)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1290)
at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:651)
at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:570)
at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:501)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:268)
at java.base/java.lang.Thread.run(Thread.java:832)
Suppressed: java.net.SocketException: Broken pipe
at java.base/sun.nio.ch.NioSocketImpl.implWrite(NioSocketImpl.java:420)
at java.base/sun.nio.ch.NioSocketImpl.write(NioSocketImpl.java:440)
at java.base/sun.nio.ch.NioSocketImpl$2.write(NioSocketImpl.java:826)
at java.base/java.net.Socket$SocketOutputStream.write(Socket.java:1052)
at java.base/sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:82)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:356)
... 30 more
Caused by: java.net.SocketException: Connection reset
at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:323)
at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350)
at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803)
at java.base/java.net.Socket$SocketInputStream.read(Socket.java:982)
at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:469)
at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:463)
at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1421)
at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1033)
... 26 more
Also, i have followed the answers given in this link - Ignore/Turn off verification of the certificate in Jmeter/Java. But still couldn't solve this issue.
As per SSL Encryption chapter of JMeter Documentation:
The JMeter HTTP samplers are configured to accept all certificates, whether trusted or not, regardless of validity periods, etc. This is to allow the maximum flexibility in testing servers.
so it's not the case JMeter doesn't trust the certificate, it must be something else.
Try adding javax.net.debug=all line to system.properties of your JMeter installation and inspect the console output, it should give you the clue where the problem is.
More information:
Debugging SSL/TLS Connections
Apache JMeter Properties Customization Guide
If you want to "turn off SSL" completely - change "Protocol" to http and "Port" to 80 in the HTTP Request Defaults (however in that case your system under test must expose a HTTP connector on port 80)
I had missed the header called "Accept-Encoding" in the HTTP Header Manager. Adding it resolved my issue.
when starting up a quarkus service (v2.1.1), and trying to connect to a keycloak instance (v15.0.1) I am getting the following exception stack trace:
ERROR [io.qua.run.Application] (Quarkus Main Thread) Failed to start application (with profile dev): io.quarkus.oidc.common.runtime.OidcEndpointAccessException
at io.quarkus.oidc.runtime.OidcProviderClient.getJsonWebKeySet(OidcProviderClient.java:75)
at io.quarkus.oidc.runtime.OidcProviderClient.lambda$getJsonWebKeySet$0(OidcProviderClient.java:54)
at io.smallrye.context.impl.wrappers.SlowContextualFunction.apply(SlowContextualFunction.java:21)
at io.smallrye.mutiny.operators.uni.UniOnItemTransform$UniOnItemTransformProcessor.onItem(UniOnItemTransform.java:36)
at io.smallrye.mutiny.vertx.AsyncResultUni.lambda$subscribe$1(AsyncResultUni.java:35)
at io.vertx.mutiny.ext.web.client.HttpRequest$10.handle(HttpRequest.java:717)
at io.vertx.mutiny.ext.web.client.HttpRequest$10.handle(HttpRequest.java:714)
at io.vertx.ext.web.client.impl.HttpContext.handleDispatchResponse(HttpContext.java:371)
at io.vertx.ext.web.client.impl.HttpContext.execute(HttpContext.java:358)
at io.vertx.ext.web.client.impl.HttpContext.next(HttpContext.java:336)
at io.vertx.ext.web.client.impl.HttpContext.fire(HttpContext.java:303)
at io.vertx.ext.web.client.impl.HttpContext.dispatchResponse(HttpContext.java:265)
at io.vertx.ext.web.client.impl.HttpContext.lambda$null$8(HttpContext.java:520)
at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:96)
at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:59)
at io.vertx.core.impl.EventLoopContext.lambda$runOnContext$0(EventLoopContext.java:37)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:832)
Here are the configs I have put:
# OIDC Configuration
quarkus.oidc.auth-server-url=https://<HOST>/auth/realms/<REALM_NAME>
quarkus.oidc.client-id=<CLIENT_ID>
# quarkus.oidc.application-type=service
quarkus.oidc.credentials.secret=<SECRET>
quarkus.oidc.tls.verification=required
# Enable Policy Enforcement
quarkus.keycloak.policy-enforcer.enable=true
Anyone got an idea what's wrong here?
Thanks to a comment above I found this quarkus config here to be helpful:
quarkus.log.min-level=DEBUG
quarkus.log.category."io.quarkus.oidc".level=DEBUG
Which gave me this error:
Caused by: org.keycloak.authorization.client.util.HttpResponseException: Unexpected response from server: 400 / Bad Request / Response from server: {"error":"invalid_client","error_description":"Bearer-only not allowed"}
So clearly, I had misconfigured the client inside keycloak. Too bad the original error message didn't give me that information by default.
I'm using dropwizard framework in my service that using jetty as the http server.
Until today the service works fine.
Since I add squid as proxy server before Ngnix I have a weird issue.
The application logs seems like everything works fine but the client of my service got 500 in specific requests.
Only after changing jetty logs to DEBUG mode, I noticed they print error message in DEBUG mode. (Strange).
org.glassfish.jersey.message.internal.OutboundMessageContext - org.eclipse.jetty.http.BadMessageException: 500: Response header too large
java.io.IOException: org.eclipse.jetty.http.BadMessageException: 500: Response header too large
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:173)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:441)
at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.write(ResponseWriter.java:325)
at java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:167)
at org.glassfish.jersey.message.internal.CommittingOutputStream.flushBuffer(CommittingOutputStream.java:307)
at org.glassfish.jersey.message.internal.CommittingOutputStream.commit(CommittingOutputStream.java:261)
at org.glassfish.jersey.message.internal.CommittingOutputStream.close(CommittingOutputStream.java:276)
at org.glassfish.jersey.message.internal.OutboundMessageContext.close(OutboundMessageContext.java:839)
at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:412)
at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:784)
at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:444)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:434)
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:329)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
at io.dropwizard.jetty.NonblockingServletHolder.handle(NonblockingServletHolder.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689)
at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:34)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:50)
at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:44)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1174)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1106)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:240)
at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:51)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:396)
at io.dropwizard.jetty.BiDiGzipHandler.handle(BiDiGzipHandler.java:68)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:524)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.eclipse.jetty.http.BadMessageException: 500: Response header too large
at org.eclipse.jetty.http.HttpGenerator.generateResponse(HttpGenerator.java:414)
at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:690)
at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)
at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:521)
at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:692)
at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:746)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:179)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:163)
... 56 common frames omitted
Caused by: java.nio.BufferOverflowException: null
at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:189)
at java.nio.ByteBuffer.put(ByteBuffer.java:859)
at org.eclipse.jetty.http.HttpGenerator.putContentLength(HttpGenerator.java:873)
at org.eclipse.jetty.http.HttpGenerator.generateHeaders(HttpGenerator.java:785)
at org.eclipse.jetty.http.HttpGenerator.generateResponse(HttpGenerator.java:399)
... 64 common frames omitted
I added my response header size in logs and it is 0.
In addition, this endpoint works 3 times and only on the 4th time it fail. Again - application return 200 but jetty insert this error.
2020-05-18 11:04:28.747 [dw-221 - POST ------- - DEBUG - org.glassfish.jersey.tracing.general - FINISHED Response status: 200/SUCCESSFUL|OK [ ---- ms]
When working that only squid OR Ngnix in the middle of the communication between the client and my service everything works well.
When working with squid and Ngnix in the middle so it fails.
Also, another important information is that the failure occurring when my service holds the connection ~45 seconds before returning the results. Seems that the Jetty error log regarding the Response header is wrong and something else with communication happens. Any ideas?
I have a database server running Oracle 18.2 XE with and Apex 19.2 application served via ORDS.
ORDS is behind IIS 10.
The application works properly but from time to time, it's down and I get the following error message... and I have to restart the server to make it work.
Has anyone an idea on how to solve that please ? Thanks.
The connection pool named: |apex|| is not correctly configured, due to the following error(s):
Exception occurred while getting connection: oracle.ucp.UniversalConnectionPoolException:
All connections in the Universal Connection Pool are in use
[TE] GET /ords/f?p=myapp:homepage start: 2020-05-07T12:28:36.661Z duration: 3016ms
ServiceUnavailableException [statusCode=503, reasons=[The connection pool named: |apex|| is not correctly configured, due to the following error(s): Exception occurred while getting connection: oracle.ucp.UniversalConnectionPoolException: All connections in the Universal Connection Pool are in use ]]
at oracle.dbtools.http.errors.ServletResponseExceptionMapper.mapError(ServletResponseExceptionMapper.java:79)
at oracle.dbtools.http.errors.ErrorPageRenderer.<init>(ErrorPageRenderer.java:43)
at oracle.dbtools.http.errors.ErrorPageRenderer.<init>(ErrorPageRenderer.java:34)
at oracle.dbtools.http.errors.ErrorPageFilter.doFilter(ErrorPageFilter.java:113)
at oracle.dbtools.http.filters.HttpFilter.doFilter(HttpFilter.java:47)
at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
at oracle.dbtools.http.secure.ForceHttpsFilter.doFilter(ForceHttpsFilter.java:74)
at oracle.dbtools.http.filters.HttpFilter.doFilter(HttpFilter.java:47)
at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
at oracle.dbtools.http.auth.ForceAuthFilter.doFilter(ForceAuthFilter.java:44)
at oracle.dbtools.http.filters.HttpFilter.doFilter(HttpFilter.java:47)
at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
at oracle.dbtools.http.filters.Filters.filter(Filters.java:67)
at oracle.dbtools.http.entrypoint.EntryPoint.service(EntryPoint.java:82)
at oracle.dbtools.http.entrypoint.EntryPointServlet.service(EntryPointServlet.java:102)
at oracle.dbtools.entrypoint.WebApplicationRequestEntryPoint.service(WebApplicationRequestEntryPoint.java:50)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1667)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:174)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:505)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:427)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:321)
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804)
at java.lang.Thread.run(Unknown Source)
Caused by: oracle.dbtools.url.mapping.TargetNotAvailableException: The connection pool named: |apex|| is not correctly configured, due to the following error(s): Exception occurred while getting connection: oracle.ucp.UniversalConnectionPoolException: All connections in the Universal Connection Pool are in use
at oracle.dbtools.url.mapping.db.PoolInjector.inject(PoolInjector.java:63)
at oracle.dbtools.url.mapping.db.PoolInjector.inject(PoolInjector.java:48)
at oracle.dbtools.url.mapping.db.DatabaseURLMappingBase.injectPLSQLGatewayConnection(DatabaseURLMappingBase.java:933)
at oracle.dbtools.url.mapping.db.DatabaseURLMappingBase.addServices(DatabaseURLMappingBase.java:317)
at oracle.dbtools.url.mapping.URLMappingBase.doFilter(URLMappingBase.java:79)
at oracle.dbtools.url.mapping.filter.URLMappingFilter.doFilter(URLMappingFilter.java:130)
at oracle.dbtools.http.filters.HttpFilter.doFilter(HttpFilter.java:47)
at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
at oracle.dbtools.http.auth.external.ExternalSessionFilter.doFilter(ExternalSessionFilter.java:59)
at oracle.dbtools.http.filters.HttpFilter.doFilter(HttpFilter.java:47)
at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
at oracle.dbtools.rt.authentication.apex.ApexSessionQueryRewriteFilter.doFilter(ApexSessionQueryRewriteFilter.java:58)
at oracle.dbtools.http.filters.HttpFilter.doFilter(HttpFilter.java:47)
at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
at oracle.dbtools.http.cors.CORSResponseFilter.doFilter(CORSResponseFilter.java:88)
at oracle.dbtools.http.filters.HttpResponseFilter.doFilter(HttpResponseFilter.java:45)
at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
at oracle.dbtools.http.filters.AbsoluteLocationFilter.doFilter(AbsoluteLocationFilter.java:65)
at oracle.dbtools.http.filters.HttpResponseFilter.doFilter(HttpResponseFilter.java:45)
at oracle.dbtools.http.filters.FilterChainImpl.doFilter(FilterChainImpl.java:64)
at oracle.dbtools.http.errors.ErrorPageFilter.doFilter(ErrorPageFilter.java:85)
... 47 more
Caused by: oracle.dbtools.common.jdbc.ConnectionPoolConfigurationException: The connection pool named: |apex|| is not correctly configured, due to the following error(s): Exception occurred while getting connection: oracle.ucp.UniversalConnectionPoolException: All connections in the Universal Connection Pool are in use
at oracle.dbtools.common.jdbc.ConnectionPoolExceptions.from(ConnectionPoolExceptions.java:46)
at oracle.dbtools.common.jdbc.DataSourceConnection.getPooledConnection(DataSourceConnection.java:226)
at oracle.dbtools.common.jdbc.DataSourceConnection.getPooledConnection(DataSourceConnection.java:192)
at oracle.dbtools.common.jdbc.DataSourceConnection.getDefaultConnection(DataSourceConnection.java:145)
at oracle.dbtools.common.jdbc.DataSourceConnection.getConnection(DataSourceConnection.java:59)
at oracle.dbtools.common.pools.DataSourceTargetImpl.getConnection(DataSourceTargetImpl.java:60)
at oracle.dbtools.common.config.db.SchemaConnectionFactoryBase.inject(SchemaConnectionFactoryBase.java:84)
at oracle.dbtools.url.mapping.db.PoolInjector.inject(PoolInjector.java:57)
... 67 more
Caused by: java.sql.SQLException: Exception occurred while getting connection: oracle.ucp.UniversalConnectionPoolException: All connections in the Universal Connection Pool are in use
at oracle.ucp.util.UCPErrorHandler.newSQLException(UCPErrorHandler.java:456)
at oracle.ucp.util.UCPErrorHandler.throwSQLException(UCPErrorHandler.java:133)
at oracle.ucp.jdbc.PoolDataSourceImpl.getConnection(PoolDataSourceImpl.java:2004)
at oracle.ucp.jdbc.PoolDataSourceImpl.access$400(PoolDataSourceImpl.java:201)
at oracle.ucp.jdbc.PoolDataSourceImpl$31.build(PoolDataSourceImpl.java:4279)
at oracle.ucp.jdbc.PoolDataSourceImpl.getConnection(PoolDataSourceImpl.java:1917)
at oracle.dbtools.common.config.db.PoolDataSourceThunk.getConnection(PoolDataSourceThunk.java:116)
at oracle.dbtools.common.config.db.RefreshablePoolDataSource.getConnection(RefreshablePoolDataSource.java:107)
at oracle.dbtools.common.config.db.PoolDataSourceThunk.getConnection(PoolDataSourceThunk.java:116)
at oracle.dbtools.common.config.db.PoolDataSourceThunk.getConnection(PoolDataSourceThunk.java:100)
at oracle.dbtools.common.jdbc.DataSourceConnection.getPooledConnection(DataSourceConnection.java:216)
... 73 more
Caused by: oracle.ucp.UniversalConnectionPoolException: All connections in the Universal Connection Pool are in use
at oracle.ucp.util.UCPErrorHandler.newUniversalConnectionPoolException(UCPErrorHandler.java:336)
at oracle.ucp.util.UCPErrorHandler.throwUniversalConnectionPoolException(UCPErrorHandler.java:59)
at oracle.ucp.util.UCPErrorHandler.throwUniversalConnectionPoolException(UCPErrorHandler.java:106)
at oracle.ucp.common.UniversalConnectionPoolImpl.borrowConnectionWithoutCountingRequests(UniversalConnectionPoolImpl.java:289)
at oracle.ucp.common.UniversalConnectionPoolImpl.borrowConnectionAndValidate(UniversalConnectionPoolImpl.java:153)
at oracle.ucp.common.UniversalConnectionPoolImpl.borrowConnection(UniversalConnectionPoolImpl.java:122)
at oracle.ucp.jdbc.JDBCConnectionPool.borrowConnection(JDBCConnectionPool.java:174)
at oracle.ucp.jdbc.oracle.OracleJDBCConnectionPool.borrowConnection(OracleJDBCConnectionPool.java:613)
at oracle.ucp.jdbc.oracle.OracleConnectionConnectionPool.borrowConnection(OracleConnectionConnectionPool.java:103)
at oracle.ucp.jdbc.PoolDataSourceImpl.getConnection(PoolDataSourceImpl.java:1981)
... 81 more
Try to find a file named apex.xml and to increase:
<entry key="jdbc.MaxLimit">...</entry>
check out the open documentation:
https://docs.oracle.com/cd/E56351_01/doc.30/e87809/about-REST-configuration-files.htm#AELIG7162
You might want to add or modify the following entries in the defaults.xml file:
<entry key="jdbc.InitialLimit">3</entry>
<entry key="jdbc.MinLimit">1</entry>
<entry key="jdbc.MaxLimit">10</entry>
you can check out the rest of the parameters for a better experience.
My Oracle support Doc ID 1592840.1 (All Connections In The Universal Connection Pool Are In Use) describes it. If you have access to MOS, have a look. I can't (i.e. I'm not allowed to) post that document's contents here, but - shortly, if it helps, download & install the latest ORDS version from the OTN((here).
There's another "solution" but it requires you to set this and modify that, which is something I can't copy/paste over here, and is too much to type to rephrase it.
I am implementing CQRS with Axon + Spring Boot. My command handles a long computation which results on a large aggregate. The problem is after some minutes the Axon server seems to cancel the command and after all an error is thrown in axon server, please see the log below:
LOGS FROM AXON SERVER:
io.netty.handler.codec.http2.Http2Exception$StreamException: Received DATA frame for an unknown stream 11
at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:129) ~[netty-codec-http2-4.1.35.Final.jar!/:4.1.35.Final]
at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.shouldIgnoreHeadersOrDataFrame(DefaultHttp2ConnectionDecoder.java:531) ~[netty-codec-http2-4.1.35.Final.jar!/:4.1.35.Final]
at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:183) ~[netty-codec-http2-4.1.35.Final.jar!/:4.1.35.Final]
at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onDataRead(Http2InboundFrameLogger.java:48) ~[netty-codec-http2-4.1.35.Final.jar!/:4.1.35.Final]
at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:422) ~[netty-codec-http2-4.1.35.Final.jar!/:4.1.35.Final]
at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:251) ~[netty-codec-http2-4.1.35.Final.jar!/:4.1.35.Final]
at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160) ~[netty-codec-http2-4.1.35.Final.jar!/:4.1.35.Final]
at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41) ~[netty-codec-http2-4.1.35.Final.jar!/:4.1.35.Final]
at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:118) ~[netty-codec-http2-4.1.35.Final.jar!/:4.1.35.Final]
at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:390) ~[netty-codec-http2-4.1.35.Final.jar!/:4.1.35.Final]
at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:450) [netty-codec-http2-4.1.35.Final.jar!/:4.1.35.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) [netty-codec-4.1.35.Final.jar!/:4.1.35.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) [netty-codec-4.1.35.Final.jar!/:4.1.35.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) [netty-codec-4.1.35.Final.jar!/:4.1.35.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.35.Final.jar!/:4.1.35.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.35.Final.jar!/:4.1.35.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.35.Final.jar!/:4.1.35.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) [netty-transport-4.1.35.Final.jar!/:4.1.35.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.35.Final.jar!/:4.1.35.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.35.Final.jar!/:4.1.35.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-transport-4.1.35.Final.jar!/:4.1.35.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.35.Final.jar!/:4.1.35.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) [netty-transport-4.1.35.Final.jar!/:4.1.35.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:617) [netty-transport-4.1.35.Final.jar!/:4.1.35.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:534) [netty-transport-4.1.35.Final.jar!/:4.1.35.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.35.Final.jar!/:4.1.35.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) [netty-common-4.1.35.Final.jar!/:4.1.35.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.35.Final.jar!/:4.1.35.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.35.Final.jar!/:4.1.35.Final]
at java.lang.Thread.run(Unknown Source) [na:1.8.0_181]
Thats all i got, and i don't know why... I am using the latest versions of axon and axon server (4.2.1)
Axon Server has a timeout for commands that is set by default to 5 minutes (300_000 ms). You can increase this timeout by setting the property axoniq.axonserver.default-command-timeout. The value of the property is the timeout in milliseconds.
The maximum message size is 4Mb by default. You can increase that by setting the property axoniq.axonserver.max-message-size in axonserver.properties.
Hope this solves your issue.
Marc
The message "Cancelling command pt.some.admtools.inhousecatalog.command.CheckoutAndParseCommand sent by 16912#POR-5CG8271235, waiting for reply from 16912#POR-5CG8271235.default" is sent when AxonServer cancels the command after 5 minutes (axoniq.axonserver.default-command-timeout).
Can you check what happens in your command handler, and how long the handling takes?